Fix race condition in UserPipelineTracker.clear() and various problems

[shunping]

The PR mainly fixes the race condition that causes the following flaky test by adding lock on the operations.

_______ InteractiveBeamComputeTest.test_async_computation_result_cancel ________
[gw4] linux -- Python 3.13.13 /runner/_work/beam/beam/sdks/python/test-suites/tox/py313/build/srcs/sdks/python/target/.tox-py313-cloud/py313-cloud/bin/python
target/.tox-py313-cloud/py313-cloud/lib/python3.13/site-packages/apache_beam/runners/interactive/testing/mock_env.py:75: in tearDown
    self.current_env.cleanup()
target/.tox-py313-cloud/py313-cloud/lib/python3.13/site-packages/apache_beam/runners/interactive/interactive_environment.py:306: in cleanup
    self.evict_tracked_pipelines(pipeline)
target/.tox-py313-cloud/py313-cloud/lib/python3.13/site-packages/apache_beam/runners/interactive/interactive_environment.py:627: in evict_tracked_pipelines
    self._tracked_user_pipelines.clear()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <apache_beam.runners.interactive.user_pipeline_tracker.UserPipelineTracker object at 0x7d5e0c53c4d0>

    def clear(self) -> None:
      """Clears the tracker of all user and derived pipelines."""
      # Remove all local_tempdir of created pipelines.
>     for p in self._pid_to_pipelines.values():
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E     RuntimeError: dictionary changed size during iteration

target/.tox-py313-cloud/py313-cloud/lib/python3.13/site-packages/apache_beam/runners/interactive/user_pipeline_tracker.py:71: RuntimeError

Additionally, it fixes a resource leak for UserPipelineTracker.evict where evicted pipelines were not removed from _pid_to_pipelines.

Finally, it fixes a typo: _memoize_pipieline -> _memoize_pipeline.

[shunping]

r: @Amar3tto @Abacn

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a race condition in the UserPipelineTracker class that frequently caused flaky tests. By implementing an RLock and ensuring that all read/write operations and iterations on internal state are protected, the changes prevent dictionary mutation errors during concurrent access. A new test case was also included to validate the fix by explicitly orchestrating a race condition.

Highlights

• Thread Safety: Introduced a reentrant lock (threading.RLock) to the UserPipelineTracker class to ensure thread-safe access to internal dictionaries.
• Race Condition Fix: Wrapped critical sections and dictionary iterations with the lock to prevent RuntimeError when the tracker is modified during iteration.
• Regression Testing: Added a new test case in user_pipeline_tracker_test.py that simulates concurrent modifications to verify the fix for the reported race condition.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[gemini-code-assist]

Code Review

This pull request introduces thread safety to the UserPipelineTracker class by adding an RLock and wrapping critical dictionary operations in with self._lock blocks. It also includes a new test case to verify the fix for a potential race condition in the clear() method. My feedback highlights that the type hints for _derived_pipelines and _pid_to_pipelines are incomplete and should be updated, and suggests refactoring the clear() method to ensure atomicity and prevent potential resource leaks during concurrent operations.

[gemini-code-assist]

Code Review

This pull request introduces thread safety to the UserPipelineTracker class by implementing a reentrant lock (RLock) across its core methods. Key changes include wrapping dictionary access and mutations within lock blocks, refining the clear method to perform I/O operations outside the lock, and adding a regression test to verify thread safety during concurrent mutations. Review feedback correctly identifies a logic error in the evict method that causes unintended tree-wide deletions when a derived pipeline is targeted, as well as a resource leak where pipelines are not removed from the internal PID tracker. Additionally, several instances of a typo in the internal method name _memoize_pipieline were flagged for correction.

[shunping]

assign set of reviewers

[github-actions]

Assigning reviewers:

R: @jrmccluskey for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).