fix: surface a deterministic exception from ParallelWorker on concurrent failures#6469
Open
DABH wants to merge 1 commit into
Open
fix: surface a deterministic exception from ParallelWorker on concurrent failures#6469DABH wants to merge 1 commit into
DABH wants to merge 1 commit into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
adk-bot
added
the
core
…ent failures asyncio.wait returns completed tasks as an unordered set. _ParallelWorker iterated that set directly to pick the exception to raise, so when two or more branches failed within the same wake-up, which branch's exception surfaced depended on set iteration order (object-hash dependent). This made failure output differ between runs — and between record and replay for frameworks that resume or replay agent workflows and filter retries by exception type (RetryConfig.exceptions). Iterate completed tasks in input-index order instead, so the lowest-index failed branch's exception is surfaced consistently. Behavior is otherwise unchanged. The regression test fails ~50% of the time without the fix (two branches failing in the same wake-up) and always passes with it.
DABH
force-pushed
the
parallel-worker-deterministic-errors
branch
from
July 24, 2026 23:14
cf84e6b to
93d0bb3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the bug
asyncio.waitreturns completed tasks as an unorderedset._ParallelWorker._run_impliterates that set directly to decide which failed branch's exception to raise, so when two or more branches fail within the same wake-up, the surfaced exception depends on set iteration order (object-hash dependent). Failure output then differs between runs - and between record and replay for embedders that resume or replay agent workflows - andRetryConfig.exceptions(which filters retries by exception type name) can make a different retry decision on each execution of the same failure.Describe the fix
Iterate completed tasks in input-index order (the
_worker_indexalready stored on each task) when scanning for failures, so the lowest-index failed branch's exception is surfaced consistently. Behavior is otherwise unchanged.Testing plan
New regression test
tests/unittests/workflow/test_workflow_parallel_worker.py::test_parallel_worker_simultaneous_failures_raise_lowest_index: two branches that both fail immediately, run 10×, asserting the propagated exception is always the lowest-index item's. It fails roughly half the time without the fix and always passes with it.All pre-commit hooks pass. Related (same determinism theme, independent change): #6468.