Fix MapAsyncPartitioned demand-resume deadlock#3365
Merged
He-Pin merged 1 commit intoJul 18, 2026
Conversation
Motivation: Completed mapAsyncPartitioned results can remain buffered until downstream demand resumes. The partitionsInProgress empty shortcut bypassed emission and could leave the stage unable to make progress. Modification: Always evaluate buffered completed elements in the ordered and unordered push paths. Add focused backpressure tests for both variants. Result: Completed elements are emitted when downstream demand resumes without changing ordering or partition-parallelism semantics. Tests: - stream-typed-tests MapAsyncPartitionedSpec: 5 repeated runs, 110/110 passed - stream-tests FlowMapAsyncPartitionedSpec: 24/24 passed - scalafmt diff mode and list check: passed - headerCreateAll, checkCodeStyle, and +headerCheckAll: passed - validatePullRequest: relevant stream batches passed; overall failed on unrelated macOS ARM64/JDK 25 environment tests documented in the PR - git diff --check: passed - Qoder stdout review: No must-fix findings References: Fixes apache#2886
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Completed mapAsyncPartitioned results can remain buffered until downstream demand resumes. The partitionsInProgress empty shortcut bypasses the push logic, so a completed buffer can stop making progress.
Modification
Remove the partitionsInProgress.isEmpty short-circuit from both ordered and unordered push paths so buffered results are always examined when demand is available.
Add focused backpressure tests for ordered and unordered variants. The tests fill parallelism, consume one result, complete the remaining futures without demand, then verify all buffered results are emitted once demand resumes.
Result
Completed buffered elements are emitted after downstream demand resumes. Ordering and per-partition parallelism semantics are unchanged.
References
Fixes #2886