Skip to content

Fix memory reservation starvation in sort-merge#20641

Closed
xudong963 wants to merge 1 commit intomainfrom
fix/sort-merge-reservation-starvation
Closed

Fix memory reservation starvation in sort-merge#20641
xudong963 wants to merge 1 commit intomainfrom
fix/sort-merge-reservation-starvation

Conversation

@xudong963
Copy link
Member

Which issue does this PR close?

  • Closes #.

Rationale for this change

This PR fixes memory reservation starvation in sort-merge when multiple sort partitions share a GreedyMemoryPool.

When multiple ExternalSorter instances run concurrently and share a single memory pool, the merge phase starves:

  1. Each partition pre-reserves sort_spill_reservation_bytes via merge_reservation
  2. When entering the merge phase, new_empty() was used to create a new reservation starting at 0 bytes, while the pre-reserved bytes sat idle in ExternalSorter.merge_reservation
  3. Those freed bytes were immediately consumed by other partitions racing for memory
  4. The merge could no longer allocate memory from the pool → OOM / starvation

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added the physical-plan Changes to the physical-plan crate label Mar 2, 2026
@xudong963 xudong963 closed this Mar 2, 2026
@xudong963 xudong963 deleted the fix/sort-merge-reservation-starvation branch March 2, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant