test: add Poll::Pending spill stream coverage for async spill re-entry paths#23353
Open
pantShrey wants to merge 4 commits into
Open
test: add Poll::Pending spill stream coverage for async spill re-entry paths#23353pantShrey wants to merge 4 commits into
pantShrey wants to merge 4 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
The SortMergeJoin refactor in #22230 introduced async spill read paths in both
materializing_stream.rs(poll_spilled_batches) andbitwise_stream.rs(the spilled-batch loop inprocess_key_match_with_filter). Both callSpillFile::read_stream()and poll the resulting byte stream, which can returnPoll::Pending.The existing spill tests exercise spilling and restoring batches, but always run against the default local-file
SpillFile, whoseread_stream()happens to resolve synchronously on first poll in practice. As a result, the re-entry logic in both streams has no dedicated test coverage.What changes are included in this PR?
PendingSpillFile/PendingTempFileFactorypair insort_merge_join/tests.rsthat wraps the default localSpillFilebackend (viaDiskManagerMode::Custom). Every spill read splits the real spill file's bytes into small fixed-size chunks and yieldsPoll::Pendingbefore each chunk, forcingSpillReaderStream's IPC decoder to genuinely suspend mid-read and resume from partially buffered state across multiple real Pending cycles, rather than completing on the first poll.materializing_spill_pending_stream, covering Inner/Left/ Right/Full joins (theBufferedBatchState::Spilledrestore path inpoll_spilled_batches).bitwise_spill_pending_stream, covering LeftSemi/LeftAnti/ RightSemi/RightAnti joins with a filter (theinner_key_spillre-entry path inprocess_key_match_with_filter).Are these changes tested?
Yes -- this is a PR to test recent changes
Are there any user-facing changes?
No.