[multistage] Fix flaky RAW_MESSAGES assertion in GrpcSenderBackpressureTest#19020
Merged
Jackie-Jiang merged 1 commit intoJul 20, 2026
Merged
Conversation
…reTest The `senderObservesBackpressureFromSlowReceiver` test asserted an exact relationship between the loop's `sendCount` and the `RAW_MESSAGES` stat (`rawMessages == sendCount || sendCount + 1`). That equality does not hold in general: `sendCount` counts every `send()` call, but `RAW_MESSAGES` is only incremented once a block passes the terminated / early-terminated guard inside `GrpcSendingMailbox.sendInternal` (`send()` returns void, so the loop cannot tell an aborted attempt from a real send). When the tail of the run interleaves with the watchdog's cancel, one or more attempts bump `sendCount` without incrementing `RAW_MESSAGES`, so `RAW_MESSAGES` can legitimately trail `sendCount`, which tripped the assertion (observed `rawMessages == sendCount - 1`). Replace the exact check with two bounds that hold regardless of interleaving: `RAW_MESSAGES <= sendCount + 1` (each send pushes at most one block, cancel adds at most one error EOS) and `RAW_MESSAGES >= polledCount` (the receiver cannot poll a block the sender never pushed). The lower bound uses the receiver-side poll count as an independent witness, so it still guards against the stat not being tracked at all. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19020 +/- ##
============================================
- Coverage 65.39% 65.37% -0.02%
Complexity 1405 1405
============================================
Files 3423 3423
Lines 215968 215968
Branches 34186 34186
============================================
- Hits 141228 141185 -43
- Misses 63380 63418 +38
- Partials 11360 11365 +5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
raghavyadav01
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
GrpcSenderBackpressureTest.senderObservesBackpressureFromSlowReceiveris flaky. It asserted an exact relationship between the send loop'ssendCountand theRAW_MESSAGESstat:That equality does not hold in general.
sendCountcounts everysend()call the loop makes, butRAW_MESSAGESis only incremented once a block passes the terminated / early-terminated guard insideGrpcSendingMailbox.sendInternal— andSendingMailbox.send(MseBlock.Data)returnsvoid, so the loop cannot tell an aborted attempt from a real send. When the tail of the run interleaves with the watchdog'scancel(), one or more attempts bumpsendCountwithout incrementingRAW_MESSAGES, soRAW_MESSAGESlegitimately trailssendCount. CI hitrawMessages == sendCount - 1, which the assertion rejected.Fix
Replace the exact check with two bounds that hold regardless of how the tail interleaves with the cancel:
RAW_MESSAGES <= sendCount + 1— eachsend()pushes at most one block throughprocessAndSend(oneRAW_MESSAGESincrement), and the cancel adds at most one error EOS.RAW_MESSAGES >= polledCount— the receiver cannot poll a block the sender never pushed.polledCountis measured independently on the receiver side, so this still guards againstRAW_MESSAGESnot being tracked at all.Test-only change; no production code is touched.
Testing
mvn -pl pinot-query-runtime -Dtest=GrpcSenderBackpressureTest testpasses; spotless / checkstyle / license checks pass on the module.