Drain backend notifications before per-call client close#5960
Merged
Conversation
JAORMX
requested review from
ChrisJBurns,
amirejaz,
blkt,
jerm-dro,
jhrozek,
rdimitrov and
tgrunnagle
as code owners
July 24, 2026 07:06
rdimitrov
previously approved these changes
Jul 24, 2026
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5960 +/- ##
==========================================
+ Coverage 71.82% 71.89% +0.07%
==========================================
Files 708 708
Lines 72798 72805 +7
==========================================
+ Hits 52287 52345 +58
+ Misses 16772 16708 -64
- Partials 3739 3752 +13 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
TestForwarding_Progress_RealBackend (and _Logging_) flaked because a fire-and-forget backend notification emitted mid tools/call was lost under load. The notification and the tool result travel on separate streams (standalone SSE vs the tools/call response), read through one FIFO receive loop; CallTool returns as soon as the result is read and its deferred Close cancels that loop, so a not-yet-drained notification is discarded and never forwarded downstream -- a permanent loss, which is why bumping the downstream wait timeout never fixed it. Blocking server->client requests (elicitation/sampling) are immune because the backend tool blocks on the response. Before Close, when server->client forwarding is bound, issue a synchronous ping as a drain barrier: its round-trip lets the receive loop drain the already-buffered notification off the shared channel and enqueue it for handling before teardown. Best-effort (the result is already in hand) and skipped entirely when forwarding is not bound, so non-forwarding calls keep the fast path. This is the minimal in-repo mitigation; the robust fix is upstream in the SDK (route mid-call notifications on the correlated request stream, or drain gracefully on close). Tracked for follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WG3mSjVGWNc8nfgbkdd79
JAORMX
force-pushed
the
deflake-integration-tests
branch
from
July 24, 2026 07:23
58ea112 to
c103295
Compare
Collaborator
Author
|
Rebased onto latest |
rdimitrov
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TestForwarding_Progress_RealBackend(and_Logging_) flaked on CI because a fire-and-forget backend notification emitted midtools/call(notifications/progress,notifications/message) was dropped under load, not merely delayed.The notification and the tool result travel on separate streams — the notification on the standalone SSE stream, the result on the
tools/callresponse stream — read through a single FIFO receive loop.CallToolreturns as soon as the result is read, and its deferredClosecancels that loop; a notification not yet drained off the channel is then discarded and never reaches the downstream client. That is why the earlier fix (bumping the downstream wait 5s→15s) never helped — the message is lost, not slow. Blocking server→client requests (elicitation/sampling) are immune because the backend tool blocks on the response, so the client is still reading when they arrive.Fix: before
Close, when server→client forwarding is bound, issue a synchronousPingas a drain barrier. The backend flushes the notification onto the wire before returning the result, so by the timeCallToolreturns its bytes are already buffered on the client; the ping's round-trip lets the receive loop drain that buffered notification off the shared channel and enqueue it for handling, andClosethen waits for enqueued handlers before teardown. Best-effort (the result is already in hand) and skipped entirely when forwarding is not bound, so non-forwarding calls keep the fast path with no extra round-trip.Type of change
Test plan
task test) —pkg/vmcp/server,pkg/vmcp/clientpass with-racetask lint-fix) — 0 issuesReproduced reliably under CPU contention before the fix (progress + logging forwarding tests time out); after the fix the full
TestForwardingcluster passes 30/30 under 24× load and under-race -count=20.API Compatibility
v1beta1API (no operator/CRD surface is touched).Special notes for reviewers
main. This PR originally also carried a fix for the standalone-SSE filtered-tool-call test, but that was landed independently in Update standalone-SSE test for filtered-tool error #5958, so that commit was dropped on rebase — only the forwarding fix remains.toolhive-coremcpcompat: route mid-call notifications on the correlated request stream instead of the lossy standalone stream, or drain gracefully on close. Worth a linked upstream issue as follow-up.tools/call, only when server→client forwarding is bound. The per-call client already does Start+Initialize+Close, so this is marginal.Generated with Claude Code