Skip to content

fix(runtime): deliver StreamStopped with a bounded blocking send - #4153

Merged
dgageot merged 1 commit into
mainfrom
fix/4136-bounded-stream-stopped-delivery
Sep 4, 2026
Merged

fix(runtime): deliver StreamStopped with a bounded blocking send#4153
dgageot merged 1 commit into
mainfrom
fix/4136-bounded-stream-stopped-delivery

Conversation

@aheritier

Copy link
Copy Markdown
Collaborator

Summary

Fixes #4136: the TUI got stuck on "Working…" after the stream had actually finished, because finalizeEventChannel emitted the terminal StreamStoppedEvent non-blockingly and silently dropped it whenever the runtime's 128-slot event buffer was full at teardown. Under normal streaming back-pressure (e.g. PersistenceObserver doing a synchronous per-delta SQLite write, or any consumer slower than the model's delta rate) the buffer is routinely full at exactly that moment, so the TUI/supervisor/leantui — which only clear their busy state on receipt of that event — never learned the stream had ended.

Change

  • finalizeEventChannel now delivers StreamStopped with a bounded blocking send (~5s deadline, new boundedChannelSink/bounded() helper in event_sink.go, mirroring the existing nonBlocking()). A consumer still draining the channel — even one that already saw ctx cancelled (Esc) but keeps draining to close — reliably receives it. It's dropped only if nothing accepts it within the deadline, i.e. the consumer has genuinely abandoned the channel.
  • Deliberately not unbounded: a consumer audit found several call sites that legitimately stop reading before close (a2a, acp, cli runner, LocalRuntime.Run, embeddedchat, server RunSession). An unbounded send there would reintroduce the exact teardown deadlock fix: guard elicitation channel close against in-flight sends #3070 fixed, which decide StreamStopped ordering vs session-end hooks and document teardown trade-offs from #3070 #3074/docs(runtime): document StreamStopped ordering and teardown trade-offs (#3074) #3275 addressed by making this emit non-blocking (and documented delivery as best-effort) in the first place.
  • Updated the finalizeEventChannel/StreamStoppedEvent doc comments to describe the new bounded-delivery contract; channel close remains the one guaranteed terminal signal.
  • pkg/a2a/adapter.go: the ADK turn-complete event was only ever sent from the StreamStoppedEvent case, so a dropped event left the turn hanging. Added a channel-close fallback so it still completes if that event was never delivered.

Tests

  • New pkg/runtime/stream_stopped_delivery_test.go: a deterministic regression test reproducing the TUI shows "working" even after stream has ended for Qwen 3.8 (served locally via vLLM) #4136 race (slow EventObserver + ≥250 streamed deltas) asserting exactly one StreamStopped, last before close; plus a "slow-but-alive consumer eventually receives it" case. Passes -count=20 with zero flakes.
  • Reworked TestLocalRuntime_FinalizeEventChannelDoesNotDeadlockWhenBufferFullAndConsumerGoneTestLocalRuntime_FinalizeEventChannelDropsStreamStoppedAfterBoundedTimeout: injects a short timeout, asserts the send waits out the deadline (proving bounded, not instant-drop) and still returns/drops for a genuinely abandoned consumer.
  • TestLocalRuntime_FinalizeEventChannelStreamStoppedIsLastBeforeClose / ...EmitsStreamStoppedOnce pass unchanged.

go build, go test ./..., golangci-lint run (v2.13.1) all clean. (pkg/rag/treesitter fails in this sandbox only — pre-existing, missing gcc for CGO, unrelated to this change.)

Independent review (sub-agent) returned only low-severity, non-blocking nits (comment verbosity, a defensive suggestion for an unreachable wait<=0 case, a minor doc-wording ambiguity) — none required changes.

finalizeEventChannel emitted the terminal StreamStoppedEvent non-blockingly
and dropped it whenever the runtime's 128-slot event buffer was full at
teardown. Under normal streaming back-pressure (a consumer, e.g. the
PersistenceObserver, doing a synchronous per-delta SQLite write) the buffer
is routinely full at that exact moment, so the TUI/supervisor/leantui never
saw the event and stayed stuck on "Working…" until Esc (#4136).

Replace the non-blocking emit with a bounded blocking send (~5s deadline,
event_sink.go's new boundedChannelSink/bounded()): a consumer still
draining the channel reliably receives it, even past context cancellation
(the TUI keeps draining after Esc); it is dropped only once nothing
accepts it within the deadline, i.e. the consumer has genuinely abandoned
the channel. Deliberately not unbounded: a consumer audit found several
call sites that legitimately stop reading before close (a2a, acp, cli
runner, LocalRuntime.Run, embeddedchat, server RunSession), and an
unbounded send there would reintroduce the #3070 teardown deadlock that
#3074/#3275 previously fixed by making the emit non-blocking in the first
place.

Also harden pkg/a2a/adapter.go: it only completed the ADK turn from the
StreamStoppedEvent case, so an ever-dropped event left the turn hanging.
Add a channel-close fallback so a closed events channel without a prior
StreamStopped still completes the turn.
@aheritier
aheritier marked this pull request as ready for review September 3, 2026 18:18
@aheritier
aheritier requested a review from a team as a code owner September 3, 2026 18:18
@aheritier aheritier added area/a2a Agent-to-Agent protocol, A2A server, inter-agent communication area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection area/testing Test infrastructure, CI/CD, test runners, evaluation kind/fix PR fixes a bug (maps to fix:). Use on PRs only. labels Sep 3, 2026
@dgageot
dgageot merged commit c4e2a7c into main Sep 4, 2026
16 checks passed
@dgageot
dgageot deleted the fix/4136-bounded-stream-stopped-delivery branch September 4, 2026 06:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/a2a Agent-to-Agent protocol, A2A server, inter-agent communication area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection area/testing Test infrastructure, CI/CD, test runners, evaluation kind/fix PR fixes a bug (maps to fix:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TUI shows "working" even after stream has ended for Qwen 3.8 (served locally via vLLM)

2 participants