Honor preemption token in DreamService LLM calls#341
Merged
rockfordlhotka merged 2 commits intomainfrom May 5, 2026
Merged
Conversation
Consolidate 15 near-identical LLM call sites in DreamService into a single InvokeDreamPassAsync<T> helper that builds messages, calls the Balanced-tier LLM with the slot's cancellation token, extracts JSON, logs warnings on parse failure, and deserializes to the target DTO. Previously 14 of 15 sites omitted the slot.Token, so when a user message preempted an in-flight dream cycle the cooperative cancel never reached the LLM call — observed user-perceived latency of 3 min 36 s in production. Threading the token through one helper plugs all sites and prevents future passes from reintroducing the leak. Net -179 lines; behavior preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds RunPassAsync(passName, body) which runs the supplied body inside a try/catch that rethrows OperationCanceledException and converts other exceptions into a per-pass error log. The 10 dream-pass methods that previously hand-rolled the same try/catch wrapper now call into this helper, eliminating ~80 lines of duplicated boilerplate. This also fixes a noisy log line surfaced during k8s validation: when preemption fired, each pass's catch (Exception ex) was catching the OperationCanceledException and logging it as "<pass> pass failed" with a stack trace before the OCE rethrew at the next ThrowIfCancellationRequested between passes. The helper has an explicit catch (OperationCanceledException) that rethrows, so OCE flows cleanly to DreamAsync's outer handler and only the single "dream cycle preempted by user request" line is logged. Also bumps Directory.Build.props from 0.10.28 to 0.10.29 for the deploy used to validate the original fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rockfordlhotka
added a commit
that referenced
this pull request
May 5, 2026
Tracks the agent image already running in the cluster (deployed during PR #341 validation). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
InvokeDreamPassAsync<TResult>helper that builds messages, calls the Balanced-tier LLM with the suppliedCancellationToken, extracts JSON, logs warnings on parse failure, and deserializes to the target DTO.DreamAsyncand 14 pass methods with calls to the helper, threadingslot.Tokenthrough every site.Why
Previously 14 of 15
_llmClient.GetResponseAsynccall sites inDreamServiceomitted the slot's cancellation token. When a user message preempted an in-flight dream cycle, the cooperative cancel signal never reached the LLM call, and the dream loop kept running pass after pass while the user waited (observed user-perceived latency: 3 min 36 s in production). Threading the token through one helper plugs all sites and prevents future passes from reintroducing the leak.Test plan
dotnet build RockBot.slnx— clean compiledotnet test RockBot.slnx— all suites pass (RockBot.Host.Tests: 601 passed)_llmClient.GetResponseAsynccall sites outside the helper"dream cycle preempted by user request"followed by a fast user reply when a user message arrives during a dream cycleCloses #333
🤖 Generated with Claude Code