engine(streaming): stop-string suffix trim across token-boundary streaming chunks - #306
Draft
jamesburton wants to merge 2 commits into
Draft
engine(streaming): stop-string suffix trim across token-boundary streaming chunks#306jamesburton wants to merge 2 commits into
jamesburton wants to merge 2 commits into
Conversation
… whole token (#107) When a `StopStringCondition` matched, `TextGenerator` removed the entire last token from the generated-id list. That over-trimmed whenever the stop string was a strict suffix of the last token's decoded text — the headline example from the issue: last token decodes to `"ld<|im_end|>"`, stop string is `"<|im_end|>"`, the user saw `"Hello, wor"` instead of `"Hello, world"`. Introduces `StopSuffixTrimmer` — a static helper that finds the longest stop-string suffix of the decoded text and trims it at the character (UTF-16) boundary, defending against splitting a surrogate pair. `CheckStopConditions` now also reports the matched condition index; when the match was a `StopStringCondition`, the call sites keep the last token in `generatedIds` and `BuildResponse` performs the suffix trim on the fully-decoded text. EOS / max-tokens / other non-string stop conditions keep the original token-removal semantics (their "token" is conceptually the terminator itself, not text-bearing). Regression tests cover the headline strict-suffix case, longest-of-multiple matches, the no-match passthrough, ignoring non-`StopStringCondition` entries, and the surrogate-pair safety trim. Note: this PR focuses on the non-streaming `Generate` paths (prefill + greedy/spec decode loops returning an `InferenceResponse`). The streaming `GenerateStream` path keeps the existing token-removal behaviour; applying the same character-level trim to the streaming SSE output is a follow-up. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…aming chunks (#121) Extends the non-streaming stop-string suffix trim from #107 to the SSE streaming path. Where a stop string may straddle multiple already-decoded tokens (e.g. "<|im_end|>" split across "<|", "im_", "end|>"), the prior streaming code emitted the partial-match prefixes verbatim and only suppressed the final token. * StreamingStopBuffer (new) — rolling tail buffer holding the last K chars (K = max stop-string decoded char-length, computed from registered stop conditions). Each appended chunk emits the safe head and retains the trailing K chars; on stop-match detection, flushes the trimmed remainder up to the match start and emits the SSE terminal sentinel. * TextGenerator.GenerateStreamingAsync — routes per-chunk text through StreamingStopBuffer when at least one StopStringCondition is registered; falls through unchanged otherwise (no overhead for stop-token-only stops). * Discriminating tests — three-token split "<|" / "im_" / "end|>" emits zero chunks containing any part of the sequence; with the buffer disabled the same input emits two chunks containing partial matches. * Surrogate-pair safety — buffer trims at char boundaries; multi-char codepoints in stop strings are not split. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses an item from #121. Stop-string suffix trimming now works correctly across token-boundary streaming chunks — a stop string that spans the boundary between two SSE chunks is detected and trimmed at the exact character boundary.
Test results
TextGeneratorStopStringStreamingTests(251 lines).Stack
This PR depends on:
Once #296 lands, this PR will rebase cleanly. The base will then be
main.