Observation
Found while addressing review feedback on #296, which trims a stop-string suffix from the final
token at a character boundary.
TextGenerator's per-token callback is Action<int> — it receives a token id. Once a token has
been partially trimmed, there is no id that represents the trimmed text, so the callback cannot
express what was actually emitted.
The reviewer suggested invoking the callback for the kept portion. That cannot work as stated:
passing the original id would render the untrimmed text and leak the stop string back to the
consumer, contradicting StopStringCondition's contract; omitting the token loses the kept
characters entirely.
Impact
Consumers driving output from the token callback see a slightly different stream from consumers
reading the returned text: the final partial token is either absent or, if naively re-rendered,
includes the stop string that was meant to be removed. The non-streaming path is correct; it is the
callback surface that cannot represent the result.
The same limitation applies to any future transformation that operates on text rather than whole
tokens.
Proposed change
Add a text-level streaming callback alongside the id-level one, e.g.
Action<ReadOnlySpan<char>> (or an overload taking both id and text, with the text authoritative).
The trimming path then emits the kept characters with no id, and ordinary tokens emit both.
Worth deciding at the same time whether the id-level callback should remain for consumers that
genuinely want ids (logprobs, speculative-decoding instrumentation) — I would keep it, and document
that it is the pre-trim stream.
Acceptance criteria
References
Observation
Found while addressing review feedback on #296, which trims a stop-string suffix from the final
token at a character boundary.
TextGenerator's per-token callback isAction<int>— it receives a token id. Once a token hasbeen partially trimmed, there is no id that represents the trimmed text, so the callback cannot
express what was actually emitted.
The reviewer suggested invoking the callback for the kept portion. That cannot work as stated:
passing the original id would render the untrimmed text and leak the stop string back to the
consumer, contradicting
StopStringCondition's contract; omitting the token loses the keptcharacters entirely.
Impact
Consumers driving output from the token callback see a slightly different stream from consumers
reading the returned text: the final partial token is either absent or, if naively re-rendered,
includes the stop string that was meant to be removed. The non-streaming path is correct; it is the
callback surface that cannot represent the result.
The same limitation applies to any future transformation that operates on text rather than whole
tokens.
Proposed change
Add a text-level streaming callback alongside the id-level one, e.g.
Action<ReadOnlySpan<char>>(or an overload taking both id and text, with the text authoritative).The trimming path then emits the kept characters with no id, and ordinary tokens emit both.
Worth deciding at the same time whether the id-level callback should remain for consumers that
genuinely want ids (logprobs, speculative-decoding instrumentation) — I would keep it, and document
that it is the pre-trim stream.
Acceptance criteria
non-streaming path, including the trimmed final token.
text equals the returned text.
References
surface, since doing so is beyond its scope
src/DotLLM.Engine/TextGenerator.cs,StopStringCondition