server: tool role detection and create checkpoints only on user messages#25416
Closed
Kononnable wants to merge 3 commits into
Closed
server: tool role detection and create checkpoints only on user messages#25416Kononnable wants to merge 3 commits into
Kononnable wants to merge 3 commits into
Conversation
pwilkin
reviewed
Jul 7, 2026
pwilkin
reviewed
Jul 7, 2026
Contributor
|
I have another patch to fix the prefill issue. Will submit later today. Detecting the tool responses still has value but that's at @pwilkin's discretion. |
2 tasks
Contributor
Author
|
Closing as related issues were fixed using current approach ( If detecting tool responses would be useful I can create a PR with just those changes (plus necessary changes not to break current checkpoints behavior). However, adding it feels like adding a dead code, unless this detection is required by some already planned feature. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Changes in #24176 introduced user message detection. There was a problem with tool call responses which were classified as user messages. Alternative approach (#24176 (comment)) was chosen to create checkpoints on user and tool response messages (guarded by a larger
checkpoint_min_step. However, this approach has a problem of not utilizing full batch size during prefill of old conversation with many small tool calls.This PR introduces detection of tool call response to autoparser and reverts the state to original idea - creating checkpoints only on user messages (no tool responses).
cc: @aldehir, @ggerganov
Prior work: #24176
Fixes: #25213
Fixes: #25320
Fixes: #25023
Additional information
Problem with recognizing tool responses is that they're send as a user message, so they're always caught by a
user_msg_start. We can check if tokens are caught by multiple delimiters and assign it to the longest one (most specialized).Requirements