Skip to content

server: tool role detection and create checkpoints only on user messages#25416

Closed
Kononnable wants to merge 3 commits into
ggml-org:masterfrom
Kononnable:tool-role-detection
Closed

server: tool role detection and create checkpoints only on user messages#25416
Kononnable wants to merge 3 commits into
ggml-org:masterfrom
Kononnable:tool-role-detection

Conversation

@Kononnable

@Kononnable Kononnable commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Overview

Changes in #24176 introduced user message detection. There was a problem with tool call responses which were classified as user messages. Alternative approach (#24176 (comment)) was chosen to create checkpoints on user and tool response messages (guarded by a larger checkpoint_min_step. However, this approach has a problem of not utilizing full batch size during prefill of old conversation with many small tool calls.
This PR introduces detection of tool call response to autoparser and reverts the state to original idea - creating checkpoints only on user messages (no tool responses).

cc: @aldehir, @ggerganov

Prior work: #24176
Fixes: #25213
Fixes: #25320
Fixes: #25023

Additional information

Problem with recognizing tool responses is that they're send as a user message, so they're always caught by a user_msg_start. We can check if tokens are caught by multiple delimiters and assign it to the longest one (most specialized).

--- Reasoning & Content Structure ---
0.06.366.668 W user_msg_start: <|im_start|>user
0.06.366.668 W assistant_msg_start: <|im_start|>assistant
0.06.366.668 W tool_response_start: <|im_start|>user
<tool_response>
0.06.366.669 W reasoning_mode: TAG_BASED

Requirements

@Kononnable Kononnable requested review from a team as code owners July 7, 2026 20:36
@github-actions github-actions Bot added the server label Jul 7, 2026
Comment thread common/chat-diff-analyzer.cpp Outdated
Comment thread common/chat-diff-analyzer.cpp Outdated
@aldehir

aldehir commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

I have another patch to fix the prefill issue. Will submit later today.

Detecting the tool responses still has value but that's at @pwilkin's discretion.

@Kononnable

Copy link
Copy Markdown
Contributor Author

Closing as related issues were fixed using current approach (is_last_user_message approach).

If detecting tool responses would be useful I can create a PR with just those changes (plus necessary changes not to break current checkpoints behavior). However, adding it feels like adding a dead code, unless this detection is required by some already planned feature.

@Kononnable Kononnable closed this Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

3 participants