Emit tool-calling special tokens in ORT GenAI config - #524
Conversation
|
|
|
@copilot other qwen 3.5+ models can use the same tags as qwen3 |
Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>
Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>
Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>
Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>
|
@copilot actually, see if you could read from tokenizer_config.json and avoid model specific logic. |
|
@copilot or tokenizer.json I should say. You decide |
Derive tool and reasoning delimiter IDs from tokenizer_config.json so exports are not tied to specific model families. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>
Avoid selecting an arbitrary delimiter ID when tokenizer metadata contains conflicting spellings. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>
Use distinct raw and parsed token ID names when reading tokenizer metadata. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>
Clarify that special token parsing reads copied tokenizer configuration from the export directory. Signed-off-by: GitHub <noreply@github.com> Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>
|
@copilot survey all models that we support that has this thinking and tool calling tokens, and make sure we have covered all forms of the contents. |
|
In a follow up PR declare model capabilities so that we assert tool calling supported models has a tool calling string emitted. |
Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>
Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>
Survey confirmed ORT GenAI recognizes the Qwen/Qwen3 and Phi delimiter spellings already mapped. 4a4afd6 now covers every supported spelling in both Hugging Face metadata layouts, with vocabulary and cross-file-conflict tests.
Implemented generic extraction from both |
Performance Comparison
|
|
@copilot resolve merge conflicts. |
There was a problem hiding this comment.
Pull request overview
Adds support for emitting ORT GenAI “tool calling” and “thinking/reasoning” delimiter token IDs into genai_config.json, sourced from tokenizer metadata and guarded to avoid overriding core BOS/EOS/PAD settings. This extends the ORT GenAI integration so downstream runtime parsing can reliably detect tool/thinking blocks for Qwen2/Qwen3 and Phi (phi3 config type).
Changes:
- Extend
GenaiConfigGeneratorwith awith_special_tokens()API and emit those fields undermodelingenai_config.json, disallowing overrides ofbos_token_id/eos_token_id/pad_token_id. - Add tokenizer-metadata parsing in the ORT GenAI exporter to discover tool/thinking delimiter IDs (only when in-vocab; ignore ambiguous/conflicting sources).
- Add regression tests covering override protection, vocab-boundary behavior, malformed tokenizer metadata, and ambiguity handling.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| src/mobius/integrations/ort_genai/genai_config.py | Add support for extra model-level special token ID fields and prevent overriding standard token IDs. |
| src/mobius/integrations/ort_genai/genai_config_test.py | Add unit test ensuring special tokens cannot replace BOS/EOS/PAD fields. |
| src/mobius/integrations/ort_genai/auto_export.py | Parse tool/thinking delimiter IDs from tokenizer metadata and inject into emitted genai config. |
| src/mobius/integrations/ort_genai/auto_export_test.py | Add exporter regression tests for delimiter extraction, boundary conditions, malformed input, and ambiguity resolution. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Signed-off-by: GitHub <noreply@github.com> Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>
|
Looks great, thanks for adding this! Could we add some unit tests that leverage Mobius to ensure the workflow is accurate? |
Download tokenizer metadata from known Qwen and Phi checkpoints and verify the generated ORT GenAI config emits the expected tool-call and reasoning token IDs.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
ORT GenAI config output now declares Qwen and Phi tool-calling delimiters, including Qwen3 reasoning delimiters, so runtime token parsing can recognize tool and thinking blocks.
Model token mappings
<tool_call>/</tool_call><think>/</think>phi3config type):<|tool_call|>/<|/tool_call|>Safety
{ "model": { "bot_token_id": 151657, "eot_token_id": 151658, "bor_token_id": 151667, "eor_token_id": 151668 } }