Skip to content

basert serve: support loading an external chat template (--chat-template) #42

Description

@chaoshou-coder

Problem. baseRT's built-in chat templates reject mid-list role:"system"
messages. The official Qwen3.5/3.6 template does:

Qwen/Qwen3.6-35B-A3B, commit 7da1103:
{%- if message.role == "system" %}
    {%- if not loop.first %}
        {{- raise_exception('System message must be at the beginning.') }}
    {%- endif %}
{%- endif %}

baseRT catches the exception and silently falls back to a generic ChatML
template: the request succeeds but loses model-specific formatting, tool-call
syntax, and reasoning delimiters. Agentic coding tools (Claude Code, and
others) inject role:"system" reminders mid-array at runtime, so this affects
every agentic workload on a Qwen3.5/3.6 model.

Request. A --chat-template <path> flag (or equivalent, as llama.cpp
provides) to load an external chat template at serve time, so deployments can
use community templates without re-converting models.

Reference. A community fix already exists:
froggeric/Qwen-Fixed-Chat-Templates
v21.3 (MIT) — replaces the exception with per-position ChatML rendering of each
system block, and adds agentic features (tool truncation, consecutive-failure
detection, inline thinking control). A reference copy and analysis are in the
cc-bridge PR (cc-bridge/misc/chat-template-fix.md and
chat_template-patched.jinja).

Impact. Fixing this at the engine level removes the system-merging
workaround from downstream bridges (cc-bridge's --no-system-merge exists for
this) and lets users deploy agentic-optimized templates without converting.


Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions