What actually affects Qwen3.6-27B tool-calling reliability

[mgabor3141]

Posting a corrected version of this writeup. The original recommended leaving MTP on; subsequent testing on a real coding-agent task showed that recommendation was wrong for an important class of prompts. Three findings below, all measured.

Stack for reference: 2x RTX 3090 TP=2, vLLM nightly bf610c2f, Qwen3.6-27B AutoRound INT4, qwen3_xml tool parser, qwen3 reasoning parser. Client is our coding agent (pi) with 8 tools and a developer-role system prompt; sampling preset matches the Qwen3 thinking recipe (temperature=0.6, top_p=0.95, top_k=20, min_p=0), which the club-3090 default dual composes already set via --override-generation-config (example: bf16.yml#L138-L141).

1. MTP=3 amplifies the CodeAct/Cline attractor on shell-flavored prompts

A real pi request, "Pull the vllm fork and check out this PR for review. Run the changed tests and validate the assumptions claimed in the PR and the linked issue.", returns 10/10 invalid responses with MTP=3 on. The model emits CodeAct grammar in the content channel (<bash>command>git clone --depth 1 https://github.com/vllm-project/vllm.git ...</bash>), no <tool_call> wrapper, zero parsable tool_calls. Turning MTP off and replaying the identical body returns 10/10 clean tool calls invoking the correct github tool with the right arguments. Everything else (sampling preset, parser, system prompt) held constant.

Config	CLEAN	PROSE	GARBLED
MTP=3, system prompt with skills listing	0	0	10
MTP=3, skills listing removed	9	1	0
MTP off, with skills listing	10	0	0
MTP off, skills listing removed	10	0	0

The mechanism is vllm-project/vllm#40875: at template-token ambiguity positions (<<function vs <<tool vs <<bash), the main head's probability mass is heavily skewed toward CodeAct-shaped tokens when the input is git/shell-flavored. MTP's draft head proposes the wrong template token, the rejection sampler accepts it (main head also assigns it >threshold probability), and the trajectory commits to invalid grammar. With MTP off the attractor still exists but the model recovers on the next token.

Two earlier measurements gave the opposite signal:

• 2026-05-20: MTP=3 produced ~50% corruption on chained tool-call agent traffic. Turned off.
• 2026-05-25: a 24-cell rebench (file-exploration prompts) found zero garble with MTP=3. Turned back on.
• 2026-05-26: replay of the user's actual shell-flavored prompt is 10/10 garbled with MTP=3. Off again.

None of those three were wrong individually; each sampled a different slice of (prompt shape, sampling, MTP). The honest summary: MTP correctness is prompt-shape dependent on Qwen3.6-27B, and we cannot afford to sample the full surface for every workload change. Keeping MTP off until one of vllm#40875 Option B/C (per-request opt-out or context-aware spec-decode) or vllm#42960 (batch-invariant GDN attention) lands.

2. The qwen3_xml streaming tool parser has two independent bugs

Both orthogonal to MTP.

Multi-function-per-tool_call (vllm-project/vllm#43713, fix in #43714). When the model emits multiple <function=...> blocks inside a single <tool_call> wrapper, the streaming parser produces invalid JSON in the tool_call arguments: {"path":"README.md"}} (extra closing brace), or {}{} when the functions have no parameters. Two interacting causes in StreamingXMLToolCallParser: function-scoped state is not cleared on </function> close, and tool_call_index is only bumped at <tool_call> open so a second function in the same tool_call slot inherits the prior function's args and triggers a duplicate-close emit on the next _auto_close_open_parameter_if_needed("function"). Observed in ~6-7% of trials on a 200-trial matrix bench. The fix maps each <function=...> to its own OpenAI tool_call slot via a functions_in_tool_call counter, clears function-scoped state on close, and gates the function-end branch on current_function_open.

Streaming id re-emission (vllm-project/vllm#43783, stacked on #43714). The parser was setting id=self.current_call_id on every emitted DeltaToolCall, including continuation/args deltas. Per the OpenAI streaming protocol, id (like name) is announced exactly once per slot; subsequent deltas must carry only index + function.arguments. The repo's own test suite had ten streaming xfails all tracing back to this. Fix: _emit_delta strips id from continuation deltas via a _slots_with_id_emitted: set[int] (single enforcement point, all callers unmodified), and _merge_new_deltas_to_single_response keys on index instead of id. With both patches applied the parser test file is 26 passed, 0 xfailed, 0 failed; pre-fix it was 15 passed, 10 xfailed.

3. Choice of parser: qwen3_xml over qwen3_coder on Qwen3.6-27B thinking mode

The qwen3_coder parser combined with --reasoning-parser qwen3 has a known streaming bug where tool_call XML leaks into delta.content instead of being structured into tool_calls (club-3090#145, club-3090#72, both closed by recommending the swap to qwen3_xml). We use qwen3_xml.

Recommended order for new troubleshooters

1. Confirm sampling actually reaches the sampler. The club-3090 dual composes set the recipe via --override-generation-config; if you proxy or wrap, log the request body that hits vLLM.
2. Disable MTP if you see CodeAct-shaped content on shell/git-flavored agent prompts. VLLM_DISABLE_MTP=true (or just drop --speculative-config from your args). Re-enable when one of the upstream opt-out mechanisms lands.
3. Use qwen3_xml tool parser. Apply vllm#43714 if you're seeing }} / {}{} in streamed tool_calls.function.arguments. Apply vllm#43783 on top if you (or your client) strictly validate that streamed DeltaToolCall.id appears only on the first delta of each slot.

What did not move the needle, in case anyone goes down those paths: single vs dual card, context length, chat_template_kwargs.enable_thinking, KV/prefix cache warmth (slight on rates, irrelevant on failure modes), or pi's maxTokens field (display-only on this provider).

Happy to share the replay harness, the captured prompt corpus (scrubbed), and the parser patch.

---

Earlier revision of this post recommended "leave MTP on" based on a 24-cell file-exploration rebench. That recommendation is withdrawn; see section 1.

[noonghunna]

Thanks @mgabor3141 — exceptionally well-measured, and it lines up with what we've seen while sharpening it.

#3 (qwen3_xml over qwen3_coder): fully aligned — same conclusion as our #145 / #72 (qwen3_coder + --reasoning-parser qwen3 leaks tool XML into delta.content). Your replay data is strong input for the open question of promoting qwen3_xml to the agent default on our dual composes.

#1 (MTP amplifies the CodeAct attractor): matches our cautious MTP stance, and the "MTP correctness is prompt-shape dependent — can't sample the whole surface per workload" framing is exactly right. One nuance from current testing: this is specific to the built-in MTP head, which shares the target's distribution, so at template-token ambiguity positions it amplifies whatever the main head already leans toward. An external / cross-attention drafter (e.g. DFlash — a separate small model reading hidden states, not the same head) is architecturally different and might not amplify the same attractor. We just got DFlash running on Qwen3.6-27B (it even sidesteps the DeltaNet KV-rollback block that kills built-in spec-dec there), but haven't measured tool-call grammar correctness under it yet — flagging it as a possibly-orthogonal path to test, not a recommendation.

#2 (qwen3_xml multi-function streaming bug): new to us — thanks for filing vllm-project/vllm#43713 + vllm-project/vllm#43714. It's a bug in the parser we recommend, so it matters directly; happy to help validate the fix on Ampere (sm_86, 2×3090).

Yes please to the scrubbed prompt corpus + replay harness — we'll cross-check on our rig and fold the attractor + multi-function shapes into our regression suite.

[mgabor3141]

Thanks @noonghunna, four things.

1. Built-in vs external drafter (#⁠1) — you're right. The mechanism I documented is specific to the built-in MTP head: it shares the target's distribution by construction, so at template-token ambiguity positions it amplifies whatever the main head already leans toward. The rejection sampler then accepts the wrong token because the main head also assigns it > threshold probability, and the trajectory commits to invalid grammar.

An external / cross-attention drafter does not share the target's distribution and need not amplify the same bias. Whether one actually doesn't is empirical, and we have not measured it. So everywhere I wrote "MTP" in section 1, read "built-in MTP". DFlash on Qwen3.6-27B sidestepping the DeltaNet KV-rollback block is independently interesting; if you measure tool-call grammar under it I'd love to see numbers either way. Happy to send captured bodies for you to replay against your DFlash-enabled stack if that's faster than re-capturing on your side.

2. Repro corpus + harness — pushed to a fork branch rather than inlined here, so it's git-trackable and you can decide if and how to mainline:

mgabor3141/club-3090 → qwen3-xml-tool-calling-repro

models/qwen3.6-27b/vllm/diagnostics/qwen3-xml-tool-calling-repro/
  README.md
  corpus/
    user-shell-prompt.json            # the failing wire body, scrubbed
    user-shell-prompt-no-skills.json  # variant for matrix row 2/4
  harness/
    replay.mjs                        # N-replay + classifier, vanilla node 22+

Recipe to reproduce section 1's matrix on your rig:

node harness/replay.mjs corpus/user-shell-prompt.json 10
node harness/replay.mjs corpus/user-shell-prompt-no-skills.json 10
# then restart vLLM without --speculative-config and repeat

The harness classifies each response as CLEAN / GARBLED / PROSE / ERROR and dumps the full per-rep JSON, so you can also inspect whether the model picked the right tool (CLEAN with hallucinated tool name is something I'd want to see separately from CLEAN with correct dispatch). README in the branch covers gotchas (sampling preset, parser choice, what each bucket means precisely).

I picked the path because it sits alongside the existing models/qwen3.6-27b/vllm/diagnostics/structured-cot-bench.md, but it's a fork branch on my side — relocate, rename, restructure, or drop entirely as fits. PR not opened on purpose; happy to open one against whatever path you'd want it at, or to leave it where rig owners can git fetch it ad-hoc.

3. Parser fix on Ampere (#⁠2) — yes please, validation on sm_86 would be great. Two heads up while you're there: there's now a follow-up stacked PR on top of #43714, vllm-project/vllm#43783, which fixes a separate documented bug in the same parser: id was being re-emitted on every streamed DeltaToolCall, but per the OpenAI streaming protocol it should appear only on the first delta of each slot (subsequent deltas carry only index + function.arguments). That was the root cause of the ten streaming xfails the parser test file has been carrying.

With both patches applied the parser test suite is 26 passed, 0 xfailed, 0 failed (was 15 passed + 10 xfailed under #43714 alone). #43783 is a draft against #43714 so they need to be tested as a stack. Both touch only Python (no kernels), so the patch is platform-agnostic in principle, but a real sm_86 run catches anything I missed.

4. Folding into your regression suite — happy to keep extending the corpus as new failure shapes turn up. If you want a specific shape for the suite (e.g. an N=3 multi-function-per-tool_call body that exercises both fixes, or a chain that drives _merge_new_deltas_to_single_response's index-matching), I can capture and scrub one against the same stack.

[noonghunna]

Ran your harness on our DFlash stack, @mgabor3141 — the external drafter does not amplify the attractor. Full 2×2 at N=10, your replay.mjs + corpus unchanged:

Cell	beellama + DFlash	(your vLLM + MTP=3)
spec-dec ON, with skills	10/10 CLEAN	0/10
spec-dec ON, no skills	10/10 CLEAN	9/10
spec-dec OFF, with skills	10/10 CLEAN	10/10
spec-dec OFF, no skills	10/10 CLEAN	10/10

DFlash-on == DFlash-off == grammar-clean (40/40, 0 GARBLED) — the spec-dec layer adds no corruption, right where the built-in MTP head catastrophically did on yours. Consistent with your section-1 mechanism: an external cross-attention drafter doesn't share the target's distribution, so it doesn't amplify the CodeAct attractor at template-ambiguity positions.

Three honest caveats on comparability:

1. Different engine + parser. This is beellama.cpp (a llama.cpp fork) with --jinja tool-parsing off the Qwen GGUF chat template — not vLLM + qwen3_xml. So it's the analogous test on our DFlash stack, not a literal repro of your matrix. Your classifier's tag-shaped-content leak signal is parser-independent, though, and it stayed at 0 across all 40.
2. We can't run the MTP arm on our stack for an in-stack contrast — built-in MTP on Qwen3-Next is blocked by the DeltaNet KV-rollback issue (which is exactly why we run DFlash). So MTP→0/10 stays your vLLM datapoint and DFlash→10/10 is ours; together they're the on-thesis pair.
3. One "CLEAN-with-odd-values" quirk for your inspect-the-dumps caveat: the github calls parse as valid JSON (hence CLEAN) but the action field comes back over-quoted — "action":"\"pr_overview\"" — in most reps. It's identical on DFlash-off, so it's a template/schema artifact, not spec-dec corruption. Flagging since you explicitly separate odd-dispatch from clean-dispatch.

Net for us: the external drafter breaks the speed-vs-reliability tradeoff your section 1 documents — DFlash spec-dec speedup without the tool-grammar penalty. Directly actionable for our Qwen agent serving.

On the other three:

• (2) harness — please open the PR at models/qwen3.6-27b/vllm/diagnostics/qwen3-xml-tool-calling-repro/ (right where you placed it, next to structured-cot-bench.md). It ran clean straight off your branch — node 20 was fine, fwiw (only fetch/fs). Happy to add our DFlash rows to its README once it lands.
• (3) parser stack #43714 + #43783 — yes, we'll validate the stack on sm_86 (2×3090) and report back; thanks for the #43783 heads-up + the id-first-delta root cause. Will test them stacked, not standalone.
• (4) more shapes — yes please, the N=3 multi-function-per-tool_call body is the one we'd want next (also exercises both parser fixes). Our per-rep dumps are here if you want to see the DFlash outputs.