vllm updates to keep an eye on

[troymroberts]

These merges have helped me a low vis a vis multi turn agentic workflows in gemma. I haven't tried them in Qwen but presumably they are equally helpful.

It completely replaces the Qwen and Gemma parsers.

It had been instrumental in getting my workflow to work reliably in Gemma and it's something you should be def keeping your eye on.

"Introduce a declarative, config-driven streaming parser engine that replaces hand-rolled per-model streaming logic with a shared state machine. The engine handles reasoning/content/tool-call transitions, incremental JSON argument extraction, and detokenizer hold-back recovery through a single ParserEngineConfig."

"Migrate Gemma4 from separate hand-coded reasoning and tool parsers to the unified ParserEngine framework introduced for Qwen3. A single state machine in vllm/parser/gemma4.py now handles both channel-based reasoning extraction and custom tool call parsing with declarative configuration. The streaming engine parser adapters are registered under the existing gemma4 name so no user-facing configuration changes are needed.

This cleans up a multitude of issues with Gemma4 models (including DiffusionGemma) when used with spec decoding, stream_interval > 1, and general handling of reasoning / tool call boundaries, especially in streaming scenarios. The biggest improvement will be felt by users in scenarios like tool call heavy streaming code assistants (Claude Code, Codex CLI, OpenCode, Pi, etc) using spec decoding (such as --speculative-config '{"model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":4}' and similar shown in model cards and recipes.vllm.ai). This also makes DiffusionGemma quite usable, as it natively emits multiple tokens at once - similar to if the regular models emitted 256 speculative tokens from a parsing standpoint."

vllm-project/vllm#45413

vllm-project/vllm#45588

[noonghunna]

Great catch, @troymroberts! We've been tracking both #45413 and #45588 closely — just added them to our UPSTREAM.md tracker.

The good news: both merged 2026-06-15 and will ship in v0.24.0 (missed the v0.23.0 tag window). The declarative ParserEngine is exactly what we need to retire our hand-rolled gemma4_tool_parser.py overlay and the qwen3coder_tool_parser_deferred_commit.py sidecar (#72 — prose tool_call SSE-silence).

The caveat: steamEngineer tested #45588 on DiffusionGemma (PR #443 ablation) and found a streaming regression — raw <|tool_call>...<tool_call|> tokens leak into content under Hermes block-canvas SSE chunking. Non-streaming tool_calls are clean, but streaming tool extraction breaks. We kept legacy mounts for now. Full ablation is documented in the patch_docs/ directory of that PR.

Adoption plan for v0.24.0: boot new image on a test branch → run stream-toolcall-probe.py for Qwen3.6-27B (check #39598 residual) → run DiffusionGemma streaming tool replay → if clean, drop vendored overlays and update composes; if broken, keep overlays and file upstream issue.

Thanks for flagging this!

[tomByrer]

Speaking of Gemma @troymroberts; have you tried this retrain/finetune yet?
https://huggingface.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF
This is v2 of the #1 HF model.

Roughly 3.5× higher than the base model on technical-agentic tasks

Maybe pair it with VibeThinker 3B for deeper reasoning?

[troymroberts]

I tend not to use fine tunes like the gemma model above, and gguf makes it a non starter for me. Ultimately whatever model i settle on needs to be served with high concurrency soo vllm at the end of the day (also giving up prompt caching in vllm for the same multi turn use case is hard lol)

And Vibe thinker has no tool support so that's out (and i'm super skeptical in general of big benchmark scores on small models, hart to not think they're benchmaxxed)