feat: add renderers package

[hallerite]

Summary

Adds packages/renderers/ — a standalone package that owns message ↔ token conversion as an alternative to vLLM's Jinja chat templates and the existing TITO/MITO server-side machinery. Every sampled assistant turn keeps its exact tokens across the rollout boundary; the trainer sees bit-for-bit what vLLM produced.

Status: experimental. Available alongside the production openai_chat_completions_token (TITO) client, which remains the recommended path for production training. Renderers offer stronger token-preservation guarantees but only ship hand-coded support for a subset of models, and corner cases are still being shaken out. The package is text-only — Qwen3VLRenderer works against the Qwen3-VL tokenizer for text conversations only; multimodal training should continue to use MITO.

Full design, motivation, and examples of the failure modes this fixes: packages/renderers/README.md.

Renderer matrix

Template family	Renderer class	Models
Qwen chatml	Qwen3Renderer / Qwen35Renderer / Qwen36Renderer / Qwen3VLRenderer	Qwen3, Qwen3.5, Qwen3.6, Qwen3-VL (text-only)
GLM (next-turn markers)	GLM5Renderer / GLM45Renderer	GLM-5, GLM-5.1, GLM-4.5, GLM-4.7
MiniMax chatml-variant	MiniMaxM2Renderer	MiniMax-M2 / M2.5
Kimi im-role format	KimiK2Renderer / KimiK25Renderer	Kimi K2, K2.5, K2.6
DeepSeek V3 tool-section	DeepSeekV3Renderer	DeepSeek V3
Nemotron chatml + XML tools	Nemotron3Renderer	Nemotron-3 Nano / Super
OpenAI harmony	GptOssRenderer	GPT-OSS
Fallback (apply_chat_template)	DefaultRenderer	anything else, with optional tool_parser / reasoning_parser

Architecture

messages → renderer.render_ids() → [token IDs]
                                       ↓
                              vLLM /v1/generate
                                       ↓
[completion IDs] → renderer.parse_response() → ParsedResponse(content, reasoning_content, tool_calls)

The Renderer Protocol:

• render() / render_ids() — messages → tokens with per-token message attribution for loss masking.
• parse_response() — completion tokens → structured message via token-ID boundary scanning (no regex on decoded text).
• get_stop_token_ids() — turn-close tokens.
• bridge_to_next_turn() — extends prev_prompt + prev_completion with the new turn's tokens; returns None if the renderer can't prove prefix-stability (caller falls back to fresh render).

Key design decisions

• Per-renderer bridges, hand-coded. No shared chatml_bridge / glm_bridge helper — that approach rendered [dummy_assistant, *new_messages] and diffed against [dummy_assistant] to extract extension tokens, which broke on templates that treated the dummy as an invalid prefix (GLM-5.1 wraps the last assistant with empty <think></think>, harmony's assistant uses different channels historical vs latest, Kimi auto-injects a default system). Each renderer's bridge now hand-emits the new-turn tokens by calling the same per-role inline helpers that render() uses, so the two paths can't silently diverge. Two small shared primitives remain: trim_to_turn_close (scan prev_completion for a template-specific close token; on truncation, append the canonical close so the bridge still extends) and reject_assistant_in_extension (bridges refuse to re-tokenize model-sampled assistant content).

• Truncation handling. Hand-coded renderers always synthesize the canonical turn-close (<|im_end|>, <|endoftext|>, harmony's <|end|>, …) when vLLM hits max_tokens and the prior completion has no close token, so the next prompt still extends the prior step's tokens verbatim; the synthetic token lands in the merged sample's prompt_ids (mask=False) and never enters loss or KL. DefaultRenderer.bridge_to_next_turn returns None unconditionally — it wraps an unknown Jinja template and can't prove the extension contract holds — so the caller falls back to a fresh re-render.

• Pluggable parsers for DefaultRenderer. Hand-coded renderers bake parsing in. DefaultRenderer takes optional tool_parser= / reasoning_parser= kwargs wired to registries in renderers.parsers. Built-ins today: qwen3, qwen3.5, glm, deepseek_v3 for tools; think for reasoning.

RendererClient + /v1/generate

Adds RendererClient (verifiers client type "renderer") — renders messages client-side, POSTs raw token IDs to vLLM's /v1/generate, parses completions back into structured responses. Multi-turn rollouts reuse the prior step's exact tokens through bridge_to_next_turn; no re-rendering of sampled content.

A RendererPool offloads sync tokenization to threads so concurrent rollouts tokenize in parallel instead of blocking the event loop.

Test plan

• packages/renderers/tests/test_render_ids.py — multi-model parity matrix vs apply_chat_template (1 documented xfail for an upstream Jinja bug on Qwen3-VL content=None).
• packages/renderers/tests/test_roundtrip.py — render → parse round-trip per renderer: content, reasoning, single and multiple tool-calls.
• packages/renderers/tests/test_bridge.py — bridge contract invariants per hand-coded renderer: extends prev verbatim, rejects assistant-role extension, synthesizes close on truncation, extension contains the new-message content.
• packages/renderers/tests/test_incremental.py — unit coverage of trim_to_turn_close + reject_assistant_in_extension edge cases.
• packages/renderers/tests/test_parsers.py / test_parse_response.py / test_parse_response_robustness.py — parsing on truncated / malformed output; includes regression test for parse_qwen3 JSON-decode-error fallback.
• tests/test_renderer_e2e.py — end-to-end TITO rollout with scripted vLLM; asserts token preservation and multi-turn bridge extension.
• E2E in prime-rl #2278: wordle multi-turn samples_per_rollout = 1.00, reward + KL track main. Qwen3.5-35B-A3B + mini-swe-agent-plus: 0 breaks vs main's 32 in the same step (see README §4 for the concrete break modes).

🤖 Generated with Claude Code

---

Note

Medium Risk
Large, mostly additive change that introduces new tokenization/parsing/bridging logic and a new dependency; errors here can subtly affect rollout correctness and RL training data integrity, though existing client paths remain available.

Overview
Introduces a new standalone packages/renderers package that implements a Renderer protocol, per-model message→token renderers (Qwen/GLM/Kimi/DeepSeek/MiniMax/Nemotron/GPT-OSS) with bridge_to_next_turn support, and a fallback DefaultRenderer that wraps tokenizer.apply_chat_template with optional tool/reasoning parsers.

Adds a renderer-backed inference path (renderers.client.completions_request) that sends prompt_token_ids to vLLM’s /generate endpoint, parses completion token IDs back into structured outputs, and supports parallel tokenization via RendererPool.

Updates docs (evaluation.md, training.md, faqs.md, reference.md) to document the new renderer client_type, expand client type listings/descriptions, and clarify RL training tradeoffs between MITO/TITO/renderer approaches.

^{Reviewed by Cursor Bugbot for commit ee6cdb5. Bugbot is set up for automated code reviews on this repo. Configure here.}

[mikasenghaas]

hm in wondering if we should bundle those somehow? so like a discriminated union of client types with some shared args but mostly disjoint

[mikasenghaas]

LFGGG

[mikasenghaas]

what sort of guarantees, if any, can we give for this fallback? what are the pros and cons compare to fallback to mito?

[hallerite]

yeah, let me be more clear about this

[mikasenghaas]

legendary

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e2ede3f. Configure here.}