v1.7.0

🚀 Shepherd Model Gateway v1.7.0 Released

This release leads with a correctness milestone. SMG runs the correctness-critical path — tokenization, function-call parsing, and reasoning parsing — natively in Rust, and that path is now proven on par with vLLM on tool-calling quality, validated by a nightly BFCL A/B harness. Plus new PD and engine-load metrics, GLM-5.x support, custom routing keys, and a major dependency sweep.

🎯 Rust-Native Correctness, Validated Against vLLM

SMG doesn't pass requests through — it owns the correctness-critical path in Rust. Tokenization, function-call parsing, and reasoning parsing all run natively in Rust rather than deferring to the engine's Python. That's where the gateway's performance and stability come from: no Python interpreter on the hot path, no GIL contention, no unhandled exceptions mid-stream.

Reimplementing that path in Rust demands one thing above all: it has to stay faithful to the reference. So we built a nightly BFCL (Berkeley Function-Calling Leaderboard) A/B harness that runs the same evaluation twice — once with SMG fronting the engine, once against pure vLLM — every night, across six major open-source models:

• DeepSeek V4
• GLM-5.2
• gpt-oss
• Kimi K2.6
• MiniMax M2.7
• Qwen3.6

Why this matters: The native-Rust critical path gives you faster, more stable function calling and reasoning — and these results prove it does so without giving up an inch of correctness. We've been closing the gaps model by model, and the latest run puts SMG on par with vLLM across the board. This isn't a one-time benchmark; it's a nightly regression gate. Correctness is something we commit to and continuously verify.

📊 New Metrics: PD Disaggregation & Engine Load

Deeper operational visibility into prefill-decode and backend load:

• smg_pd_* metrics — Dedicated PD disaggregation metrics for monitoring prefill/decode split behavior
• smg_engine_* gauges — Engine load signals re-exported as first-class gauges
• SGLang reasoning token usage — Reasoning token counts now surfaced in usage stats

Impact: PD deployments get purpose-built metrics for the disaggregated serving path, and engine load is now visible through standard gauges — better autoscaling signals and clearer production dashboards.

🎯 Custom Routing Key on Any Policy

X-SMG-Routing-Key header now honored across all routing policies:

• Force routing decisions with an explicit key regardless of the active policy
• Pin related requests to the same worker for session affinity or cache locality
• Works with cache-aware, load-based, and every other policy

Impact: Application-level control over routing without changing your policy. Send the same routing key for a conversation or tenant and keep requests landing on the same worker.

🔧 Tool-Call Parsing Hardening

Substantial robustness work across model families:

• MiniMax-M2 — Parse all parallel invokes within a wrapper
• qwen_xml and glm4_moe — Coerce XML tool-call args by declared schema type
• gpt-oss (Harmony) — Emit parallel tool calls on the Chat API
• OpenAI-compat — Return content: null instead of whitespace when emitting tool calls

Impact: More reliable tool calling across MiniMax, Qwen, GLM, and gpt-oss — correct argument typing, proper parallel-call handling, and OpenAI-compatible response shapes. This hardening is exactly what the BFCL parity above measures.

📦 Major Dependency Modernization

Stack-wide upgrade sweep, including security-sensitive dependencies:

• Security/crypto: jsonwebtoken 9.3→10.4, sha2 0.10→0.11
• Networking: reqwest 0.12→0.13, redis 0.27→1.2 (+ deadpool)
• Observability: opentelemetry 0.27→0.32 stack
• Tokenization: tiktoken-rs 0.9→0.12
• MCP: rmcp 0.8→1.7
• WASM: wasmtime 41→45
• WebRTC: str0m 0.19→0.20
• Core: rand 0.9→0.10, schemars 0.8→1.2
• Raised compatible dependency floors across the board

⚡ Performance

• TokenSpeed multimodal tensor transport — Optimized multimodal tensor handling (contributed by NVIDIA, thanks @yechank-nvidia)
• Cache env lookups off the request hot path — No repeated environment lookups per request

🐛 Notable Fixes

• Protocols: Accept min_tokens=0 in OpenAI requests
• gRPC: Drop stale WatchLoadUpdateReq piggyback, SGLang floor bumped to 0.5.13
• Packaging: Include LICENSE files in setuptools/maturin Python packages
• Multimodal: Gate shm-namespace test to Linux

🧪 Testing & CI

• Multi-model BFCL matrix — Nightly A/B on per-model runners across DeepSeek V4, GLM-5.2, gpt-oss, Kimi K2.6, MiniMax M2.7, and Qwen3.6
• Hermetic test images — Multimodal tests serve images from the repo instead of picsum.photos
• TRT-LLM startup — Pin Open MPI to loopback to fix flaky 1-GPU startup

🏗️ Infrastructure

• aarch64 wheels — Built on manylinux_2_28 for ARM compatibility
• Hermetic PyPI upload — Metadata 2.4 compliance via hermetic Python

🙏 Welcome New Contributors

4 first-time contributors landed in this release — thank you all!

Full Changelog: v1.6.0...v1.7.0

Upgrade now: pip install smg --upgrade

🐑 Rust-native correctness. Production-proven.

What's Changed

• feat(bfcl): multi-model nightly A/B matrix on per-model runners by @key4ng in #1791
• feat(metrics): add smg_pd_* PD disaggregation metrics by @slin1237 in #1781
• feat(observability): re-export engine load signals as smg_engine_* gauges by @slin1237 in #1780
• feat(grpc): add sglang reasoning token usage by @Moersity in #1747
• test(e2e): serve multimodal test images from repo, not picsum.photos by @gongwei-130 in #1801
• refactor(grpc): reduce proto-change friction in the Go bindings by @slin1237 in #1800
• perf: cache env lookups off request hot paths by @gongwei-130 in #1799
• chore(deps): bump tiktoken-rs 0.9->0.12 by @slin1237 in #1807
• chore(deps): bump rmcp 0.8 -> 1.7 by @slin1237 in #1814
• chore(deps): bump redis 0.27->1.2 (+ deadpool 0.13, deadpool-redis 0.23) by @slin1237 in #1805
• chore(deps): bump sha2 0.10 -> 0.11 by @slin1237 in #1804
• chore(deps): bump opentelemetry 0.27->0.32 stack by @slin1237 in #1813
• build(deps): bump rand 0.9 -> 0.10 by @slin1237 in #1808
• chore(deps): bump schemars 0.8 -> 1.2 by @slin1237 in #1812
• chore(deps): bump wasmtime and wasmtime-wasi 41->45 by @slin1237 in #1806
• chore(deps): bump str0m 0.19 -> 0.20 by @slin1237 in #1811
• build(deps): raise compatible dependency floors by @slin1237 in #1809
• build(deps): bump reqwest 0.12 -> 0.13 by @slin1237 in #1803
• chore(deps): bump jsonwebtoken 9.3 -> 10.4 by @slin1237 in #1810
• perf(tokenspeed): optimize multimodal tensor transport by @yechank-nvidia in #1604
• fix(bfcl): max_model_len 32768 + OpenAICompletionsHandler for multi-turn FC by @key4ng in #1819
• test(e2e): don't assert exact image count for duplicate images by @slin1237 in #1821
• chore(ci): disable BFCL PR trigger, make nightly benchmark weekly by @key4ng in #1832
• fix(e2e): pin trtllm Open MPI to loopback to fix flaky 1-GPU startup by @key4ng in #1823
• fix(tool_parser): parse all parallel invokes in a minimax_m2 wrapper by @key4ng in #1824
• chore(deps): bump actions/cache from 5 to 6 by @dependabot[bot] in #1826
• chore(deps): update opencv requirement from 0.98.2 to 0.99.0 by @dependabot[bot] in #1828
• chore(deps): bump actions/checkout from 6 to 7 by @dependabot[bot] in #1825
• ci(benchmarks): bump genai-bench timeout 480s -> 600s by @key4ng in #1838
• ci(bfcl): add GLM-5.2-FP8 nightly leg (TP=8 sequential, whole node) by @key4ng in #1834
• feat(parser): route GLM-5.x to existing GLM-4.7/4.5 parsers by @key4ng in #1833
• ci(bfcl): set max_model_len=auto and capture both arms' transcripts by @key4ng in #1840
• fix(grpc): return content:null instead of whitespace when emitting tool calls (OpenAI-compat) by @key4ng in #1822
• fix(tokenizer): serialize DeepSeek V3.2/V4 prompt JSON with json.dumps spacing by @key4ng in #1839
• fix(tool_parser): coerce XML tool-call args by declared schema type (qwen_xml, glm4_moe) by @key4ng in #1841
• fix(tokenizer): render object-form tool-call arguments in DeepSeek V3.2/V4 encoders by @key4ng in #1844
• feat(harmony): emit parallel tool calls for gpt-oss on the Chat API by @key4ng in #1845
• fix(harmony): revert no-op <|call|> stop filter for gpt-oss (#1845) by @key4ng in #1846
• fix(multimodal): gate shm-namespace test to Linux by @slin1237 in #1849
• feat(routing): honor X-SMG-Routing-Key on any policy (#1843) by @slin1237 in #1848
• fix: Include LICENSE files in setuptools-/maturin-based Python packages by @xhochy in #1847
• fix(protocols): accept min_tokens=0 in OpenAI requests by @Azure99 in #1851
• fix(grpc): drop stale WatchLoadUpdateReq piggyback, bump sglang floor to 0.5.13 by @junliu-mde in #1850
• chore: bump versions for v1.7.0 release by @slin1237 in #1854
• fix(ci): build aarch64 wheels on manylinux_2_28 by @slin1237 in #1855
• fix(ci): use hermetic Python for PyPI upload (Metadata 2.4) by @slin1237 in #1856

New Contributors

• @Moersity made their first contribution in #1747
• @xhochy made their first contribution in #1847
• @Azure99 made their first contribution in #1851
• @junliu-mde made their first contribution in #1850

Full Changelog: v1.6.0...v1.7.0