V1.8.0

[slin1237]

🚀 Shepherd Model Gateway v1.8.0 Released

Our biggest multimodal release yet: EPD (Encode-Prefill-Decode) disaggregation, engine-agnostic tensor transport over SHM and RDMA, and audio support (Qwen3 ASR + Omni) — plus dedicated multimodal + audio support for Thinking Machines' Inkling model, per-tenant API keys, LoRA adapter management, and a second nightly correctness harness.

⚠️ Breaking change: The legacy smg server command has been removed. Use smg launch or smg serve instead. Update your deployment scripts before upgrading.

🧩 EPD: Encode-Prefill-Decode Disaggregation

Multimodal serving now disaggregates the encode stage from prefill and decode.

Just as PD disaggregation separates prefill from decode, EPD adds a third independently-scheduled stage — encode — for multimodal inputs. The vision/audio encoder runs on its own workers, so image, video, and audio preprocessing no longer competes with the prefill/decode path:

• Dedicated encode routing — Multimodal inputs routed to encode workers, encoded tensors forwarded to prefill
• Unified Mode-parameterized router — Regular, PD, and EPD collapsed into a single router with a shared EncodeStage, rather than three separate code paths
• Python CLI + Helm exposure — Configure EPD directly from the CLI
• EPD smoke CI — 4-GPU H100 multimodal coverage in the nightly matrix

Impact: Encode-heavy workloads (lots of images or video) scale independently. You can provision encode capacity separately from prefill/decode, and a burst of multimodal requests won't starve your text generation throughput.

🚚 Engine-Agnostic Tensor Transport: SHM + RDMA

Encoded multimodal tensors now move between workers over a configurable, engine-neutral transport layer.

Moving preprocessed tensors from encode workers to prefill workers efficiently is the hard part of EPD. This release generalizes that transport across engines and interconnects:

• Configurable tensor transport — Engine-agnostic interface, not hardwired per backend
• vLLM SHM transport — Shared-memory tensor handoff for same-node encode→prefill
• RDMA pixel transport — New mm_rdma crate for cross-node tensor movement, routed through a shared payload resolver
• Shared proto handles — ShmHandle and RemoteTensorHandle hoisted to common.proto for reuse across backends
• Centralized backend×modality capability check — One place that knows which engine supports which modality over which transport

Impact: Multimodal disaggregation works the same way whether tensors move over shared memory on one node or RDMA across nodes, and the same transport layer serves vLLM, TokenSpeed, and future backends.

🎙️ Audio Support: Qwen3 ASR + Omni

SMG now handles audio — the modality lineup is text, image, video, and audio.

• Qwen3 ASR — Automatic speech recognition
• Qwen3 Omni — Full multimodal (text + vision + audio)
• Audio-processor selection on ModelProcessorSpec — Per-model audio preprocessing
• Verified correct — Full-tensor parity test for Qwen3 audio log-Mel features against HuggingFace transformers

Impact: Speech and omni-modal models run through the same gateway, the same routing, and the same APIs as everything else — with audio preprocessing validated bit-for-bit against the reference.

🧠 Dedicated Support: Thinking Machines' Inkling

SMG adds first-class, end-to-end support for Inkling — Thinking Machines' multimodal model — including its full modality set.

• End-to-end integration — Complete serving-path support, from request handling through response assembly
• Multimodality (image) — Image input support, with multimodal placeholders aligned to the Inkling checkpoint template
• Audio — Audio input support, built on the same audio pipeline landing in this release

Impact: Inkling runs through SMG with dedicated handling across text, image, and audio — the same routing, caching, tokenization, and API surface as every other supported model, purpose-fit to Inkling's checkpoint. Thinking Machines models get a first-class home on SMG.

🎨 Multimodal Performance (contributed by NVIDIA)

Encoder and preprocessing optimizations from @yechank-nvidia:

• Reduced TokenSpeed encoder input transport overhead
• Optimized Qwen vision preprocessing
• Optimized OpenCV video decoding and thread allocation

Impact: Lower latency across vision and video workloads. Thanks again to @yechank-nvidia for continued multimodal contributions.

✅ Correctness: Second Nightly Harness (tau2-bench)

Building on the v1.7.0 BFCL parity work, we've added a second correctness gate.

• tau2-bench nightly A/B — Parser verification against pure vLLM, complementing BFCL
• Alternating schedule — BFCL runs Mon/Wed/Fri, tau2-bench runs Tue/Thu/Sat
• Hardened runs — Fail-fast timeouts and quieter logs

Impact: Two independent nightly harnesses now verify SMG's Rust-native tokenization, tool-call parsing, and reasoning parsing stay faithful to the reference. We're closing correctness gaps continuously, not once — and now double-checking with two different benchmarks.

🔐 Per-Tenant API Keys

Serving-path authentication now supports per-tenant API keys:

• Issue distinct API keys per tenant
• Authenticate serving requests against tenant-scoped keys

Impact: Multi-tenant deployments get proper key isolation on the serving path — each tenant authenticates with its own credentials.

🔌 LoRA Adapter Management

Dynamic LoRA adapter RPCs exposed in the SGLang scheduler proto:

• Load, unload, and list LoRA adapters at runtime
• No restart required to swap adapters

Impact: Serve and rotate fine-tuned LoRA adapters on the fly — load a new adapter, list what's active, unload what's stale, all through gRPC.

🔎 Local Model Discovery

• Discover local HTTP model IDs from /v1/models — Auto-populate model IDs from HTTP-mode workers instead of manual configuration

🧠 Thinking / Reasoning Resolution

Cleaner, layer-unified control over reasoning activation:

• reasoning_effort "none"/"minimal" → enable_thinking=false — Honor OpenAI-style effort hints on the gRPC chat path
• Unified thinking resolution by layer — One consistent resolution path instead of scattered logic
• Kimi K2.6 reasoning parser fix — Corrected kimi_k25 → kimi_thinking parser mapping

🤖 New Parser Support

• Sarashina — New tool-call parser

🎧 Realtime API on HTTP Router

Realtime API support added to the HTTP router.

🐛 Notable Fixes

• HTTP: Set application/json Content-Type for PD non-streaming responses
• PD: Only treat line-anchored data: [DONE] as stream terminator, route chat using full message history
• Worker: Query engine-specific HTTP load endpoint for vLLM/SGLang
• Tool parser: Treat qwen_xml tool-call arg values literally
• Data connector: Decode Postgres JSON columns as Value, not String
• gRPC: GetTokenizer RPC implemented for TokenSpeed backend

📦 Dependencies

• nixl-sys 0.10.1→1.3.1, str0m 0.20→0.21, lz4_flex 0.13→0.14, wasm-encoder 0.252→0.253

🏗️ Infrastructure

• TokenSpeed engine image — New release workflow for TokenSpeed Docker images
• Project governance docs — Added governance documentation
• CI fixes for SGLang e2e (protobuf gencode), MLX (transformers 5.13), and TokenSpeed (cutlass-dsl pin)

🙏 Welcome New Contributors

5 first-time contributors landed in this release — thank you all!

Full Changelog: v1.7.0...v1.8.0

Upgrade now: pip install smg --upgrade

🐑 Any modality. Any engine. Production-proven.

What's Changed

• fix(http): set application/json Content-Type for PD non-streaming responses by @chengxumiaodaren in fix(http): set application/json Content-Type for PD non-streaming responses #1857
• chore(deps): update str0m requirement from 0.20 to 0.21 by @dependabot[bot] in chore(deps): update str0m requirement from 0.20 to 0.21 #1859
• feat(tau2): nightly tau2-bench A/B for parser verification by @key4ng in feat(tau2): nightly tau2-bench A/B for parser verification #1864
• ci: alternate BFCL (Mon/Wed/Fri) and tau2 (Tue/Thu/Sat) nightlies by @key4ng in ci: alternate BFCL (Mon/Wed/Fri) and tau2 (Tue/Thu/Sat) nightlies #1866
• fix(ci): kimi-k2.6 SMG reasoning parser (kimi_k25 → kimi_thinking) by @key4ng in fix(ci): kimi-k2.6 SMG reasoning parser (kimi_k25 → kimi_thinking) #1868
• feat(realtime): support the Realtime API on the HTTP router by @slin1237 in feat(realtime): support the Realtime API on the HTTP router #1870
• fix(ci): pin nvidia-cutlass-dsl 4.5.2 for TokenSpeed (uv + build subprocess pip) by @key4ng in fix(ci): pin nvidia-cutlass-dsl 4.5.2 for TokenSpeed (uv + build subprocess pip) #1869
• fix(tau2): harden nightly — fail-fast timeouts + silence cost-lookup log spam by @key4ng in fix(tau2): harden nightly — fail-fast timeouts + silence cost-lookup log spam #1872
• fix(worker): query engine-specific HTTP load endpoint for vLLM/SGLang by @XinyueZhang369 in fix(worker): query engine-specific HTTP load endpoint for vLLM/SGLang #1867
• fix(pd): only treat line-anchored data: [DONE] as stream terminator by @junliu-mde in fix(pd): only treat line-anchored data: [DONE] as stream terminator #1878
• fix(ci): unbreak sglang/e2e (protobuf gencode) and MLX (transformers 5.13) jobs by @key4ng in fix(ci): unbreak sglang/e2e (protobuf gencode) and MLX (transformers 5.13) jobs #1880
• fix(grpc/chat): honor reasoning_effort "none"/"minimal" as enable_thinking=false by @qywu in fix(grpc/chat): honor reasoning_effort "none"/"minimal" as enable_thinking=false #1876
• refactor(grpc/chat): unify thinking resolution by layer by @slin1237 in refactor(grpc/chat): unify thinking resolution by layer #1882
• feat(discovery): discover local HTTP model ids from /v1/models by @SpencerGarnets in feat(discovery): discover local HTTP model ids from /v1/models #1793
• perf(multimodal): reduce TokenSpeed encoder input transport overhead by @yechank-nvidia in perf(multimodal): reduce TokenSpeed encoder input transport overhead #1879
• perf(multimodal): Optimize Qwen vision preprocessing by @yechank-nvidia in perf(multimodal): Optimize Qwen vision preprocessing #1875
• perf(multimdoal): Optimize OpenCV video decoding and thread allocation by @yechank-nvidia in perf(multimdoal): Optimize OpenCV video decoding and thread allocation #1865
• docs: add project governance documentation by @slin1237 in docs: add project governance documentation #1883
• feat(multimodal): add EPD encode routing by @chenht2022 in feat(multimodal): add EPD encode routing #1852
• fix(import): Fix TokenSpeed KV events import by @lightseek-bot in fix(import): Fix TokenSpeed KV events import #1884
• chore(deps): update wasm-encoder requirement from 0.252 to 0.253 by @dependabot[bot] in chore(deps): update wasm-encoder requirement from 0.252 to 0.253 #1886
• fix(multimodal): expose EPD routing in Python CLI by @chenht2022 in fix(multimodal): expose EPD routing in Python CLI #1889
• refactor(multimodal): split grpc multimodal.rs into a module directory by @slin1237 in refactor(multimodal): split grpc multimodal.rs into a module directory #1890
• refactor(proto): hoist ShmHandle/RemoteTensorHandle to common.proto by @slin1237 in refactor(proto): hoist ShmHandle/RemoteTensorHandle to common.proto #1891
• feat(multimodal): configurable engine-agnostic tensor transport by @slin1237 in feat(multimodal): configurable engine-agnostic tensor transport #1892
• feat(multimodal): vLLM SHM tensor transport by @slin1237 in feat(multimodal): vLLM SHM tensor transport #1893
• feat(multimodal): optimize EPD encode routing by @chenht2022 in feat(multimodal): optimize EPD encode routing #1853
• feat(multimodal): vLLM video via shared Modality enum by @slin1237 in feat(multimodal): vLLM video via shared Modality enum #1895
• chore(deps): Remove legacy smg server command by @lightseek-bot in chore(deps): Remove legacy smg server command #1897
• fix(tool_parser): treat qwen_xml tool-call arg values literally by @key4ng in fix(tool_parser): treat qwen_xml tool-call arg values literally #1899
• fix(k8s): qualify agentic service image references by @key4ng in fix(k8s): qualify agentic service image references #1900
• feat(grpc): expose LoRA adapter load/unload/list RPCs in the SGLang scheduler proto by @yuanzh89 in feat(grpc): expose LoRA adapter load/unload/list RPCs in the SGLang scheduler proto #1901
• feat(multimodal): support Qwen3 ASR and Omni by @chenht2022 in feat(multimodal): support Qwen3 ASR and Omni #1905
• refactor(multimodal): extract engine-neutral RDMA pixel transport into crates/mm_rdma by @slin1237 in refactor(multimodal): extract engine-neutral RDMA pixel transport into crates/mm_rdma #1908
• fix(multimodal): address Qwen3 review findings by @lightseek-bot in fix(multimodal): address Qwen3 review findings #1907
• refactor(multimodal): move audio-processor selection onto ModelProcessorSpec by @slin1237 in refactor(multimodal): move audio-processor selection onto ModelProcessorSpec #1913
• refactor(grpc): centralize backend×modality capability check by @slin1237 in refactor(grpc): centralize backend×modality capability check #1914
• refactor(multimodal): route RDMA through the shared tensor payload resolver by @slin1237 in refactor(multimodal): route RDMA through the shared tensor payload resolver #1915
• test(multimodal): full-tensor parity for Qwen3 audio log-Mel vs transformers by @slin1237 in test(multimodal): full-tensor parity for Qwen3 audio log-Mel vs transformers #1910
• refactor(grpc): unify regular/PD/EPD into one Mode-parameterized router + EncodeStage by @slin1237 in refactor(grpc): unify regular/PD/EPD into one Mode-parameterized router + EncodeStage #1923
• chore(deps): update nixl-sys requirement from 0.10.1 to 1.3.1 by @dependabot[bot] in chore(deps): update nixl-sys requirement from 0.10.1 to 1.3.1 #1922
• feat(e2e): EPD multimodal smoke CI on 4-gpu-h100 by @key4ng in feat(e2e): EPD multimodal smoke CI on 4-gpu-h100 #1924
• chore(deps): update lz4_flex requirement from 0.13 to 0.14 by @dependabot[bot] in chore(deps): update lz4_flex requirement from 0.13 to 0.14 #1919
• ci(docker): add TokenSpeed engine image release workflow by @key4ng in ci(docker): add TokenSpeed engine image release workflow #1927
• feat(inkling): add end-to-end SMG support by @lightseek-bot in feat(inkling): add end-to-end SMG support #1926
• fix(pd): route chat using full message history by @Moersity in fix(pd): route chat using full message history #1929
• fix(inkling): align multimodal placeholders with checkpoint template by @chenht2022 in fix(inkling): align multimodal placeholders with checkpoint template #1931
• feat(grpc): implement GetTokenizer RPC for TokenSpeed backend by @key4ng in feat(grpc): implement GetTokenizer RPC for TokenSpeed backend #1934
• fix(e2e): wait for gateway readiness by @ChetanXpro in fix(e2e): wait for gateway readiness #1936
• feat(auth): add per-tenant API keys for serving-path authentication by @XinyueZhang369 in feat(auth): add per-tenant API keys for serving-path authentication #1932
• fix(data_connector): decode Postgres JSON columns as Value, not String by @XinyueZhang369 in fix(data_connector): decode Postgres JSON columns as Value, not String #1935
• feat(tool_parser): add Sarashina tool-call parser by @Juno13340 in feat(tool_parser): add Sarashina tool-call parser #1937
• chore: bump versions for v1.8.0 release by @slin1237 in chore: bump versions for v1.8.0 release #1938

New Contributors

• @chengxumiaodaren made their first contribution in fix(http): set application/json Content-Type for PD non-streaming responses #1857
• @SpencerGarnets made their first contribution in feat(discovery): discover local HTTP model ids from /v1/models #1793
• @yuanzh89 made their first contribution in feat(grpc): expose LoRA adapter load/unload/list RPCs in the SGLang scheduler proto #1901
• @ChetanXpro made their first contribution in fix(e2e): wait for gateway readiness #1936
• @Juno13340 made their first contribution in feat(tool_parser): add Sarashina tool-call parser #1937

Full Changelog: v1.7.0...v1.8.0

---

This discussion was created from the release V1.8.0.