v1.9.0

🚀 Shepherd Model Gateway v1.9.0 Released

A focused release headlined by Kimi K3 support and tenant rate limiting — extending the multi-tenancy foundation from v1.8.0 — plus Responses API in PD mode, batched completion prompts, model aliases, and an engine-version bump across all four backends.

🤖 Kimi K3 Support

Full end-to-end integration for Kimi K3 across text, vision, tools, and reasoning:

• Dedicated vision processor — K3 gets its own multimodal image processor, with correct modality ordering
• Tool calling — Forced tool calls correctly close the prefilled channel
• Thinking-effort defaults — Media wrapper and thinking-effort resolution handled at the prompt-encoding layer

🔀 Responses API in PD Mode

/v1/responses now serves under prefill-decode disaggregation:

• Full Responses API support in the PD gRPC pipeline
• Batched completion prompts supported in the pipeline

🏷️ Model Aliases

End-to-end model alias support:

• Refer to a model by an alias throughout the request path
• Aliases resolve consistently from API boundary to backend

🔍 Request ID Propagation

Backend request IDs now derived from the request ID (rid) and the middleware request ID:

• Correlate a request across the gateway→backend boundary
• Consistent IDs for tracing and debugging

🤖 Engine Version Bumps

All four backends updated:

• vLLM — Floor raised to 0.26, Docker matrix refreshed
• SGLang — Bumped to 0.5.16
• TensorRT-LLM — Bumped to 1.3.0rc22
• TokenSpeed — Bumped to latest

🔧 Engine Version Automation

New CI automation to keep pace with upstream engine releases:

• Engine version watch — Automated detection of new engine versions, running weekly
• Nightly failure triage — Automated triage of nightly benchmark failures

🐛 Notable Fixes

• Tool parser: Ignore <|END_ACTION|> inside Cohere JSON strings (no more false stops mid-argument)
• Protocols: Accept function tools without a parameters field
• MCP: Preserve explicit tool_choice for the initial responses request
• Gateway: Apply PUT /workers/{id} instead of failing in the background, reject schemeless worker URLs at the API boundary
• Gateway: Scrape engine metrics via base URL under --dp-aware
• Scheduler: Retain worker capacity tracking
• gRPC: Load Harmony encoding at gpt-oss worker registration

📦 Dependencies & Infrastructure

• Dependencies: tokio-tungstenite 0.29→0.30, wasm-encoder 0.253→0.254, plus GitHub Actions bumps (setup-python, setup-go, labeler, stale)
• CI: Repaired crate publication order, aligned TokenSpeed with the promoted EPD package
• Docs: Documented XGrammar and fixed its pipeline order in the gRPC pipeline docs

🙏 Welcome New Contributors

3 first-time contributors landed in this release — thank you all!

Full Changelog: v1.8.0...v1.9.0

Upgrade now: pip install smg --upgrade

🐑 Frontier models. Production-proven.

What's Changed

• fix(ci): repair crate publication order by @slin1237 in #1940
• fix(tool_parser): ignore <|END_ACTION|> inside Cohere JSON strings by @santhreal in #1941
• fix(tokenspeed): align with promoted EPD package by @lightseek-bot in #1944
• chore(deps): update wasm-encoder requirement from 0.253 to 0.254 by @dependabot[bot] in #1953
• chore(deps): bump actions/setup-python from 6 to 7 by @dependabot[bot] in #1950
• chore(deps): bump actions/setup-go from 6 to 7 by @dependabot[bot] in #1948
• chore(deps): bump actions/labeler from 6 to 7 by @dependabot[bot] in #1949
• chore(deps): update tokio-tungstenite requirement from 0.29 to 0.30 by @dependabot[bot] in #1954
• feat(grpc): serve /v1/responses in PD mode by @slin1237 in #1956
• feat(grpc): support batched completion prompts in the pipeline by @slin1237 in #1957
• fix(grpc): load Harmony encoding at gpt-oss worker registration by @slin1237 in #1959
• feat(rate_limit): add tenant rate-limit policy schema and compilation by @XinyueZhang369 in #1958
• feat(gateway): derive backend request ids from rid and the middleware request id by @slin1237 in #1960
• fix(mcp): preserve explicit tool_choice for the initial responses request by @slin1237 in #1963
• fix(gateway): apply PUT /workers/{id} instead of failing in the background by @junliu-mde in #1967
• feat(rate_limit): add reserve/settle engine by @XinyueZhang369 in #1966
• fix(router): support model aliases end to end by @junliu-mde in #1964
• feat(kimi-k3): add K3 support by @lightseek-bot in #1968
• chore(deps): bump vLLM floor to 0.26 and refresh docker matrix by @slin1237 in #1971
• docs(grpc-pipeline): document XGrammar and fix its pipeline order by @ighutake-debug in #1976
• feat(rate_limit): wire tenant rate limiter into CLI and AppContext by @XinyueZhang369 in #1969
• fix(protocols): accept function tools without parameters field by @ighutake-debug in #1975
• chore(deps): bump SGLang to 0.5.16 by @slin1237 in #1970
• chore(deps): bump TensorRT-LLM to 1.3.0rc22 by @slin1237 in #1972
• ci: add engine version watch and nightly failure triage automation by @slin1237 in #1973
• ci: run engine version watch weekly by @slin1237 in #1981
• chore(deps): bump TokenSpeed to 0f68676 by @slin1237 in #1982
• fix(kimi-k3): let forced tool calls close the prefilled channel by @key4ng in #1983
• chore(deps): bump actions/stale from 10 to 11 by @dependabot[bot] in #1985
• fix(multimodal): give Kimi-K3 its own vision processor and fix alpha ordering by @key4ng in #1984
• fix(kimi-k3): move the media wrapper and thinking-effort default to the prompt-encoding layer by @key4ng in #1995
• fix(gateway): scrape engine metrics via base URL under --dp-aware by @ighutake-debug in #1994
• fix(scheduler): retain worker capacity tracking by @ShiroKSH in #1997
• fix(gateway): reject schemeless worker URLs at the API boundary by @ighutake-debug in #1977
• chore(release): bump versions for v1.9.0 by @slin1237 in #1996

New Contributors

• @santhreal made their first contribution in #1941
• @ighutake-debug made their first contribution in #1976
• @ShiroKSH made their first contribution in #1997

Full Changelog: v1.8.0...v1.9.0

V1.8.0

🚀 Shepherd Model Gateway v1.8.0 Released

Our biggest multimodal release yet: EPD (Encode-Prefill-Decode) disaggregation, engine-agnostic tensor transport over SHM and RDMA, and audio support (Qwen3 ASR + Omni) — plus dedicated multimodal + audio support for Thinking Machines' Inkling model, per-tenant API keys, LoRA adapter management, and a second nightly correctness harness.

⚠️ Breaking change: The legacy smg server command has been removed. Use smg launch or smg serve instead. Update your deployment scripts before upgrading.

🧩 EPD: Encode-Prefill-Decode Disaggregation

Multimodal serving now disaggregates the encode stage from prefill and decode.

Just as PD disaggregation separates prefill from decode, EPD adds a third independently-scheduled stage — encode — for multimodal inputs. The vision/audio encoder runs on its own workers, so image, video, and audio preprocessing no longer competes with the prefill/decode path:

• Dedicated encode routing — Multimodal inputs routed to encode workers, encoded tensors forwarded to prefill
• Unified Mode-parameterized router — Regular, PD, and EPD collapsed into a single router with a shared EncodeStage, rather than three separate code paths
• Python CLI + Helm exposure — Configure EPD directly from the CLI
• EPD smoke CI — 4-GPU H100 multimodal coverage in the nightly matrix

Impact: Encode-heavy workloads (lots of images or video) scale independently. You can provision encode capacity separately from prefill/decode, and a burst of multimodal requests won't starve your text generation throughput.

🚚 Engine-Agnostic Tensor Transport: SHM + RDMA

Encoded multimodal tensors now move between workers over a configurable, engine-neutral transport layer.

Moving preprocessed tensors from encode workers to prefill workers efficiently is the hard part of EPD. This release generalizes that transport across engines and interconnects:

• Configurable tensor transport — Engine-agnostic interface, not hardwired per backend
• vLLM SHM transport — Shared-memory tensor handoff for same-node encode→prefill
• RDMA pixel transport — New mm_rdma crate for cross-node tensor movement, routed through a shared payload resolver
• Shared proto handles — ShmHandle and RemoteTensorHandle hoisted to common.proto for reuse across backends
• Centralized backend×modality capability check — One place that knows which engine supports which modality over which transport

Impact: Multimodal disaggregation works the same way whether tensors move over shared memory on one node or RDMA across nodes, and the same transport layer serves vLLM, TokenSpeed, and future backends.

🎙️ Audio Support: Qwen3 ASR + Omni

SMG now handles audio — the modality lineup is text, image, video, and audio.

• Qwen3 ASR — Automatic speech recognition
• Qwen3 Omni — Full multimodal (text + vision + audio)
• Audio-processor selection on ModelProcessorSpec — Per-model audio preprocessing
• Verified correct — Full-tensor parity test for Qwen3 audio log-Mel features against HuggingFace transformers

Impact: Speech and omni-modal models run through the same gateway, the same routing, and the same APIs as everything else — with audio preprocessing validated bit-for-bit against the reference.

🧠 Dedicated Support: Thinking Machines' Inkling

SMG adds first-class, end-to-end support for Inkling — Thinking Machines' multimodal model — including its full modality set.

• End-to-end integration — Complete serving-path support, from request handling through response assembly
• Multimodality (image) — Image input support, with multimodal placeholders aligned to the Inkling checkpoint template
• Audio — Audio input support, built on the same audio pipeline landing in this release

Impact: Inkling runs through SMG with dedicated handling across text, image, and audio — the same routing, caching, tokenization, and API surface as every other supported model, purpose-fit to Inkling's checkpoint. Thinking Machines models get a first-class home on SMG.

🎨 Multimodal Performance (contributed by NVIDIA)

Encoder and preprocessing optimizations from @yechank-nvidia:

• Reduced TokenSpeed encoder input transport overhead
• Optimized Qwen vision preprocessing
• Optimized OpenCV video decoding and thread allocation

Impact: Lower latency across vision and video workloads. Thanks again to @yechank-nvidia for continued multimodal contributions.

✅ Correctness: Second Nightly Harness (tau2-bench)

Building on the v1.7.0 BFCL parity work, we've added a second correctness gate.

• tau2-bench nightly A/B — Parser verification against pure vLLM, complementing BFCL
• Alternating schedule — BFCL runs Mon/Wed/Fri, tau2-bench runs Tue/Thu/Sat
• Hardened runs — Fail-fast timeouts and quieter logs

Impact: Two independent nightly harnesses now verify SMG's Rust-native tokenization, tool-call parsing, and reasoning parsing stay faithful to the reference. We're closing correctness gaps continuously, not once — and now double-checking with two different benchmarks.

🔐 Per-Tenant API Keys

Serving-path authentication now supports per-tenant API keys:

• Issue distinct API keys per tenant
• Authenticate serving requests against tenant-scoped keys

Impact: Multi-tenant deployments get proper key isolation on the serving path — each tenant authenticates with its own credentials.

🔌 LoRA Adapter Management

Dynamic LoRA adapter RPCs exposed in the SGLang scheduler proto:

• Load, unload, and list LoRA adapters at runtime
• No restart required to swap adapters

Impact: Serve and rotate fine-tuned LoRA adapters on the fly — load a new adapter, list what's active, unload what's stale, all through gRPC.

🔎 Local Model Discovery

• Discover local HTTP model IDs from /v1/models — Auto-populate model IDs from HTTP-mode workers instead of manual configuration

🧠 Thinking / Reasoning Resolution

Cleaner, layer-unified control over reasoning activation:

• reasoning_effort "none"/"minimal" → enable_thinking=false — Honor OpenAI-style effort hints on the gRPC chat path
• Unified thinking resolution by layer — One consistent resolution path instead of scattered logic
• Kimi K2.6 reasoning parser fix — Corrected kimi_k25 → kimi_thinking parser mapping

🤖 New Parser Support

• Sarashina — New tool-call parser

🎧 Realtime API on HTTP Router

Realtime API support added to the HTTP router.

🐛 Notable Fixes

• HTTP: Set application/json Content-Type for PD non-streaming responses
• PD: Only treat line-anchored data: [DONE] as stream terminator, route chat using full message history
• Worker: Query engine-specific HTTP load endpoint for vLLM/SGLang
• Tool parser: Treat qwen_xml tool-call arg values literally
• Data connector: Decode Postgres JSON columns as Value, not String
• gRPC: GetTokenizer RPC implemented for TokenSpeed backend

📦 Dependencies

• nixl-sys 0.10.1→1.3.1, str0m 0.20→0.21, lz4_flex 0.13→0.14, wasm-encoder 0.252→0.253

🏗️ Infrastructure

• TokenSpeed engine image — New release workflow for TokenSpeed Docker images
• Project governance docs — Added governance documentation
• CI fixes for SGLang e2e (protobuf gencode), MLX (transformers 5.13), and TokenSpeed (cutlass-dsl pin)

🙏 Welcome New Contributors

5 first-time contributors landed in this release — thank you all!

Full Changelog: v1.7.0...v1.8.0

Upgrade now: pip install smg --upgrade

🐑 Any modality. Any engine. Production-proven.

What's Changed

• fix(http): set application/json Content-Type for PD non-streaming responses by @chengxumiaodaren in #1857
• chore(deps): update str0m requirement from 0.20 to 0.21 by @dependabot[bot] in #1859
• feat(tau2): nightly tau2-bench A/B for parser verification by @key4ng in #1864
• ci: alternate BFCL (Mon/Wed/Fri) and tau2 (Tue/Thu/Sat) nightlies by @key4ng in #1866
• fix(ci): kimi-k2.6 SMG reasoning parser (kimi_k25 → kimi_thinking) by @key4ng in #1868
• feat(realtime): support the Realtime API on the HTTP router by @slin1237 in #1870
• fix(ci): pin nvidia-cutlass-dsl 4.5.2 for TokenSpeed (uv + build subprocess pip) by @key4ng in #1869
• fix(tau2): harden nightly — fail-fast timeouts + silence cost-lookup log spam by @key4ng in #1872
• fix(worker): query engine-specific HTTP load endpoint for vLLM/SGLang by @XinyueZhang369 in #1867
• fix(pd): only treat line-anchored data: [DONE] as stream terminator by @junliu-mde in #1878
• fix(ci): unbreak sglang/e2e (protobuf gencode) and MLX (transformers 5.13) jobs by @key4ng in #1880
• fix(grpc/chat): honor reasoning_effort "none"/"minimal" as enable_thinking=false by @qywu in #1876
• refactor(grpc/chat): unify thinking resolution by layer by @slin1237 in #1882
• feat(discovery): discover local HTTP model ids from /v1/models by @SpencerGarnets in #1793
• perf(multimodal): reduce TokenSpeed encoder input transport overhead by @yechank-nvidia in #1879
• perf(multimodal): Optimize Qwen vision preprocessing by @yechank-nvidia in https://git...

v1.7.0

🚀 Shepherd Model Gateway v1.7.0 Released

This release leads with a correctness milestone. SMG runs the correctness-critical path — tokenization, function-call parsing, and reasoning parsing — natively in Rust, and that path is now proven on par with vLLM on tool-calling quality, validated by a nightly BFCL A/B harness. Plus new PD and engine-load metrics, GLM-5.x support, custom routing keys, and a major dependency sweep.

🎯 Rust-Native Correctness, Validated Against vLLM

SMG doesn't pass requests through — it owns the correctness-critical path in Rust. Tokenization, function-call parsing, and reasoning parsing all run natively in Rust rather than deferring to the engine's Python. That's where the gateway's performance and stability come from: no Python interpreter on the hot path, no GIL contention, no unhandled exceptions mid-stream.

Reimplementing that path in Rust demands one thing above all: it has to stay faithful to the reference. So we built a nightly BFCL (Berkeley Function-Calling Leaderboard) A/B harness that runs the same evaluation twice — once with SMG fronting the engine, once against pure vLLM — every night, across six major open-source models:

• DeepSeek V4
• GLM-5.2
• gpt-oss
• Kimi K2.6
• MiniMax M2.7
• Qwen3.6

Why this matters: The native-Rust critical path gives you faster, more stable function calling and reasoning — and these results prove it does so without giving up an inch of correctness. We've been closing the gaps model by model, and the latest run puts SMG on par with vLLM across the board. This isn't a one-time benchmark; it's a nightly regression gate. Correctness is something we commit to and continuously verify.

📊 New Metrics: PD Disaggregation & Engine Load

Deeper operational visibility into prefill-decode and backend load:

• smg_pd_* metrics — Dedicated PD disaggregation metrics for monitoring prefill/decode split behavior
• smg_engine_* gauges — Engine load signals re-exported as first-class gauges
• SGLang reasoning token usage — Reasoning token counts now surfaced in usage stats

Impact: PD deployments get purpose-built metrics for the disaggregated serving path, and engine load is now visible through standard gauges — better autoscaling signals and clearer production dashboards.

🎯 Custom Routing Key on Any Policy

X-SMG-Routing-Key header now honored across all routing policies:

• Force routing decisions with an explicit key regardless of the active policy
• Pin related requests to the same worker for session affinity or cache locality
• Works with cache-aware, load-based, and every other policy

Impact: Application-level control over routing without changing your policy. Send the same routing key for a conversation or tenant and keep requests landing on the same worker.

🔧 Tool-Call Parsing Hardening

Substantial robustness work across model families:

• MiniMax-M2 — Parse all parallel invokes within a wrapper
• qwen_xml and glm4_moe — Coerce XML tool-call args by declared schema type
• gpt-oss (Harmony) — Emit parallel tool calls on the Chat API
• OpenAI-compat — Return content: null instead of whitespace when emitting tool calls

Impact: More reliable tool calling across MiniMax, Qwen, GLM, and gpt-oss — correct argument typing, proper parallel-call handling, and OpenAI-compatible response shapes. This hardening is exactly what the BFCL parity above measures.

📦 Major Dependency Modernization

Stack-wide upgrade sweep, including security-sensitive dependencies:

• Security/crypto: jsonwebtoken 9.3→10.4, sha2 0.10→0.11
• Networking: reqwest 0.12→0.13, redis 0.27→1.2 (+ deadpool)
• Observability: opentelemetry 0.27→0.32 stack
• Tokenization: tiktoken-rs 0.9→0.12
• MCP: rmcp 0.8→1.7
• WASM: wasmtime 41→45
• WebRTC: str0m 0.19→0.20
• Core: rand 0.9→0.10, schemars 0.8→1.2
• Raised compatible dependency floors across the board

⚡ Performance

• TokenSpeed multimodal tensor transport — Optimized multimodal tensor handling (contributed by NVIDIA, thanks @yechank-nvidia)
• Cache env lookups off the request hot path — No repeated environment lookups per request

🐛 Notable Fixes

• Protocols: Accept min_tokens=0 in OpenAI requests
• gRPC: Drop stale WatchLoadUpdateReq piggyback, SGLang floor bumped to 0.5.13
• Packaging: Include LICENSE files in setuptools/maturin Python packages
• Multimodal: Gate shm-namespace test to Linux

🧪 Testing & CI

• Multi-model BFCL matrix — Nightly A/B on per-model runners across DeepSeek V4, GLM-5.2, gpt-oss, Kimi K2.6, MiniMax M2.7, and Qwen3.6
• Hermetic test images — Multimodal tests serve images from the repo instead of picsum.photos
• TRT-LLM startup — Pin Open MPI to loopback to fix flaky 1-GPU startup

🏗️ Infrastructure

• aarch64 wheels — Built on manylinux_2_28 for ARM compatibility
• Hermetic PyPI upload — Metadata 2.4 compliance via hermetic Python

🙏 Welcome New Contributors

4 first-time contributors landed in this release — thank you all!

Full Changelog: v1.6.0...v1.7.0

Upgrade now: pip install smg --upgrade

🐑 Rust-native correctness. Production-proven.

What's Changed

• feat(bfcl): multi-model nightly A/B matrix on per-model runners by @key4ng in #1791
• feat(metrics): add smg_pd_* PD disaggregation metrics by @slin1237 in #1781
• feat(observability): re-export engine load signals as smg_engine_* gauges by @slin1237 in #1780
• feat(grpc): add sglang reasoning token usage by @Moersity in #1747
• test(e2e): serve multimodal test images from repo, not picsum.photos by @gongwei-130 in #1801
• refactor(grpc): reduce proto-change friction in the Go bindings by @slin1237 in #1800
• perf: cache env lookups off request hot paths by @gongwei-130 in #1799
• chore(deps): bump tiktoken-rs 0.9->0.12 by @slin1237 in #1807
• chore(deps): bump rmcp 0.8 -> 1.7 by @slin1237 in #1814
• chore(deps): bump redis 0.27->1.2 (+ deadpool 0.13, deadpool-redis 0.23) by @slin1237 in #1805
• chore(deps): bump sha2 0.10 -> 0.11 by @slin1237 in #1804
• chore(deps): bump opentelemetry 0.27->0.32 stack by @slin1237 in #1813
• build(deps): bump rand 0.9 -> 0.10 by @slin1237 in #1808
• chore(deps): bump schemars 0.8 -> 1.2 by @slin1237 in #1812
• chore(deps): bump wasmtime and wasmtime-wasi 41->45 by @slin1237 in #1806
• chore(deps): bump str0m 0.19 -> 0.20 by @slin1237 in #1811
• build(deps): raise compatible dependency floors by @slin1237 in #1809
• build(deps): bump reqwest 0.12 -> 0.13 by @slin1237 in #1803
• chore(deps): bump jsonwebtoken 9.3 -> 10.4 by @slin1237 in #1810
• perf(tokenspeed): optimize multimodal tensor transport by @yechank-nvidia in #1604
• fix(bfcl): max_model_len 32768 + OpenAICompletionsHandler for multi-turn FC by @key4ng in #1819
• test(e2e): don't assert exact image count for duplicate images by @slin1237 in #1821
• chore(ci): disable BFCL PR trigger, make nightly benchmark weekly by @key4ng in #1832
• fix(e2e): pin trtllm Open MPI to loopback to fix flaky 1-GPU startup by @key4ng in #1823
• fix(tool_parser): parse all parallel invokes in a minimax_m2 wrapper by @key4ng in #1824
• chore(deps): bump actions/cache from 5 to 6 by @dependabot[bot] in #1826
• chore(deps): update opencv requirement from 0.98.2 to 0.99.0 by @dependabot[bot] in #1828
• chore(deps): bump actions/checkout from 6 to 7 by @dependabot[bot] in #1825
• ci(benchmarks): bump genai-bench timeout 480s -> 600s by @key4ng in #1838
• ci(bfcl): add GLM-5.2-FP8 nightly leg (TP=8 sequential, whole node) by @key4ng in #1834
• feat(parser): route GLM-5.x to existing GLM-4.7/4.5 parsers by @key4ng in #1833
• ci(bfcl): set max_model_len=auto and capture both arms' transcripts by @key4ng in #1840
• fix(grpc): return content:null instead of whitespace when emitting tool calls (OpenAI-compat) by @key4ng in #1822
• fix(tokenizer): serialize DeepSeek V3.2/V4 prompt JSON with json.dumps spacing by @key4ng in #1839
• fix(tool_parser): coerce XML tool-call args by declared schema type (qwen_xml, glm4_moe) by @key4ng in #1841
• fix(tokenizer): render object-form tool-call arguments in DeepSeek V3.2/V4 encoders by @key4ng in #1844
• feat(harmony): emit parallel tool calls for gpt-oss on the Chat API by @key4ng in #1845
• fix(...

v1.6.0

🚀 Shepherd Model Gateway v1.6.0 Released

A performance and production-hardening release: unified SSE codec across every streaming path, KV cache events for vLLM and TokenSpeed, routing hot-path optimizations, multimodal optimizations, and graceful shutdown hardening.

⚡ Streaming Performance: Unified SSE Codec

Every streaming path now runs through a single shared SSE codec — no more per-router SSE parsing and re-encoding:

Anthropic Messages — Shared SseDecoder for upstream, SseEncoder for output
OpenAI Responses — Shared encoder for streaming output, shared codec for block parsing
gRPC-regular and gRPC-harmony — Shared encoder for inline SSE framing
HTTP PD — Shared encoder for logprob-merge SSE re-encode

Impact: Consistent, optimized SSE handling across the entire gateway. One codec, tested once, used everywhere — lower CPU overhead on streaming responses and a single place to maintain the protocol.

🧠 Event-Driven Cache-Aware Routing: vLLM + TokenSpeed

KV cache event support extended to two more backends via the SubscribeKvEvents bridge:

vLLM KV cache events — Real-time cache state for cache-aware routing
TokenSpeed KV cache events — Same event-driven routing for TokenSpeed deployments
Removed the 2048-worker cap — Scale cache-aware routing past 2048 workers
Surface KV subscription failures — No more silent failures in the event pipeline

Impact: Cache-aware routing now learns live cache state from vLLM and TokenSpeed, not just SGLang — routing requests to the worker most likely to have a warm prefix. Plus large-fleet deployments are no longer capped at 2048 workers.

🗺️ Cache-Aware & Worker Routing Performance

Lower per-request routing overhead on the hot path:

Single-pass routing state — Gather all worker routing state in one pass instead of repeated scans per decision
Filter without wasted clones — Worker filtering clones only matching workers, not the entire fleet
Decode-faster-than-prefill imbalance fixed — Cache-aware routing no longer skews load when decode outpaces prefill

Impact: Cheaper routing decisions per request and less allocation pressure under high worker counts — compounding with the 2048-worker cap removal above.

🎨 Multimodal Optimizations

Faster vision processing and a generalized multimodal ABI (contributed by NVIDIA):

Multimodal ABI generalization — Cleaner backend-agnostic multimodal interface
Optimized video decode and Qwen VL preprocessing — Faster multimodal request handling
Qwen2-VL preprocessor_config overrides — Honor model-specific preprocessing config
OpenCV video feature — Setup script for the opencv-video build feature

Impact: Lower latency on vision and video workloads, with a cleaner multimodal interface that generalizes across backends. Thanks to @yechank-nvidia for the contributions.

🔌 Tokenizer Performance

Shared L1 prefix on cache hit — Fused miss-path tokenization, less redundant work
Offload encode to a bounded blocking pool — Tokenizer encode no longer blocks the async runtime
Encode L1 cache suffix without special tokens — Correct cache key derivation

🛡️ Graceful Shutdown Hardening

Production-grade drain and shutdown behavior:

Keep accepting during LB propagation window — No dropped requests while load balancers learn the node is draining
Drain in-flight gRPC RPCs before closing ZMQ — Clean gRPC servicer shutdown
O(1) readiness checks — Plus optional dedicated probe listener (--health-check-port) and drain-aware probes

Impact: Zero-downtime deployments. Nodes drain cleanly, in-flight requests complete, and dedicated health-check ports keep probes responsive even under load.

⚖️ DP-Aware Routing for vLLM

Data-parallel routing for vLLM gRPC workers:

Rank pinning for DP workers
Mooncake DP mint for PD disaggregation

🎯 Passthrough Routing Policy

New passthrough routing policy for single-backend serving — skip routing overhead entirely when there's only one backend.

📊 Observability

Tokio event-loop canary — Runtime metrics sampler to detect event-loop stalls
Skip metrics collection when no client is subscribed — No overhead when nobody's watching
Removed /ws/metrics WebSocket endpoint — Retired in favor of standard metrics

💬 Messages API

Accept system role in messages[] and forward in place on /v1/messages

🧩 WASM

--enable-wasm flag added to the Python CLI and Helm chart

🧪 Testing Infrastructure

No-GPU routing experiments and continuous quality benchmarks:

Mock worker fleet — Multi-port mock HTTP/gRPC worker and realistic engine simulator for routing A/B testing without GPUs
Nightly BFCL benchmarks — SMG frontend vs pure vLLM across non_live, live, and multi_turn (verifying SMG adds no quality regression)

🐛 Notable Fixes

Tool parser: MiniMax-M2 unknown tool names forwarded instead of leaking markup, args coerced by declared schema type, byte-offset and char-boundary correctness for partial JSON
gRPC: TRT-LLM health classified by exact OK status (not substring), SGLang token IDs sent as array("q") to match contract
Responses: function_call serde alignment
TokenSpeed: Allow idle gRPC keepalive pings
Gateway: Valid-JSON SSE error frame on PD streaming decode error, HTTP metrics path label bounded to matched route

🧹 Cleanup

Removed the smg-tui crate and dependencies
Dead code and stale comment cleanup across auth, WASM, MCP, multimodal, protocols, and more

🏗️ Runtime & Infrastructure

Slimmed runtime config to --runtime-worker-threads override with container-aware defaults
Lazy-load generated gRPC proto re-exports
CI: bin-packed GPU runners via Volcano, RDMA for Mooncake PD

🙏 Welcome New Contributors

7 first-time contributors landed in this release — thank you all!

Full Changelog: v1.5.0...v1.6.0

Upgrade now: pip install smg --upgrade

🐑 Shepherd your LLM infrastructure with confidence.

What's Changed

• test(mock-worker): add multi-port mock HTTP/gRPC worker and scale-test rig by @slin1237 in #1696
• fix(model_gateway): bound HTTP metrics path label to matched route by @slin1237 in #1679
• chore(routers/http): remove dead code and trim stale comments by @slin1237 in #1651
• perf(observability): skip metrics_ws collection when no client is subscribed by @slin1237 in #1687
• fix(mock-worker): keep the fleet running when one worker exits by @slin1237 in #1697
• fix(tool_parser): make partial JSON consumed position a byte offset by @slin1237 in #1677
• fix(grpc): classify TRT-LLM health by exact OK status, not substring by @slin1237 in #1681
• feat(grpc): add vLLM KV cache event support (SubscribeKvEvents bridge) by @key4ng in #1652
• fix(tokenizer): encode L1 cache suffix without special tokens by @slin1237 in #1680
• refactor(multimodal): Multimodal ABI generalization by @yechank-nvidia in #1602
• revert(ci): use NVMe storage for H100 runner workspaces by @key4ng in #1704
• fix(ci): fix 2-gpu runner for RDMA by @XinyueZhang369 in #1705
• perf(worker): clone only matching workers in get_workers_filtered by @slin1237 in #1699
• ci(release-docker): fail loud on push detection and add run summary by @key4ng in #1710
• perf(smg): mitigate the worker-sync lag-resync burst by @CatherineSue in #1667
• perf(mesh): bound op-log memory at O(live keys) and share round snapshots by @CatherineSue in #1671
• fix(ci): lower vllm mooncake pd mmlu concurrency by @key4ng in #1720
• fix(ci): make Oracle test users unique by @key4ng in #1719
• fix(ci): pin CUDA 13 CUTLASS DSL libs by @key4ng in #1717
• chore(tui): remove smg-tui crate and related dependencies by @key4ng in #1718
• ci: cancel PR workflows on close by @key4ng in #1723
• ci: increase gpu timeout budgets by @key4ng in #1721
• feat(grpc): DP-aware routing for vLLM gRPC workers (rank pinning + Mooncake DP mint) by @CatherineSue in #1673
• fix(tool_parser): iterate char boundaries in ends_with_partial_token by @slin1237 in #1676
• perf(tokenizer): share L1 prefix on hit and fuse miss-path tokenization by @slin1237 in #1709
• fix(kv_index): remove the 2048-worker cap and surface KV subscription failures by @slin1237 in #1706
• ci: use rdma for mooncake pd by @key4ng in #1722
• chore(ci): temporarily disable nightly MLX benchmark schedule by @key4ng in #1726
• fix(grpc/sglang): send token IDs as array("q") to match SGLang contract by @gongwei-130 in #1725
• fix(ci): pin FastAPI below 0.137 for vLLM by @key4ng in #1727
• feat(observability): add ...

v1.5.0

🚀 Shepherd Model Gateway v1.5.0 Released

Our biggest release yet: priority scheduling with preemption, two new backends — TokenSpeed and MLX — bringing SMG to five supported engines, and complete Responses API protocol coverage.

🎛️ Priority Scheduler

New request scheduler with admission control, prioritization, and preemption:

• Priority classes — Configurable request classes with policy-driven admission
• Admission control — Inflight tracking, queue management, and capacity slots
• Preemption — Victim search and preempt path for high-priority requests under load
• Capacity-proportional reservations — Floor + share guarantees per priority class, restored automatically after backend capacity recovers
• Full observability — Operational metrics, tracing, and autoscaling gauges via metrics sampler
• Battle-tested — Integration tests covering rejection, clamping, preemption, and starvation scenarios

Impact: Run mixed workloads on shared capacity. Latency-sensitive interactive traffic preempts batch jobs, priority classes get guaranteed capacity shares, and autoscalers get the signals they need.

⚡ TokenSpeed Backend

TokenSpeed joins SMG with full first-class gRPC integration:

• Complete gRPC pipeline — Native TokenSpeed gRPC client, Python servicer, and router wiring
• Multimodal support — Vision-language model (VLM) input through the gRPC pipeline
• Admin operations — FlushCache and profile RPCs with worker-abstracted admin ops
• Load reporting — Server-info label conversion for load-aware routing
• Production CI — Full install and GPU E2E coverage from day one

Impact: TokenSpeed deployments get everything SMG offers — cache-aware routing, load balancing, priority scheduling, the full API surface (Chat Completions, Responses, Messages), and multimodal — through the same battle-tested gRPC pipeline that powers SGLang, vLLM, and TensorRT-LLM.

🍎 MLX Backend: Apple Silicon Support

SMG now runs on Apple Silicon via MLX:

• Native MlxEngine gRPC proto, Rust client, and Python gRPC servicer
• Per-step admission and own-thread BatchGenerator for stable concurrency
• Coalesced concurrent prefill admission for agent workloads
• macOS CI workflow with E2E coverage
• Nightly benchmark: MLX direct HTTP vs Router+gRPC

Impact: Develop and serve on Mac. The same gateway that fronts your GPU fleet now runs your local MLX models — same APIs, same routing, same tooling.

SMG now supports five backends:vLLM, TokenSpeed, TensorRT-LLM, SGLang and MLX.

🛠️ Responses API: Complete Protocol Surface

Full OpenAI Responses API spec coverage — tools, content parts, and typed protocol items:

New hosted tools:

• image_generation — Full streaming events wired across OpenAI, gRPC-regular, and gRPC-harmony routers, with MCP-backed dispatch and output metadata (action, background, format, quality, size)
• web_search (non-preview) with results and return_token_budget
• file_search, computer use, local_shell, containerized shell, apply_patch, custom tools, and namespace tool grouping

Protocol completeness:

• Typed content parts and annotations with round-trip tests
• ConversationRef union typing, item_reference inputs, compaction items
• Full ToolChoice variant coverage
• Fail-fast on unknown content/items — no more silent swallowing

Impact: SMG remains the only gateway with Responses API support for open-source models and third-party vendors — now with the complete protocol surface, including hosted tools like image generation and computer use.

💬 Messages API Enhancements

• HTTP router support — Messages API now available on the HTTP router, not just gRPC
• Adaptive thinking — Support adaptive thinking on /v1/messages
• Protocol fidelity — Preserve system text and unknown fields end-to-end

🤖 New Model Support

• DeepSeek V3.2 and V4 — Chat-template encoders with tools, thinking introspection, and DSML tool call parser
• Kimi K2 / K2.5 / K2.6 — Tiktoken tokenizer support, K2.5 vision via gRPC router, chat template tool injection
• Qwen3.5 family — Routed to Qwen3-VL multimodal processor
• GPT-4o and modern OpenAI models — o200k_base tokenizer support

⚖️ Smarter Load Balancing

New policies and KV-aware routing signals:

• least_load policy — Routes by token-work expected-wait, not just request counts
• kv_pressure_weight policy — Balances on KV cache pressure
• KV-aware imbalance triggers — Spread and overload detection via load monitor
• DP-aware load balancing — Scheduler load info forwarded from gRPC backends, DP logical workers routed via base endpoint
• GetLoads endpoint for the vLLM engine service

🔄 Worker Lifecycle State Machine

Workers now have a real state machine:

• WorkerStatus enum with Pending start semantics and Draining state
• Workflow-driven drain for graceful removal
• Event-driven WorkerMonitor with richer event payloads
• WorkerCapacity tracker with 4-tier capacity sourcing
• max_running_requests surfaced from SGLang /server_info

🔧 PD Disaggregation Reliability

• NIXL PD now actually transfers KV cache — kv_transfer_params relayed correctly
• vLLM MooncakeConnector PD driven with router-minted kv_transfer_params
• Smooth regular→PD transition — no disruption when enabling disaggregation
• Fixed PD cache-aware policy lifecycle

🌐 Golang gRPC Client

Official Go client package joins Python, Rust, and Java SDKs.

🎙️ Audio Transcriptions

New /v1/audio/transcriptions multipart route.

📈 Performance

• /workers endpoint 5x faster
• Fused cache-aware match+insert — Single tree descent instead of two, with concurrency stress coverage
• K8s discovery — Label selectors pushed to the API server instead of client-side filtering
• Python bindings — GIL released while the router server runs

🐛 Notable Fixes

• Tokenizer: HuggingFace tojson formatting parity, EOS token IDs from generation_config.json, L1 prefix cache keyed on add_special_tokens
• Reasoning: Fresh parser on non-streaming path (no shared mutex), whitespace preserved in BaseReasoningParser
• MCP: Internal self-brought MCP details hidden from final responses, citations/sources/query included when available, forwarded request headers preserved
• gRPC: request.seed passed through to vLLM, backend sampling defaults applied, ResponsesRequest sampling params passed to all backends
• Readiness: Wait for gRPC worker tokenizer autoload before reporting ready
• Build: Debug symbols kept in release binaries

🏗️ Infrastructure

• Nightly GHCR releases for SMG + vLLM/SGLang/TRT-LLM engine images
• Dev wheels published to GitHub Releases and wheel index
• CI: vLLM ≥0.22.1 (cu13 stack), SGLang 0.5.12.post1, TensorRT-LLM 1.3.0rc18
• Rust toolchain 1.95.0

🙏 Welcome New Contributors

25 first-time contributors landed in this release — thank you all!

Full Changelog: v1.4.1...v1.5.0

Upgrade now: pip install smg --upgrade

🐑 Shepherd your LLM infrastructure with confidence.

What's Changed

• feat(image): add image generation tool support by @TingtingZhou7 in #1057
• refactor(core): move worker domain files from core/ to worker/ by @slin1237 in #1079
• refactor(core): move workflow files to workflow/, delete core/ by @slin1237 in #1081
• refactor: move job_queue.rs from worker/ to workflow/ by @slin1237 in #1083
• fix(python): add router arg fallback disable flag by @gongwei-130 in #1072
• Revert "feat(image): add image generation tool support" by @CatherineSue in #1087
• fix(docker): install smg-grpc-servicer from source in engine images by @key4ng in #1086
• feat(mcp): hide internal self-brought MCP details from final responses by @zhoug9127 in #1061
• refactor(protocols): add WorkerStatus enum to protocol crate by @slin1237 in #1093
• feat(mlx): add MlxEngine gRPC proto and Rust client by @key4ng in #1034
• ci: allow 'ci-approved' label to bypass fork PR approval gate by @slin1237 in #1095
• refactor(worker): expand WorkerEvent with richer payloads by @slin1237 in #1100
• refactor(worker): replace healthy AtomicBool with status AtomicU8 by @slin1237 in #1101
• ci: apply change detection to pull_request_target events by @ai-jz in #1104
• refactor(worker): land Pending start semantics and state machine by @slin1237 in #1102
• refactor(worker): extract health loop into WorkerManager by @slin1237 in #1105
• test(completions): add E2E tests for /v1/completions gRPC endpoint by @vschandramourya in #1021
• refactor(worker): reorganize WorkerRegistry methods and doc API by @slin1237 in #1113
• fix(deps): bump google.golang.org/grpc to v1.79.3 for CVE-2026-33186 by @slin1237 in #1120
• refactor(worker): create event-driven WorkerMonitor (PR 8) by @slin1237 in #1118
• refactor(worker): delete set_healthy and migrate every caller (PR 9) by @slin1237 in https://github.com...

v1.4.1

🚀 Shepherd Model Gateway v1.4.1 Released

Patch release with mesh HA stability fix, DP rank scheduling, reasoning parser fixes, and engine version bumps.

Mesh HA Stability Fix

Fixed premature worker removal during rolling deploys:

• Workers synced via mesh with health: false were being removed by the health checker before they had a chance to pass local health checks
• Fix: health checker now only removes workers whose health check actually failed this tick, not workers that are merely marked unhealthy from mesh state
• Eliminates the 500/503 error spike during gateway redeploys with --remove-unhealthy-workers enabled

DP Rank Scheduling

Data-parallel rank scheduling for multi-GPU inference:

• Supports scheduling with the minimum number of required ranks
• New scheduling policy for DP-aware worker selection

MCP Tool Improvements

• Argument overrides (#1048) -- Add support for argument overrides with MCP tools, enabling per-request customization of MCP tool call parameters
• Passthrough output flattening (#1041) -- MCP passthrough mcp_call output now flattened to plain strings for consistency
• ID normalization (#989) -- MCP call item IDs normalized to mcp_ prefix for OpenAI alignment

Reasoning Parser Fixes

• Thinking toggle detection (#1031) -- Detect thinking toggle from chat template and override parser state automatically
• NanoV3/Nemotron fix (#1067) -- Changed parser to always_in_reasoning=false to fix incorrect reasoning block detection
• Harmony routing (#1025) -- Route reasoning_content to analysis channel per Harmony spec

Bug Fixes

• Routing: Eliminate unconditional token allocation on the hot path (#1024)
• Responses API: Stop defaulting top_p for omitted requests (#1043), unify upstream header handling (#1029)
• gRPC: Update vLLM imports for inputs reorganization (#1033)
• Frontend: Fix smg serve rejecting vLLM OpenAI args (#832)
• Discovery: Periodic reconciliation with identity-based pod equality (#1039)

Engine Version Bumps

• vLLM: v0.18.0 -> v0.19.0
• SGLang: v0.5.9/v0.5.10rc0 -> v0.5.10
• TensorRT-LLM: 1.3.0rc8 -> 1.3.0rc10

Infrastructure

• Claude review workflow hardened with incremental reviews and auto-approve (#1036, #1040, #1042)
• E2E worker failure diagnostics and cleanup improvements (#1015)
• gRPC package releases: smg-grpc-proto 0.4.6, smg-grpc-servicer 0.5.2

Upgrade now: pip install smg --upgrade

🐑 Shepherd your LLM infrastructure with confidence.

Docker Images

Pre-built engine images on GitHub Container Registry:

SGLang:

docker pull ghcr.io/lightseekorg/smg:1.4.1-sglang-v0.5.10

vLLM:

docker pull ghcr.io/lightseekorg/smg:1.4.1-vllm-v0.19.0

TensorRT-LLM:

docker pull ghcr.io/lightseekorg/smg:1.4.1-trtllm-1.3.0rc10

All images for v1.4.1:

Engine	Tag	Pull Command
sglang	1.4.1-sglang-v0.5.10	docker pull ghcr.io/lightseekorg/smg:1.4.1-sglang-v0.5.10
trtllm	1.4.1-trtllm-1.3.0rc10	docker pull ghcr.io/lightseekorg/smg:1.4.1-trtllm-1.3.0rc10
vllm	1.4.1-vllm-v0.19.0	docker pull ghcr.io/lightseekorg/smg:1.4.1-vllm-v0.19.0

What's Changed

• perf: Eliminate unconditional token allocation on the routing hot path by @ppraneth in #1024
• refactor(e2e): rename worker_args to sglang_args by @CatherineSue in #1019
• fix(ci): improve e2e worker failure diagnostics and cleanup by @key4ng in #1015
• feat(metrics-ws): [2/4] add protocol types and watch registry by @key4ng in #982
• fix(harmony): route reasoning_content to analysis channel per Harmony spec by @CatherineSue in #1025
• fix(openai): unify responses upstream header handling by @zhaowenzi in #1029
• fix(grpc): update vLLM imports for inputs reorganization by @CatherineSue in #1033
• fix(reasoning): detect thinking toggle from chat template and override parser state by @CatherineSue in #1031
• fix(ci): harden Claude review workflow with incremental reviews and resilience by @key4ng in #1036
• fix(ci): fix comment fetch, add review summary, and auto-approve by @key4ng in #1040
• fix(ci): handle array-format execution output in review summary by @key4ng in #1042
• fix(mcp): flatten passthrough mcp_call output to plain strings by @zhaowenzi in #1041
• feat(metrics-ws): [3/4] add event-driven and polled collectors by @key4ng in #1027
• fix(responses): stop defaulting top_p for omitted requests by @zhaowenzi in #1043
• fix(frontend): Fix smg serve reject vLLM OpenAI args by @YouNeedCryDear in #832
• feat(realtime-api): WebRTC relay bridge by @pallasathena92 in #733
• feat(overrides): add support for argument overrides with mcp tools by @Tobel158 in #1048
• fix(mcp): normalize mcp_call item IDs to use mcp_ prefix for OpenAI alignment by @zhaowenzi in #989
• feat: supports dp rank scheduling and scheduling with the minimun number of… by @jiashaokun-1 in #1007
• fix(discovery): periodic reconciliation with identity-based pod equality by @Kangyan-Zhou in #1039
• chore(deps): update wasm-encoder requirement from 0.245 to 0.246 by @dependabot[bot] in #1054
• chore(deps): update lz4_flex requirement from 0.11 to 0.13 by @dependabot[bot] in #1053
• chore(deps): update str0m requirement from 0.16 to 0.18 by @dependabot[bot] in #1052
• chore(deps): bump vllm base image from v0.18.0 to v0.19.0 by @slin1237 in #1066
• fix(reasoning): change NanoV3/Nemotron parser to always_in_reasoning=false by @CatherineSue in #1067
• chore(deps): bump sglang from 0.5.9/0.5.10rc0 to 0.5.10 by @slin1237 in #1064
• feat(metrics-ws): [4/4] add /ws/metrics endpoint with subscription support by @key4ng in #1050
• fix(mesh): prevent premature removal of unhealthy workers by health checker by @slin1237 in #1076
• chore(deps): bump TensorRT-LLM from 1.3.0rc8 to 1.3.0rc10 by @slin1237 in #1077
• chore(grpc): release smg-grpc-proto 0.4.6 and smg-grpc-servicer 0.5.2 by @slin1237 in #1078
• chore: bump versions for v1.4.1 release by @slin1237 in #1080

New Contributors

• @Tobel158 made their first contribution in #1048
• @jiashaokun-1 made their first contribution in #1007

Full Changelog: v1.4.0...v1.4.1

v1.4.0

🚀 Shepherd Model Gateway v1.4.0 Released

The biggest SMG release yet -- Kubernetes-native deployment via Helm, a terminal dashboard, 200x mesh memory reduction, 7-11x faster multimodal preprocessing, native Completion API over gRPC, and per-model retry configuration.

Kubernetes-Native Deployment with Helm

Production-ready Helm chart for deploying SMG on Kubernetes:

• One-command deployment -- helm install smg oci://ghcr.io/lightseekorg/smg-helm deploys the full gateway stack
• Router + Worker deployment -- A single chart deploys both the gateway router and inference engine workers (vLLM, SGLang, TRT-LLM) with GPU scheduling
• Mesh HA with service discovery -- Deploy multiple gateway replicas as a StatefulSet with automatic gossip-based peer discovery via --router-selector
• Full K8s integration -- RBAC, Ingress, HPA, PDB, ServiceMonitor, Grafana dashboard ConfigMap, JSON Schema validation at helm lint time
• 5 example configurations -- Router-only, with-postgres, with-service-discovery, with-ingress, with-monitoring

Impact: Zero-to-production SMG deployment on Kubernetes with a single helm install. Declarative configuration, automatic scaling, and built-in observability.

Terminal Dashboard (smg-tui)

Full-featured terminal UI for real-time monitoring and interactive chat:

• 7 tabs -- Pulse (real-time dashboard with sparklines), Workers (per-worker stats + circuit breaker state), Chat (streaming markdown playground), Logs (per-component with ANSI stripping), Benchmark, Traffic, Mesh
• Worker management -- Quick-add presets for OpenAI/Anthropic/xAI/Gemini, local worker launch with automatic GPU selection via nvidia-smi, GPU claim tracking to prevent double-allocation
• Gateway auto-start -- smg-tui --auto-start launches the gateway, polls health, and cleans up on exit
• Chat playground -- Streaming SSE with live cursor, markdown rendering, multi-turn support, Tab to cycle models

Mesh Performance & Reliability Revolution

Eliminated catastrophic memory growth and achieved >200x improvement in mesh resource usage:

• Delta encoding (#899): Only send new tree operations since last sync -- 40x smaller sync payloads (18.3 MB → 417 KB), gzip compression for additional 5-8x wire reduction
• Lazy serialization (#919): Moved full TreeState serialization off the hot path -- memory: OOM crash → 31 MB stable, CPU: 280-345% → 56-58%, latency: 12s degrading → stable
• CRDT bypass (#961): Moved tree state out of CRDT operation log -- eliminated ~1 GB/1.5hr memory leak under sustained load
• Two-layer sync fix (#1011): Eliminated remaining memory leaks in the tree sync protocol
• Snapshot serialization (#974): Structure-preserving radix tree snapshots for mesh sync -- shared prefixes stored once, replacing 40 MB flat operation replay with compact tree format
• Timeout enforcement (#952): Consistent timeout contract across all RPC and stream paths
• Health mirroring (#912, #892): Mesh-synced workers now register locally for health checking with proper status mirroring

Benchmark Results (20 min, 500 rps, 20K-char prompts):

• 565,920 requests, 0 errors
• Memory plateaus at ~2.3 GB (no linear growth)

7-11x Faster Multimodal Image Preprocessing

SMG now matches or beats HuggingFace Python preprocessing performance:

• SIMD resize -- Replaced image crate (pure Rust) with fast_image_resize v6 (AVX2/SSE4.1) for 10-25x faster resize
• Fused operations -- Combined to_tensor_and_normalize(), zero-copy patchify_into(), fused pad + normalize + tile split for Llama4
• Additional optimizations -- Thread-local Resizer reuse, eliminated DynamicImage clones, optimized serialization and tensor conversion

Benchmark Results (Qwen3-VL):

Image Size	Before	After	vs HuggingFace Python
224×224	4.77 ms	0.44 ms (10.8x)	2.5x faster
640×480	15.5 ms	1.59 ms (9.7x)	1.8x faster
1024×768	40.6 ms	4.31 ms (9.4x)	1.6x faster
1920×1080	286 ms	39.6 ms (7.2x)	~parity

Native Completion API over gRPC

Full /v1/completions support through the gRPC pipeline with streaming and PD disaggregation:

• 6-PR pipeline -- CompletionRequest type, preparation stage, request building with backend sampling params, response processing, pipeline wiring, streaming support
• Streaming -- OpenAI-compatible SSE events with per-index stop decoder tracking, echo and suffix handling
• PD mode -- Dual streaming for prefill-decode disaggregation
• Type safety -- Native RequestType::Completion throughout the pipeline, exhaustive match arms in shared stages

Per-Model Retry Configuration

Different models can now have different retry policies:

• WorkerRegistry integration -- Workers declare per-model retry config via WorkerSpec.resilience, stored in WorkerRegistry with last-write-wins semantics
• All routers updated -- HTTP, gRPC, OpenAI, Gemini, gRPC PD, and HTTP PD routers all look up per-model config at request time, falling back to the global default
• Cleanup on removal -- Retry config is automatically cleaned up when the last worker for a model is removed

Impact: GPU-constrained models can have longer timeouts and more retries, while fast models use aggressive retry budgets. No more one-size-fits-all.

Three-Phase Graceful Shutdown

Replace fixed-timeout shutdown with an intelligent Gate → Drain → Teardown approach:

• Phase 1 (Gate): Stop accepting new requests
• Phase 2 (Drain): Wait for in-flight requests to complete (up to configured timeout)
• Phase 3 (Teardown): MCP orchestrator cleanup + exit

Impact: Requests finishing in 2s no longer wait 28s for a fixed grace period. Requests needing 35s no longer get killed at 30s. The system drains to zero when possible.

Worker Registry & REST API Improvements

• Model field required (#713) -- Clients omitting model now get 400 Bad Request instead of silent "unknown" injection. Matches OpenAI API spec. Breaking change.
• REST semantics (#875) -- POST /workers (create-only, 409 on conflict), PUT /workers/{id} (full replace), PATCH /workers/{id} (partial update). Breaking change: PUT now requires WorkerSpec instead of WorkerUpdateRequest.
• Split register paths (#836) -- register() (create-only), replace() (overwrite-then-diff, no transient gap), register_or_replace() (idempotent upsert)

vLLM gRPC Embedding Support

End-to-end embedding pipeline for vLLM via gRPC:

• Rust gateway + Python servicer (calls engine.encode() with PoolingParams)
• Flattened SGLang EmbedResponse proto (removed oneof, uses tonic::Status for errors)
• Removed SGLang-specific log_metrics and cached_tokens from embed/classify protos

DeepSeek V3.1 Tool Call Parser

Native parser for DeepSeek V3.1's tool calling format:

• Handles V3.1's simplified format (no function type prefix, no markdown code blocks)
• Complete + streaming (parse_incremental) support
• Auto-registered for deepseek-v3.1* and deepseek-ai/DeepSeek-V3.1* model patterns
• E2E validated against live DeepSeek V3.1 (FP8) on 8×H200

Additional Features

• Configurable storage hook context (#807) -- Map HTTP headers to storage hook request context via storage_context_headers
• Conversation memories schema (#976) -- First-class conversation_memories table in data-connector with Oracle Flyway DDL and insert seam
• gRPC health checking (#885) -- Standard grpc.health.v1 health service for vLLM workers
• Model metadata in GetModelInfo (#871) -- vLLM GetModelInfo RPC now returns model metadata fields
• Metrics server refactored to axum (#966) -- Foundation for /ws/metrics WebSocket endpoint
• max_total_num_tokens in GetServerInfo (#817) -- Aligns gRPC response with HTTP server

Performance Improvements

• Tokenizer: Optimized stop decoder and incremental sequence decoding (#990)
• Routing: Optimized extract_text_for_routing string handling (#967)
• Mesh: Eliminated per-request CRDT serialization in sync_tree_operation (#948)
• Multimodal: Thread-local Resizer reuse (#923), eliminated DynamicImage clones (#928), optimized serialization and tensor conversion (#1012)

Bug Fixes

• Multimodal: Fixed Phi-3-vision for string-format chat templates (#942), LLaVA-Next anyres multi-crop for vLLM gRPC (#941), hardened registry matching and token geometry (#945), propagated placeholder resolution errors (#943), fixed images smaller than patch_size × merge_size (#908), use preprocessor token counts in LlavaSpec (#958), fall back to config.model_type for aliased model IDs (#898)
• Chat Templates: Inject special tokens (bos_token, eos_token) into chat template context (#914), correct content format detection for Qwen3-style templates (#981), inject special tokens inside tokenizer impls (#918)
• Protocol: Accept null for boolean fields logprobs and stream (#1020), validate reasoning parser name at CLI and startup (#901)
• gRPC: Fix assistant tool_calls message serialization for chat templates (#1023), include stop tokens in TRT-LLM output for Harmony parsing (#879), handle vllm log forwarding on servicer side (#975)
• Mesh: Stop advertising 0.0.0.0 to peers (#883), set tonic message size limits to match application limit (#893), prevent duplicate store events from inflating tree_sizes (#946)
• Gateway: Update metric when removing unhealthy workers (#884), filter empty-string backend defaults in CLI arg fallback (#934)
• Responses API: Align store=false state persistence behavior (#916)
• Serve: Respect user-set CUDA_VISIBLE_DEVICES in gp...

v1.3.3

🚀 Shepherd Model Gateway v1.3.3 Released

Major performance release with 7x faster mesh synchronization and critical bug fixes.

⚡ Mesh Performance Revolution

Switched mesh serialization from JSON to bincode with dramatic performance improvements:

Benchmark Results (production workload - 1024 operations, 4000 tokens):

• Serialization: 7.1x faster (35.5ms → 5.0ms)
• Deserialization: 14.8x faster (63.4ms → 4.3ms)
• Wire size: 4.3x smaller (67.9MB → 15.7MB)
• Multi-model aggregate (10 models): 4.3x smaller (679MB → 157MB)

Additional mesh improvements:

• Operation log auto-compaction and tombstone GC
• Skip full-store scans when nothing has changed
• Prevent stale snapshot chunks from mixing across retries
• Break infinite retry loop for oversized incremental updates

Impact: Massive reduction in network bandwidth and CPU usage for multi-node deployments. Mesh state synchronization is now 7-15x faster with 4.3x less bandwidth consumption.

🎯 Structured Output Support

response_format support in Chat Completions API for Harmony models:

• JSON schema constrained output
• Structured generation for tool calling and data extraction
• Fixed structural tag triggers for json_schema mode

🔧 PD Disaggregation Improvements

Enhanced reliability for prefill-decode mode:

• Abort both PD requests when one side hits transport error (prevents hanging requests)
• Handle mismatched metric labels in PD disaggregation mode
• Fixed classify race condition with URL-based detection

🐛 Bug Fixes

• Protocol: Validate /v1/messages tool_choice contract
• Harmony: Include developer message when instructions are present
• Gateway: Disable auto-detection if "runtime": "sglang" explicitly set
• Client: Auto-close streaming responses on iteration exhaustion
• Docker: Install gRPC proto and servicer for vLLM images

📚 Documentation Overhaul

Comprehensive audit and fixes across all documentation:

• Quickstart and getting-started guides
• Worker configuration and gRPC pipeline
• Tokenizer, MCP, and WASM plugin extensibility
• PD disaggregation and cache-aware routing
• Reliability features and monitoring
• Configuration, metrics, and architecture
• API reference documentation

🏗️ Infrastructure

• Default engine versions: vLLM 0.18.0, TensorRT-LLM 1.3.0rc8
• Added minimaxai/minimax-m2 to nightly benchmarks
• Improved E2E test infrastructure with parametrized fixtures

Full Changelog: v1.3.2...v1.3.3

Upgrade now: pip install smg --upgrade

🐑 Shepherd your LLM infrastructure with confidence.

Docker Images

Pre-built engine images on GitHub Container Registry:

SGLang:

docker pull ghcr.io/lightseekorg/smg:1.3.3-sglang-v0.5.9

vLLM:

docker pull ghcr.io/lightseekorg/smg:1.3.3-vllm-v0.18.0

TensorRT-LLM:

docker pull ghcr.io/lightseekorg/smg:1.3.3-trtllm-1.3.0rc8

All images for v1.3.3

Engine	Tag	Pull Command
sglang	1.3.3-sglang-v0.5.9	docker pull ghcr.io/lightseekorg/smg:1.3.3-sglang-v0.5.9
trtllm	1.3.3-trtllm-1.3.0rc8	docker pull ghcr.io/lightseekorg/smg:1.3.3-trtllm-1.3.0rc8
vllm	1.3.3-vllm-v0.18.0	docker pull ghcr.io/lightseekorg/smg:1.3.3-vllm-v0.18.0

What's Changed

• chore(release): bump llm-multimodal to 1.4.0 by @slin1237 in #788
• fix(harmony): fix structural tag triggers for json_schema constrained output by @CatherineSue in #789
• feat(harmony): support response_format in Chat Completions by @CatherineSue in #791
• refactor(e2e): reorganize Harmony tests, add validation, remove unused gateway args by @CatherineSue in #796
• chore(deps): bump dorny/paths-filter from 3 to 4 by @dependabot[bot] in #792
• fix(ci): fix mergify stale/close rules that never trigger by @CatherineSue in #798
• fix(ci): use ignore-pr-updates for stale PR detection, disable issues by @CatherineSue in #800
• chore(deps): update tokio-tungstenite requirement from 0.28 to 0.29 by @dependabot[bot] in #793
• feat(core): add per-worker resilience and HTTP pool config types by @CatherineSue in #799
• fix(ci): remove ignore-pr-updates that marks active PRs as stale by @CatherineSue in #804
• refactor(openai): cleanup dead code, redundant state, and hot-path inefficiencies by @slin1237 in #802
• fix(mesh): break infinite retry loop for oversized incremental updates by @slin1237 in #808
• feat(core): wire per-worker resilience and HTTP client into BasicWorker by @CatherineSue in #803
• test(mesh): add serialization benchmark for mesh state sync by @slin1237 in #810
• test(e2e): re-enable skipped tests for vLLM and TRT-LLM by @CatherineSue in #806
• perf(mesh): switch all mesh serialization from JSON to bincode by @slin1237 in #809
• fix(mesh): use bincode for snapshot generation to match receivers by @slin1237 in #816
• feat(gateway): Propagate otel context for distributed tracing by @ekzhang in #814
• perf(mesh): skip full-store scans when nothing has changed by @slin1237 in #823
• fix(ci): drop [grpc] extra from nightly vllm install by @CatherineSue in #826
• feat(ci): add minimaxai/minimax-m2 to nightly benchmark by @smfirmin in #795
• fix(gateway): Disable auto-detection if "runtime": "sglang"explciitly set by @ekzhang in #820
• perf(mesh): add operation log auto-compaction and tombstone GC by @slin1237 in #825
• refactor(e2e): replace smg_compare with parametrized api_client fixture by @CatherineSue in #812
• fix(client): auto-close streaming responses on iteration exhaustion by @CatherineSue in #835
• refactor(e2e): add model fixture, remove deprecated smg fixture by @CatherineSue in #834
• fix(protocol): validate /v1/messages tool_choice contract by @nishanthp in #833
• fix(mesh): prevent stale snapshot chunks from mixing across retries by @slin1237 in #837
• test(mesh): improve benchmark summary with timing and side-by-side comparison by @slin1237 in #841
• fix(docker): install gRPC proto and servicer for vLLM images by @slin1237 in #843
• fix(gateway): use URL-based detection to eliminate classify race condition by @slin1237 in #839
• ci: bump default engine vllm(0.18.0) and trt(1.3.0rc8) versions by @slin1237 in #845
• fix(gateway): handle mismatched metric labels in PD disaggregation mode by @slin1237 in #846
• fix(pd): abort both PD requests when one side hits a transport error by @slin1237 in #844
• docs(quickstart): audit and fix getting-started documentation by @slin1237 in #848
• docs(extensibility): audit and fix tokenizer, MCP, and WASM plugin documentation by @slin1237 in #849
• docs(workers): audit and fix worker configuration and gRPC pipeline documentation by @slin1237 in #850
• docs(reliability): audit and fix reliability feature documentation by @slin1237 in #851
• docs(operations): audit and fix monitoring and data connection documentation by @slin1237 in #854
• docs(routing): audit and fix PD disaggregation and cache-aware routing documentation by @slin1237 in #853
• docs(config): audit and fix configuration, metrics, and architecture documentation by @slin1237 in #852
• docs(api): audit and fix API reference documentation by @slin1237 in #855
• fix(ci): scope VERSION_OVERRIDE to smg crate only by @slin1237 in #856
• chore(release): bump version to 1.3.3 by @slin1237 in #857

New Contributors

• @smfirmin made their first contribution in #795
• @nishanthp made their first contribution in #833

Full Changelog: v1.3.2...v1.3.3

v1.3.2

🚀 Shepherd Model Gateway v1.3.2 Released

Feature release adding multimodal support to Messages API and Python mesh bindings.

🎨 Multimodal Support for Messages API

Complete vision/image support in Messages API gRPC pipeline:

• Native image processing for Messages API requests
• Works across all gRPC backends (SGLang, vLLM, TensorRT-LLM)
• Full feature parity with Anthropic's Messages API including vision

Impact: Messages API now supports both text and vision workloads. Deploy vision-language models with full reasoning and thinking capabilities through the Messages API protocol.

🌐 Mesh High Availability in Python

--enable-mesh support added to Python bindings:

• Configure mesh HA directly from Python CLI
• Distributed state synchronization accessible from Python deployments
• Complete Python API coverage for mesh features

🐛 Bug Fixes

• API: Restored model_id field in /workers response
• Harmony: Reject ignore_eos with HTTP 400 for compatibility
• Harmony: Include developer message when instructions are present
• Lossy UTF-8 decode fallback for malformed text
• Fixed gRPC PD mode detection
• Enabled loop_controls for Jinja2 templates

📚 Documentation

• Added comprehensive Messages API documentation
• External providers integration guide
• Mesh HA deployment documentation
• Docker/PyPI badges added to README

Full Changelog: v1.3.1...v1.3.2

Upgrade now: pip install smg --upgrade

🐑 Shepherd your LLM infrastructure with confidence.

What's Changed

• fix(api): restore model_id field in /workers response by @slin1237 in #774
• feat(python): add --enable-mesh support to Python bindings by @slin1237 in #775
• fix(ci): bump NCCL to 2.28+ for TensorRT-LLM compatibility by @slin1237 in #777
• fix(harmony): reject ignore_eos for Harmony models with HTTP 400 by @CatherineSue in #778
• fix(ci): install NCCL 2.28+ after TRT-LLM requirements to prevent downgrade by @slin1237 in #779
• fix(harmony): include developer message when instructions are present by @CatherineSue in #781
• feat(gateway): add multimodal support to Messages API gRPC pipeline by @slin1237 in #776
• fix(ci): use k8s pod env vars for API keys instead of GitHub secrets by @slin1237 in #782
• feat(tokenizer): lossy UTF-8 decode fallback, fix gRPC PD mode detection, enable loop_controls by @Kangyan-Zhou in #769
• docs: add Messages API, external providers, and mesh HA documentation by @slin1237 in #786
• docs: add Docker/PyPI badges and release docker notes script by @slin1237 in #787
• chore: add release 1.3.2 by @slin1237 in #780

New Contributors

• @Kangyan-Zhou made their first contribution in #769

Full Changelog: v1.3.1...v1.3.2

v1.3.1

🚀 Shepherd Model Gateway v1.3.1 Released

Minor release with operational improvements and bug fixes.

🛠️ New Features

Operational improvements:

• --remove-unhealthy-workers flag - Automatically remove workers that fail health checks
• disable_tokenizer_autoload support - Skip automatic tokenizer loading for custom configurations

🐛 Bug Fixes

• Gateway: Index external workers by all discovered models (not just primary model)
• CI: Added VERSION_OVERRIDE to version check script

Full Changelog: v1.3.0...v1.3.1

Upgrade now: pip install smg --upgrade

🐑 Shepherd your LLM infrastructure with confidence.

What's Changed

• fix(deps): remove aws-lc-sys to fix aarch64 PyPI build by @slin1237 in #762
• feat(gateway): add --remove-unhealthy-workers by @ekzhang in #714
• feat(gateway): support disable_tokenizer_autoload by @Huixxi in #740
• feat(messages): add unit tests for Messages API streaming & response … by @ConnorLi96 in #763
• fix(gateway): index external workers by all discovered models by @zhaowenzi in #756
• fix(ci): add VERSION_OVERRIDE to check-versions by @slin1237 in #764
• feat(interactions): Implement steps to handle non-stream interactions req without tool call by @XinyueZhang369 in #723
• chore: auto-close stale PRs after 30 days and validate DCO sign-off identity by @CatherineSue in #743
• chore(release): bump version to 1.3.1 by @slin1237 in #767

New Contributors

• @Huixxi made their first contribution in #740

Full Changelog: v1.3.0...v1.3.1