Skip to content

v1.6.0

Choose a tag to compare

View all tags
@slin1237 slin1237 released this 21 Jun 02:50
· 359 commits to main since this release
v1.6.0
d4e082f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

🚀 Shepherd Model Gateway v1.6.0 Released

A performance and production-hardening release: unified SSE codec across every streaming path, KV cache events for vLLM and TokenSpeed, routing hot-path optimizations, multimodal optimizations, and graceful shutdown hardening.

⚡ Streaming Performance: Unified SSE Codec

Every streaming path now runs through a single shared SSE codec — no more per-router SSE parsing and re-encoding:

Anthropic Messages — Shared SseDecoder for upstream, SseEncoder for output
OpenAI Responses — Shared encoder for streaming output, shared codec for block parsing
gRPC-regular and gRPC-harmony — Shared encoder for inline SSE framing
HTTP PD — Shared encoder for logprob-merge SSE re-encode

Impact: Consistent, optimized SSE handling across the entire gateway. One codec, tested once, used everywhere — lower CPU overhead on streaming responses and a single place to maintain the protocol.

🧠 Event-Driven Cache-Aware Routing: vLLM + TokenSpeed

KV cache event support extended to two more backends via the SubscribeKvEvents bridge:

vLLM KV cache events — Real-time cache state for cache-aware routing
TokenSpeed KV cache events — Same event-driven routing for TokenSpeed deployments
Removed the 2048-worker cap — Scale cache-aware routing past 2048 workers
Surface KV subscription failures — No more silent failures in the event pipeline

Impact: Cache-aware routing now learns live cache state from vLLM and TokenSpeed, not just SGLang — routing requests to the worker most likely to have a warm prefix. Plus large-fleet deployments are no longer capped at 2048 workers.

🗺️ Cache-Aware & Worker Routing Performance

Lower per-request routing overhead on the hot path:

Single-pass routing state — Gather all worker routing state in one pass instead of repeated scans per decision
Filter without wasted clones — Worker filtering clones only matching workers, not the entire fleet
Decode-faster-than-prefill imbalance fixed — Cache-aware routing no longer skews load when decode outpaces prefill

Impact: Cheaper routing decisions per request and less allocation pressure under high worker counts — compounding with the 2048-worker cap removal above.

🎨 Multimodal Optimizations

Faster vision processing and a generalized multimodal ABI (contributed by NVIDIA):

Multimodal ABI generalization — Cleaner backend-agnostic multimodal interface
Optimized video decode and Qwen VL preprocessing — Faster multimodal request handling
Qwen2-VL preprocessor_config overrides — Honor model-specific preprocessing config
OpenCV video feature — Setup script for the opencv-video build feature

Impact: Lower latency on vision and video workloads, with a cleaner multimodal interface that generalizes across backends. Thanks to @yechank-nvidia for the contributions.

🔌 Tokenizer Performance

Shared L1 prefix on cache hit — Fused miss-path tokenization, less redundant work
Offload encode to a bounded blocking pool — Tokenizer encode no longer blocks the async runtime
Encode L1 cache suffix without special tokens — Correct cache key derivation

🛡️ Graceful Shutdown Hardening

Production-grade drain and shutdown behavior:

Keep accepting during LB propagation window — No dropped requests while load balancers learn the node is draining
Drain in-flight gRPC RPCs before closing ZMQ — Clean gRPC servicer shutdown
O(1) readiness checks — Plus optional dedicated probe listener (--health-check-port) and drain-aware probes

Impact: Zero-downtime deployments. Nodes drain cleanly, in-flight requests complete, and dedicated health-check ports keep probes responsive even under load.

⚖️ DP-Aware Routing for vLLM

Data-parallel routing for vLLM gRPC workers:

Rank pinning for DP workers
Mooncake DP mint for PD disaggregation

🎯 Passthrough Routing Policy

New passthrough routing policy for single-backend serving — skip routing overhead entirely when there's only one backend.

📊 Observability

Tokio event-loop canary — Runtime metrics sampler to detect event-loop stalls
Skip metrics collection when no client is subscribed — No overhead when nobody's watching
Removed /ws/metrics WebSocket endpoint — Retired in favor of standard metrics

💬 Messages API

Accept system role in messages[] and forward in place on /v1/messages

🧩 WASM

--enable-wasm flag added to the Python CLI and Helm chart

🧪 Testing Infrastructure

No-GPU routing experiments and continuous quality benchmarks:

Mock worker fleet — Multi-port mock HTTP/gRPC worker and realistic engine simulator for routing A/B testing without GPUs
Nightly BFCL benchmarks — SMG frontend vs pure vLLM across non_live, live, and multi_turn (verifying SMG adds no quality regression)

🐛 Notable Fixes

Tool parser: MiniMax-M2 unknown tool names forwarded instead of leaking markup, args coerced by declared schema type, byte-offset and char-boundary correctness for partial JSON
gRPC: TRT-LLM health classified by exact OK status (not substring), SGLang token IDs sent as array("q") to match contract
Responses: function_call serde alignment
TokenSpeed: Allow idle gRPC keepalive pings
Gateway: Valid-JSON SSE error frame on PD streaming decode error, HTTP metrics path label bounded to matched route

🧹 Cleanup

Removed the smg-tui crate and dependencies
Dead code and stale comment cleanup across auth, WASM, MCP, multimodal, protocols, and more

🏗️ Runtime & Infrastructure

Slimmed runtime config to --runtime-worker-threads override with container-aware defaults
Lazy-load generated gRPC proto re-exports
CI: bin-packed GPU runners via Volcano, RDMA for Mooncake PD

🙏 Welcome New Contributors

7 first-time contributors landed in this release — thank you all!

Full Changelog: v1.5.0...v1.6.0

Upgrade now: pip install smg --upgrade

🐑 Shepherd your LLM infrastructure with confidence.

What's Changed

  • test(mock-worker): add multi-port mock HTTP/gRPC worker and scale-test rig by @slin1237 in #1696
  • fix(model_gateway): bound HTTP metrics path label to matched route by @slin1237 in #1679
  • chore(routers/http): remove dead code and trim stale comments by @slin1237 in #1651
  • perf(observability): skip metrics_ws collection when no client is subscribed by @slin1237 in #1687
  • fix(mock-worker): keep the fleet running when one worker exits by @slin1237 in #1697
  • fix(tool_parser): make partial JSON consumed position a byte offset by @slin1237 in #1677
  • fix(grpc): classify TRT-LLM health by exact OK status, not substring by @slin1237 in #1681
  • feat(grpc): add vLLM KV cache event support (SubscribeKvEvents bridge) by @key4ng in #1652
  • fix(tokenizer): encode L1 cache suffix without special tokens by @slin1237 in #1680
  • refactor(multimodal): Multimodal ABI generalization by @yechank-nvidia in #1602
  • revert(ci): use NVMe storage for H100 runner workspaces by @key4ng in #1704
  • fix(ci): fix 2-gpu runner for RDMA by @XinyueZhang369 in #1705
  • perf(worker): clone only matching workers in get_workers_filtered by @slin1237 in #1699
  • ci(release-docker): fail loud on push detection and add run summary by @key4ng in #1710
  • perf(smg): mitigate the worker-sync lag-resync burst by @CatherineSue in #1667
  • perf(mesh): bound op-log memory at O(live keys) and share round snapshots by @CatherineSue in #1671
  • fix(ci): lower vllm mooncake pd mmlu concurrency by @key4ng in #1720
  • fix(ci): make Oracle test users unique by @key4ng in #1719
  • fix(ci): pin CUDA 13 CUTLASS DSL libs by @key4ng in #1717
  • chore(tui): remove smg-tui crate and related dependencies by @key4ng in #1718
  • ci: cancel PR workflows on close by @key4ng in #1723
  • ci: increase gpu timeout budgets by @key4ng in #1721
  • feat(grpc): DP-aware routing for vLLM gRPC workers (rank pinning + Mooncake DP mint) by @CatherineSue in #1673
  • fix(tool_parser): iterate char boundaries in ends_with_partial_token by @slin1237 in #1676
  • perf(tokenizer): share L1 prefix on hit and fuse miss-path tokenization by @slin1237 in #1709
  • fix(kv_index): remove the 2048-worker cap and surface KV subscription failures by @slin1237 in #1706
  • ci: use rdma for mooncake pd by @key4ng in #1722
  • chore(ci): temporarily disable nightly MLX benchmark schedule by @key4ng in #1726
  • fix(grpc/sglang): send token IDs as array("q") to match SGLang contract by @gongwei-130 in #1725
  • fix(ci): pin FastAPI below 0.137 for vLLM by @key4ng in #1727
  • feat(observability): add tokio event-loop canary and runtime metrics sampler by @slin1237 in #1707
  • feat(router): add --enable-wasm flag to Python CLI and Helm chart by @neverCase in #1729
  • fix(model_gateway): emit valid-JSON SSE error frame on PD streaming decode error by @slin1237 in #1682
  • feat(multimodal): optimize video decode and Qwen VL preprocessing by @yechank-nvidia in #1603
  • ci: run build-wheel and radix-tree benchmark on the CPU runner; let CPU runners use GPU-node CPU by @slin1237 in #1730
  • chore: add OpenCV setup script for the opencv-video feature by @slin1237 in #1731
  • perf(server): O(1) readiness + optional dedicated probe listener (--health-check-port), drain-aware probes by @slin1237 in #1711
  • feat(mock-worker): realistic engine simulator for no-GPU routing A/B by @slin1237 in #1713
  • perf(anthropic): decode upstream SSE via shared SseDecoder by @XinyueZhang369 in #1634
  • chore: remove dead code and trim narrative comments across 4 crates by @slin1237 in #1732
  • fix(responses): align function_call serde by @shenoyvvarun in #1601
  • chore: audit cleanup batch 2 — dead code + comment trims (auth, wasm) by @slin1237 in #1733
  • chore(observability): remove /ws/metrics WebSocket endpoint and collectors by @key4ng in #1740
  • feat(bfcl): nightly BFCL A/B — SMG frontend vs pure vLLM by @key4ng in #1724
  • fix(tool-parser): forward unknown MiniMax-M2 tool names instead of leaking markup by @aistackdev in #1728
  • chore(workflow): trim narrative comments in the engine by @slin1237 in #1735
  • chore(protocols): trim narrative/stale comments by @slin1237 in #1736
  • chore(mesh): remove dead mTLS alias + unused field, trim comments by @slin1237 in #1737
  • chore(data-connector): remove dead ResponseChain methods + duplicate migration arrays by @slin1237 in #1739
  • chore(mcp): remove orphaned oauth module + dead contains_url by @slin1237 in #1741
  • chore(reasoning-parser): remove unused pooling API, trim comments by @slin1237 in #1742
  • fix(server): keep accepting during LB propagation window on shutdown by @slin1237 in #1745
  • fix(grpc_servicer): drain in-flight RPCs before closing ZMQ on shutdown by @slin1237 in #1744
  • chore(multimodal): remove dead transforms/accessors, trim comments by @slin1237 in #1738
  • perf(grpc): offload tokenizer encode to a bounded blocking pool by @slin1237 in #1746
  • perf(anthropic): use shared SseEncoder for streaming SSE output by @XinyueZhang369 in #1633
  • fix(cache-aware): fix load imbalance when decode is faster than prefill by @SYChen123 in #1714
  • chore(deps): update tower-http requirement from 0.6 to 0.7 by @dependabot[bot] in #1757
  • chore(deps): update pyo3 requirement from 0.28.2 to 0.29.0 by @dependabot[bot] in #1756
  • chore(deps): update wasm-encoder requirement from 0.251 to 0.252 by @dependabot[bot] in #1754
  • perf(cache-aware): gather worker routing state in one pass by @slin1237 in #1758
  • fix(tool-parser): coerce minimax_m2 args by declared schema type by @slin1237 in #1761
  • fix(multimodal): honor Qwen2-VL preprocessor_config overrides by @slin1237 in #1752
  • fix(ci): pin grpcio generated-code companions to the protobuf-6 stable line by @key4ng in #1765
  • fix(tokenspeed): allow idle gRPC keepalive pings by @FlamingoPg in #1766
  • fix(ci): allocate BFCL arm ports dynamically to avoid host-port collisions by @key4ng in #1763
  • ci: bin-pack GPU CI runners via Volcano + sync runner values with live cluster by @key4ng in #1762
  • feat(bfcl): nightly covers non_live + live + multi_turn by @key4ng in #1764
  • perf(openai-responses): use shared SseEncoder for streaming SSE output by @XinyueZhang369 in #1759
  • perf(openai-responses): parse SSE blocks via shared codec by @XinyueZhang369 in #1760
  • fix(ci): key main concurrency on commit SHA so every commit runs by @key4ng in #1767
  • fix(grpc-proto): lazy-load generated re-exports by @lightseek-bot in #1769
  • perf(http-pd): use shared SseEncoder for logprob-merge SSE re-encode by @XinyueZhang369 in #1768
  • chore(grpc): raise Python grpc dependency floor by @lightseek-bot in #1785
  • perf(runtime): slim to --runtime-worker-threads override; rely on container-aware default (#1695) by @slin1237 in #1786
  • feat(grpc): add TokenSpeed KV cache event support (SubscribeKvEvents bridge) by @key4ng in #1771
  • perf(grpc-harmony): use shared SseEncoder for streaming SSE output by @XinyueZhang369 in #1783
  • perf(grpc-regular): use shared SseEncoder for inline SSE framing by @XinyueZhang369 in #1772
  • chore(git): update CODEOWNERS by @lightseek-bot in #1790
  • fix(messages): accept system role in messages[] and forward in place (/v1/messages) by @qywu in #1796
  • feat(policies): add passthrough routing policy for single-backend serving by @slin1237 in #1797
  • chore: bump versions for v1.6.0 release by @slin1237 in #1792

New Contributors

Full Changelog: v1.5.0...v1.6.0

Contributors

shenoyvvarun, CatherineSue, and 12 other contributors
Assets 2
Loading