v1.9.0

🚀 Shepherd Model Gateway v1.9.0 Released

A focused release headlined by Kimi K3 support and tenant rate limiting — extending the multi-tenancy foundation from v1.8.0 — plus Responses API in PD mode, batched completion prompts, model aliases, and an engine-version bump across all four backends.

🤖 Kimi K3 Support

Full end-to-end integration for Kimi K3 across text, vision, tools, and reasoning:

• Dedicated vision processor — K3 gets its own multimodal image processor, with correct modality ordering
• Tool calling — Forced tool calls correctly close the prefilled channel
• Thinking-effort defaults — Media wrapper and thinking-effort resolution handled at the prompt-encoding layer

🔀 Responses API in PD Mode

/v1/responses now serves under prefill-decode disaggregation:

• Full Responses API support in the PD gRPC pipeline
• Batched completion prompts supported in the pipeline

🏷️ Model Aliases

End-to-end model alias support:

• Refer to a model by an alias throughout the request path
• Aliases resolve consistently from API boundary to backend

🔍 Request ID Propagation

Backend request IDs now derived from the request ID (rid) and the middleware request ID:

• Correlate a request across the gateway→backend boundary
• Consistent IDs for tracing and debugging

🤖 Engine Version Bumps

All four backends updated:

• vLLM — Floor raised to 0.26, Docker matrix refreshed
• SGLang — Bumped to 0.5.16
• TensorRT-LLM — Bumped to 1.3.0rc22
• TokenSpeed — Bumped to latest

🔧 Engine Version Automation

New CI automation to keep pace with upstream engine releases:

• Engine version watch — Automated detection of new engine versions, running weekly
• Nightly failure triage — Automated triage of nightly benchmark failures

🐛 Notable Fixes

• Tool parser: Ignore <|END_ACTION|> inside Cohere JSON strings (no more false stops mid-argument)
• Protocols: Accept function tools without a parameters field
• MCP: Preserve explicit tool_choice for the initial responses request
• Gateway: Apply PUT /workers/{id} instead of failing in the background, reject schemeless worker URLs at the API boundary
• Gateway: Scrape engine metrics via base URL under --dp-aware
• Scheduler: Retain worker capacity tracking
• gRPC: Load Harmony encoding at gpt-oss worker registration

📦 Dependencies & Infrastructure

• Dependencies: tokio-tungstenite 0.29→0.30, wasm-encoder 0.253→0.254, plus GitHub Actions bumps (setup-python, setup-go, labeler, stale)
• CI: Repaired crate publication order, aligned TokenSpeed with the promoted EPD package
• Docs: Documented XGrammar and fixed its pipeline order in the gRPC pipeline docs

🙏 Welcome New Contributors

3 first-time contributors landed in this release — thank you all!

Full Changelog: v1.8.0...v1.9.0

Upgrade now: pip install smg --upgrade

🐑 Frontier models. Production-proven.

What's Changed

• fix(ci): repair crate publication order by @slin1237 in #1940
• fix(tool_parser): ignore <|END_ACTION|> inside Cohere JSON strings by @santhreal in #1941
• fix(tokenspeed): align with promoted EPD package by @lightseek-bot in #1944
• chore(deps): update wasm-encoder requirement from 0.253 to 0.254 by @dependabot[bot] in #1953
• chore(deps): bump actions/setup-python from 6 to 7 by @dependabot[bot] in #1950
• chore(deps): bump actions/setup-go from 6 to 7 by @dependabot[bot] in #1948
• chore(deps): bump actions/labeler from 6 to 7 by @dependabot[bot] in #1949
• chore(deps): update tokio-tungstenite requirement from 0.29 to 0.30 by @dependabot[bot] in #1954
• feat(grpc): serve /v1/responses in PD mode by @slin1237 in #1956
• feat(grpc): support batched completion prompts in the pipeline by @slin1237 in #1957
• fix(grpc): load Harmony encoding at gpt-oss worker registration by @slin1237 in #1959
• feat(rate_limit): add tenant rate-limit policy schema and compilation by @XinyueZhang369 in #1958
• feat(gateway): derive backend request ids from rid and the middleware request id by @slin1237 in #1960
• fix(mcp): preserve explicit tool_choice for the initial responses request by @slin1237 in #1963
• fix(gateway): apply PUT /workers/{id} instead of failing in the background by @junliu-mde in #1967
• feat(rate_limit): add reserve/settle engine by @XinyueZhang369 in #1966
• fix(router): support model aliases end to end by @junliu-mde in #1964
• feat(kimi-k3): add K3 support by @lightseek-bot in #1968
• chore(deps): bump vLLM floor to 0.26 and refresh docker matrix by @slin1237 in #1971
• docs(grpc-pipeline): document XGrammar and fix its pipeline order by @ighutake-debug in #1976
• feat(rate_limit): wire tenant rate limiter into CLI and AppContext by @XinyueZhang369 in #1969
• fix(protocols): accept function tools without parameters field by @ighutake-debug in #1975
• chore(deps): bump SGLang to 0.5.16 by @slin1237 in #1970
• chore(deps): bump TensorRT-LLM to 1.3.0rc22 by @slin1237 in #1972
• ci: add engine version watch and nightly failure triage automation by @slin1237 in #1973
• ci: run engine version watch weekly by @slin1237 in #1981
• chore(deps): bump TokenSpeed to 0f68676 by @slin1237 in #1982
• fix(kimi-k3): let forced tool calls close the prefilled channel by @key4ng in #1983
• chore(deps): bump actions/stale from 10 to 11 by @dependabot[bot] in #1985
• fix(multimodal): give Kimi-K3 its own vision processor and fix alpha ordering by @key4ng in #1984
• fix(kimi-k3): move the media wrapper and thinking-effort default to the prompt-encoding layer by @key4ng in #1995
• fix(gateway): scrape engine metrics via base URL under --dp-aware by @ighutake-debug in #1994
• fix(scheduler): retain worker capacity tracking by @ShiroKSH in #1997
• fix(gateway): reject schemeless worker URLs at the API boundary by @ighutake-debug in #1977
• chore(release): bump versions for v1.9.0 by @slin1237 in #1996

New Contributors

• @santhreal made their first contribution in #1941
• @ighutake-debug made their first contribution in #1976
• @ShiroKSH made their first contribution in #1997

Full Changelog: v1.8.0...v1.9.0