v1.3.3

🚀 Shepherd Model Gateway v1.3.3 Released

Major performance release with 7x faster mesh synchronization and critical bug fixes.

⚡ Mesh Performance Revolution

Switched mesh serialization from JSON to bincode with dramatic performance improvements:

Benchmark Results (production workload - 1024 operations, 4000 tokens):

• Serialization: 7.1x faster (35.5ms → 5.0ms)
• Deserialization: 14.8x faster (63.4ms → 4.3ms)
• Wire size: 4.3x smaller (67.9MB → 15.7MB)
• Multi-model aggregate (10 models): 4.3x smaller (679MB → 157MB)

Additional mesh improvements:

• Operation log auto-compaction and tombstone GC
• Skip full-store scans when nothing has changed
• Prevent stale snapshot chunks from mixing across retries
• Break infinite retry loop for oversized incremental updates

Impact: Massive reduction in network bandwidth and CPU usage for multi-node deployments. Mesh state synchronization is now 7-15x faster with 4.3x less bandwidth consumption.

🎯 Structured Output Support

response_format support in Chat Completions API for Harmony models:

• JSON schema constrained output
• Structured generation for tool calling and data extraction
• Fixed structural tag triggers for json_schema mode

🔧 PD Disaggregation Improvements

Enhanced reliability for prefill-decode mode:

• Abort both PD requests when one side hits transport error (prevents hanging requests)
• Handle mismatched metric labels in PD disaggregation mode
• Fixed classify race condition with URL-based detection

🐛 Bug Fixes

• Protocol: Validate /v1/messages tool_choice contract
• Harmony: Include developer message when instructions are present
• Gateway: Disable auto-detection if "runtime": "sglang" explicitly set
• Client: Auto-close streaming responses on iteration exhaustion
• Docker: Install gRPC proto and servicer for vLLM images

📚 Documentation Overhaul

Comprehensive audit and fixes across all documentation:

• Quickstart and getting-started guides
• Worker configuration and gRPC pipeline
• Tokenizer, MCP, and WASM plugin extensibility
• PD disaggregation and cache-aware routing
• Reliability features and monitoring
• Configuration, metrics, and architecture
• API reference documentation

🏗️ Infrastructure

• Default engine versions: vLLM 0.18.0, TensorRT-LLM 1.3.0rc8
• Added minimaxai/minimax-m2 to nightly benchmarks
• Improved E2E test infrastructure with parametrized fixtures

Full Changelog: v1.3.2...v1.3.3

Upgrade now: pip install smg --upgrade

🐑 Shepherd your LLM infrastructure with confidence.

Docker Images

Pre-built engine images on GitHub Container Registry:

SGLang:

docker pull ghcr.io/lightseekorg/smg:1.3.3-sglang-v0.5.9

vLLM:

docker pull ghcr.io/lightseekorg/smg:1.3.3-vllm-v0.18.0

TensorRT-LLM:

docker pull ghcr.io/lightseekorg/smg:1.3.3-trtllm-1.3.0rc8

All images for v1.3.3

Engine	Tag	Pull Command
sglang	1.3.3-sglang-v0.5.9	docker pull ghcr.io/lightseekorg/smg:1.3.3-sglang-v0.5.9
trtllm	1.3.3-trtllm-1.3.0rc8	docker pull ghcr.io/lightseekorg/smg:1.3.3-trtllm-1.3.0rc8
vllm	1.3.3-vllm-v0.18.0	docker pull ghcr.io/lightseekorg/smg:1.3.3-vllm-v0.18.0

What's Changed

• chore(release): bump llm-multimodal to 1.4.0 by @slin1237 in #788
• fix(harmony): fix structural tag triggers for json_schema constrained output by @CatherineSue in #789
• feat(harmony): support response_format in Chat Completions by @CatherineSue in #791
• refactor(e2e): reorganize Harmony tests, add validation, remove unused gateway args by @CatherineSue in #796
• chore(deps): bump dorny/paths-filter from 3 to 4 by @dependabot[bot] in #792
• fix(ci): fix mergify stale/close rules that never trigger by @CatherineSue in #798
• fix(ci): use ignore-pr-updates for stale PR detection, disable issues by @CatherineSue in #800
• chore(deps): update tokio-tungstenite requirement from 0.28 to 0.29 by @dependabot[bot] in #793
• feat(core): add per-worker resilience and HTTP pool config types by @CatherineSue in #799
• fix(ci): remove ignore-pr-updates that marks active PRs as stale by @CatherineSue in #804
• refactor(openai): cleanup dead code, redundant state, and hot-path inefficiencies by @slin1237 in #802
• fix(mesh): break infinite retry loop for oversized incremental updates by @slin1237 in #808
• feat(core): wire per-worker resilience and HTTP client into BasicWorker by @CatherineSue in #803
• test(mesh): add serialization benchmark for mesh state sync by @slin1237 in #810
• test(e2e): re-enable skipped tests for vLLM and TRT-LLM by @CatherineSue in #806
• perf(mesh): switch all mesh serialization from JSON to bincode by @slin1237 in #809
• fix(mesh): use bincode for snapshot generation to match receivers by @slin1237 in #816
• feat(gateway): Propagate otel context for distributed tracing by @ekzhang in #814
• perf(mesh): skip full-store scans when nothing has changed by @slin1237 in #823
• fix(ci): drop [grpc] extra from nightly vllm install by @CatherineSue in #826
• feat(ci): add minimaxai/minimax-m2 to nightly benchmark by @smfirmin in #795
• fix(gateway): Disable auto-detection if "runtime": "sglang"explciitly set by @ekzhang in #820
• perf(mesh): add operation log auto-compaction and tombstone GC by @slin1237 in #825
• refactor(e2e): replace smg_compare with parametrized api_client fixture by @CatherineSue in #812
• fix(client): auto-close streaming responses on iteration exhaustion by @CatherineSue in #835
• refactor(e2e): add model fixture, remove deprecated smg fixture by @CatherineSue in #834
• fix(protocol): validate /v1/messages tool_choice contract by @nishanthp in #833
• fix(mesh): prevent stale snapshot chunks from mixing across retries by @slin1237 in #837
• test(mesh): improve benchmark summary with timing and side-by-side comparison by @slin1237 in #841
• fix(docker): install gRPC proto and servicer for vLLM images by @slin1237 in #843
• fix(gateway): use URL-based detection to eliminate classify race condition by @slin1237 in #839
• ci: bump default engine vllm(0.18.0) and trt(1.3.0rc8) versions by @slin1237 in #845
• fix(gateway): handle mismatched metric labels in PD disaggregation mode by @slin1237 in #846
• fix(pd): abort both PD requests when one side hits a transport error by @slin1237 in #844
• docs(quickstart): audit and fix getting-started documentation by @slin1237 in #848
• docs(extensibility): audit and fix tokenizer, MCP, and WASM plugin documentation by @slin1237 in #849
• docs(workers): audit and fix worker configuration and gRPC pipeline documentation by @slin1237 in #850
• docs(reliability): audit and fix reliability feature documentation by @slin1237 in #851
• docs(operations): audit and fix monitoring and data connection documentation by @slin1237 in #854
• docs(routing): audit and fix PD disaggregation and cache-aware routing documentation by @slin1237 in #853
• docs(config): audit and fix configuration, metrics, and architecture documentation by @slin1237 in #852
• docs(api): audit and fix API reference documentation by @slin1237 in #855
• fix(ci): scope VERSION_OVERRIDE to smg crate only by @slin1237 in #856
• chore(release): bump version to 1.3.3 by @slin1237 in #857

New Contributors

• @smfirmin made their first contribution in #795
• @nishanthp made their first contribution in #833

Full Changelog: v1.3.2...v1.3.3