v1.2.0: Enterprise Model Routing, SDKs and Pareto Benchmark Suite

Model Router v1.2.0 Release Notes

Model Router v1.2.0 delivers the complete production milestone for intelligent, cost-aware LLM request routing, native drop-in SDKs, automated benchmarking, contextual multi-armed bandit auto-tuning, multi-tenant workspace isolation, and full-stack observability.

Highlights

1. Official Client SDKs

• Python SDK (modelrouter-sdk): Sync and async OpenAI drop-in completion client, real-time SSE token streaming, and explainability dry-run inspector.
• TypeScript and JavaScript SDK (@picadolabs/modelrouter-sdk): Native fetch-based client for Node.js 18+, Bun, Deno, and modern browser environments with async iterator stream parsing.

2. Automated Benchmark Suite and Pareto Evaluator

• Added CLI command modelrouter benchmark supporting GSM8K (math reasoning), HumanEval (code synthesis and assertion checks), and MMLU (general knowledge) datasets.
• Multi-objective Cost-Accuracy Pareto Frontier evaluator computing non-dominated candidate sets.
• Terminal ASCII 2D scatter visualizer and exports in JSON and Markdown table formats.

3. Contextual Thompson Sampling Bandit Auto-Tuning

• Reinforcement learning engine using Beta-distribution priors to adapt candidate quality scores online based on real-time latency, throughput, and error feedback.

4. Distributed Tracing and Prometheus Metrics

• Standardized OpenTelemetry tracing with span context propagation across Analyzer, Scoring Engine, and Provider Execution layers.
• Native Prometheus metrics scraping endpoint exposed at /metrics tracking request counts, routing latency histograms, token volume, and circuit breaker health.

5. Multi-Tenant Workspaces and Rate Limiting

• Tenant workspace isolation with cryptographically hashed API keys (mr_live_...).
• Sliding-window token-bucket rate limiter enforcing RPM constraints per workspace and API key.

6. Semantic Embedding Classifier

• Fast ONNX dense vector embedding similarity with continuous complexity scoring (0.05 to 0.99) operating under 3ms overhead.

Full Changelog

• feat(sdk): publish lightweight Python & TypeScript SDKs for drop-in routing (#10) by @Kaap10 in #35
• feat(benchmark): automated routing benchmark suite and cost-accuracy Pareto evaluator (#20) by @Kaap10 in #34
• feat(observability): export OpenTelemetry distributed traces and Prometheus metrics (#19) by @Kaap10 in #33
• feat(ml): implement Thompson Sampling RL to auto-tune model selection policies (#9) by @Kaap10 in #32
• feat(classifier): add ONNX-based semantic embedding classifier (#5) by @Kaap10 in #31
• feat(auth): add API key management and token-bucket rate limiting for workspaces (#18) by @Kaap10 in #30
• docs: add modern zoomed system architecture flowchart to README by @Kaap10 in 1cc0985
• chore: bump version to 1.2.0 by @Kaap10 in 45c7513

Full Comparison: v1.1.0...v1.2.0

v1.1.0 - Enterprise Scalability, Circuit Breakers & Streaming

We are pleased to announce v1.1.0, marking the completion of Phase 5 and the initiation of Phase 6 on our path to an enterprise-ready MVP. This release introduces significant improvements to scalability, resilience, and inference speed.

What's New

• High-Speed Cloud Providers: Added official adapters for Groq, DeepSeek, and Together AI for low-latency inference.
• Token Streaming Proxy: Added a real-time SSE token streaming proxy to /api/generate, reducing perceived latency for frontend applications.
• Circuit Breaker: Implemented a robust sliding-window circuit breaker to dynamically bypass failing upstream providers, preventing cascading latency spikes during outages or rate limits.
• PostgreSQL & Alembic Integration: Upgraded the storage engine to support postgresql+asyncpg:// with connection pooling for multi-tenant scalability, backed by Alembic for schema migrations (SQLite support remains intact for local development).
• Redis Pub/Sub: Rearchitected the AI Traffic Control Room's live traffic feed to utilize Redis Pub/Sub instead of SQLite polling, enabling the dashboard to scale horizontally across multiple Uvicorn workers.

Upgrading

\�ash
pip install --upgrade model-router-cli
\