Skip to content

TokenMiser v0.5.0

Choose a tag to compare

View all tags
@thinkflow19 thinkflow19 released this 29 Jul 07:05
· 18 commits to main since this release
v0.5.0
29b4b29

First feature release: TokenMiser is now a full Rust LLM cost-routing proxy — a 9-crate workspace built on Pingora.

Highlights

  • Drop-in OpenAI-compatible proxy — /v1/chat/completions, streaming (SSE) and non-streaming, with per-response audit headers (x-tokenmiser-cache, x-tokenmiser-tier, x-tokenmiser-difficulty, x-tokenmiser-routed-to)
  • Tiered router — Tier 0 heuristics → Tier 1 semantic classifier → Tier 2 speculative cascade, plus a Rhai policy DSL with tokenmiser policy test replay
  • Dual-layer semantic cache — L1 exact-match LRU + L2 semantic (bge-small embeddings, precision-tuned 0.87 cosine threshold)
  • Provider adapters — OpenAI, Anthropic, Ollama behind one trait; local-first: auto-detects a running Ollama and routes easy traffic there with zero API keys and zero config
  • Cost ledger + live dashboard — real-time USD spent/saved from a canonical pricing.json; htmx dashboard at / and JSON /stats
  • Quality judge — shadow-mode A/B with LLM-as-judge win-rate auto-gate
  • MCP budget gateway — per-agent/per-tool spend caps via /v1/mcp/*

No telemetry, ever. Full details in CHANGELOG.md.

Install (macOS arm64)

curl -sL https://github.com/openintelligence-labs/tokenmiser/releases/download/v0.5.0/tokenmiser-v0.5.0-aarch64-apple-darwin.tar.gz | tar xz
./tokenmiser-v0.5.0-aarch64-apple-darwin/tokenmiser

Other platforms: cargo build --release from source.

SHA-256 (tokenmiser-v0.5.0-aarch64-apple-darwin.tar.gz): 888c0e8e861ea7f142d0f72a25a0b73147ccc11a04c782388871183792a21a3e

Assets 3
Loading