Skip to content

LocalBench SMB v2.1.0: 180-Case Router & Zero-Token Decision Shootout

Latest

Choose a tag to compare

@atxatlarge-code atxatlarge-code released this 14 Nov 14:49
· 1 commit to main since this release

What's New in LocalBench SMB v2.1.0

Highlights

  • Zero-Output-Token Closed-Set Client (gemini_decision.py): Supports both Google gemini-decision-1 (v1beta/decisions) and local Ollama System One decision runners (/v1/systemone) using parallel self-contained choice / noul schemas.
  • Hybrid Specialist Router (specialist_router.py): Integrates router:gemini-decision-1+gemma4-2b+qwen2.5-coder:1.5b-mlx (99.35% across all 180 cases / 99.7% SMB Ops), running #6 Data Filtering 1.81x faster and #9 Task Routing 1.39x faster alongside the 99.5% (180/180) 100% on-device Metal router baseline.
  • Live 3-Way Closed-Set Shootout (140 cases across 7 pillars, 0 output tokens) — see blueprints/11_GEMINI_DECISION_1_SHOOTOUT.md:
    • gemini-decision-1: 99.64% Closed-Set 7 (139/140, 100.0% private holdout), 289.3 ms p50, $0.007437 total API spend.
    • tev1:latest (4B, local /v1/systemone): 93.93% Closed-Set 7 (130/140, 100% on 5/7 pillars), 300.3 ms p50 (42.34s total vs. 51.44s), $0.00 / 0 MB cloud egress.
    • tev1:0.8b (local /v1/systemone): 76.61% Closed-Set 7 (100/140), 64.0 ms p50 (10.12s total), $0.00.
  • Privacy & Holdout Safeguards: Scrubbed hardcoded local paths and redacted predicted answers on private holdout result exports; kept datasets/localbench_smb_private.jsonl, .env*, smoke runs, and inbox captures excluded via .gitignore.
  • MIT Licensed: Full v2.0/v2.1 evaluator (eval_localbench.py), router (specialist_router.py), Blueprints #1-#11, and 90-case public v2.0 dataset under LICENSE.