Skip to content

Releases

LocalBench SMB v2.1.0: 180-Case Router & Zero-Token Decision Shootout

Choose a tag to compare

@atxatlarge-code atxatlarge-code released this 14 Nov 14:49

What's New in LocalBench SMB v2.1.0

Highlights

  • Zero-Output-Token Closed-Set Client (gemini_decision.py): Supports both Google gemini-decision-1 (v1beta/decisions) and local Ollama System One decision runners (/v1/systemone) using parallel self-contained choice / noul schemas.
  • Hybrid Specialist Router (specialist_router.py): Integrates router:gemini-decision-1+gemma4-2b+qwen2.5-coder:1.5b-mlx (99.35% across all 180 cases / 99.7% SMB Ops), running #6 Data Filtering 1.81x faster and #9 Task Routing 1.39x faster alongside the 99.5% (180/180) 100% on-device Metal router baseline.
  • Live 3-Way Closed-Set Shootout (140 cases across 7 pillars, 0 output tokens) — see blueprints/11_GEMINI_DECISION_1_SHOOTOUT.md:
    • gemini-decision-1: 99.64% Closed-Set 7 (139/140, 100.0% private holdout), 289.3 ms p50, $0.007437 total API spend.
    • tev1:latest (4B, local /v1/systemone): 93.93% Closed-Set 7 (130/140, 100% on 5/7 pillars), 300.3 ms p50 (42.34s total vs. 51.44s), $0.00 / 0 MB cloud egress.
    • tev1:0.8b (local /v1/systemone): 76.61% Closed-Set 7 (100/140), 64.0 ms p50 (10.12s total), $0.00.
  • Privacy & Holdout Safeguards: Scrubbed hardcoded local paths and redacted predicted answers on private holdout result exports; kept datasets/localbench_smb_private.jsonl, .env*, smoke runs, and inbox captures excluded via .gitignore.
  • MIT Licensed: Full v2.0/v2.1 evaluator (eval_localbench.py), router (specialist_router.py), Blueprints #1-#11, and 90-case public v2.0 dataset under LICENSE.

LocalBench SMB v1.2.0 (9 Pillars, 45 Test Cases, Division 1 & 2 Leaderboards)

Choose a tag to compare

@atxatlarge-code atxatlarge-code released this 04 Nov 21:18

Public release of LocalBench SMB v1.2.0: the practical benchmark for running private, $0/month local AI across 9 everyday business workflows (email, writing, docs, voice, chat, spreadsheets, and automation).

Highlights

  • 9 Business AI Pillars (45 Real-World Tests, 5 per Pillar): Email Triage (01_email_triage), Business Writing (02_business_writing), Company Docs Q&A (03_company_docs), Voice Commands (04_voice_commands), Team Chat (05_team_chat), Data Filtering (06_data_filtering), Business Formulas (07_business_formulas), Process Automation (08_process_automation), and Task Routing (09_task_routing).
  • Division 1 (Single-Model Leaderboard) & Division 2 (Composite / Multi-Model Systems Leaderboard) with runtime tags (LiteRT Metal, MLX Metal, Ollama GGUF).
  • 10 Plain-English Architecture Blueprints under blueprints/.