Repository navigation
Releases
Releases · atxatlarge-code/localbench-smb
LocalBench SMB v2.1.0: 180-Case Router & Zero-Token Decision Shootout
What's New in LocalBench SMB v2.1.0
Highlights
- Zero-Output-Token Closed-Set Client (
gemini_decision.py): Supports both Googlegemini-decision-1(v1beta/decisions) and local Ollama System One decision runners (/v1/systemone) using parallel self-containedchoice/noulschemas. - Hybrid Specialist Router (
specialist_router.py): Integratesrouter:gemini-decision-1+gemma4-2b+qwen2.5-coder:1.5b-mlx(99.35% across all 180 cases / 99.7% SMB Ops), running#6 Data Filtering1.81x faster and#9 Task Routing1.39x faster alongside the 99.5% (180/180) 100% on-device Metal router baseline. - Live 3-Way Closed-Set Shootout (
140cases across 7 pillars,0output tokens) — seeblueprints/11_GEMINI_DECISION_1_SHOOTOUT.md:gemini-decision-1: 99.64% Closed-Set 7 (139/140, 100.0% private holdout),289.3 msp50,$0.007437total API spend.tev1:latest(4B, local/v1/systemone): 93.93% Closed-Set 7 (130/140, 100% on 5/7 pillars),300.3 msp50 (42.34stotal vs.51.44s), $0.00 / 0 MB cloud egress.tev1:0.8b(local/v1/systemone): 76.61% Closed-Set 7 (100/140), 64.0 ms p50 (10.12s total), $0.00.
- Privacy & Holdout Safeguards: Scrubbed hardcoded local paths and redacted
predictedanswers on private holdout result exports; keptdatasets/localbench_smb_private.jsonl,.env*, smoke runs, and inbox captures excluded via.gitignore. - MIT Licensed: Full v2.0/v2.1 evaluator (
eval_localbench.py), router (specialist_router.py), Blueprints #1-#11, and 90-case public v2.0 dataset underLICENSE.
LocalBench SMB v1.2.0 (9 Pillars, 45 Test Cases, Division 1 & 2 Leaderboards)
Public release of LocalBench SMB v1.2.0: the practical benchmark for running private, $0/month local AI across 9 everyday business workflows (email, writing, docs, voice, chat, spreadsheets, and automation).
Highlights
- 9 Business AI Pillars (45 Real-World Tests, 5 per Pillar): Email Triage (
01_email_triage), Business Writing (02_business_writing), Company Docs Q&A (03_company_docs), Voice Commands (04_voice_commands), Team Chat (05_team_chat), Data Filtering (06_data_filtering), Business Formulas (07_business_formulas), Process Automation (08_process_automation), and Task Routing (09_task_routing). - Division 1 (Single-Model Leaderboard) & Division 2 (Composite / Multi-Model Systems Leaderboard) with runtime tags (
LiteRT Metal,MLX Metal,Ollama GGUF). - 10 Plain-English Architecture Blueprints under
blueprints/.