v1.11.0: Round 8, seven more models from OpenRouter

Released 2026-10-25, commit 3c300e4.

Round 8 runs seven decision models that joined OpenRouter's Decisions API since round 4 on all 3,250 text cases, and the four that read images on the vision track. Both indexes have a new leader. The data, the other runs and their scores are unchanged.

New on the board

Model	Maker	System One Index	Rank	Vision Index	Rank
Decider V1.2	Perplexity	84.6	1st of 30	83.8	5th of 11
Drex Max	Nace AI	84.1	2nd
d1-ultra	Liquid AI	82.9	3rd	85.6	2nd
Decider Flash V1.2	Perplexity	81.9	7th	77.5	10th
Span-02	Respan	79.5	11th	86.1	1st
Drex v2.0	Nace AI	78.6	14th
Classify-1	ScaleDown	30.6	30th

• Decider V1.2 leads the System One Index, 1.2 points above Decider V1.1, and is the best calibrated model on the board (0.7%). Drex Max is 0.5 points behind it, and d1-ultra is 3rd.
• Span-02 leads the Vision Index, 1.0 point above Jev 1.14, and loses far less than any other model reading Arabic from a picture. It's Respan's first model to take a normal state; Span-01 only takes a conversation.
• Decider Flash V1.2 is the fastest API model on the board (104 ms median) and, with Span-02, the cheapest ($0.005 per 1,000 calls).
• Classify-1 rejects most of our cases: it only accepts a state that is plain text, so it answered the 200 DAIR Emotion cases and failed the other 3,050, which count as wrong. It also rejects images, so it has no vision score.
• Drex Max, Drex v2.0 and Classify-1 don't read images. None of the new models answers Box questions.
• Latency was timed from a different cloud server than the board's 15 October timings. A control run of Jev 1.14 measured 122 ms there against 136 ms on the board, so the new latencies compare to within about 10%.

Retired

Decider V1.1 27B has left OpenRouter's catalog and returns 404, so it's tagged retired and left out of the charts and rankings like Decider V1. Its run stays on the board for reference. Decider V1.2 replaces it in the live race, where the other new models (except Classify-1) join it too.

Left out

Baseten's Mammoth 1 (27B) and Elephant 1 (9B) are on OpenRouter's free tier only, which allows 1,000 requests a day against the 3,341 a run takes, so they're left out of this round. Their MIT open weights are on the roadmap.

Roadmap

Added: Baseten's Mammoth 1 and Elephant 1 as open weights; THX-01 and vLLM's Decision-1.0 Eos 0.8B for the next CPU round; Cloudflare Clairic, which extracts typed values rather than making decisions, needs a track of its own; the unconfirmed Fastino GLiDE v1.2 and the Clinical System One (CS1) paper are on the watch list.

v1.10.0: Eight small open-weight models, run on a CPU

Released 2026-10-18, commit 9c0918c.

Round 7 runs eight open-weight models from the roadmap on all 3,250 text cases, from a 4-vCPU cloud server with no GPU. The data, the other runs and their scores are unchanged.

New on the board

Model	Maker	System One Index	Rank
d1-3B	Liquid AI	78.4	10th of 24
d3 nano	vLLM Semantic Router	78.0	11th
d3 lite	vLLM Semantic Router	74.9	15th
d3 edge	vLLM Semantic Router	65.9	18th
d1-omni-600M	Liquid AI	65.2	19th
Kev 0.8B	Jared Palmer	63.6	21st
Naluri-308M	Jericho Siahaya	61.0	22nd
Naluri-279M	Jericho Siahaya	50.4	24th

• d1-3B is the first open-weight model in the top 10, with d3 nano right behind it. Both place above Drex v1.5, Kev 4B and GPT-6 Luna Decisions at under 3 GB.
• d1-3B is also the second best calibrated model on the board (2.4%, behind Clef's 2.2%).
• The smallest models keep most of their English accuracy and lose much more in Arabic: d1-omni-600M is within 6 points of the full d1 in English and 30 below it in Arabic.
• The Naluri encoders score exactly what their model cards report. They are trained on the train splits of three datasets tested here; their notes on the site say which.
• The GGUF models ran on llama.cpp b11716 at Q8_0, the Naluri encoders on PyTorch. Every call in the published runs succeeded. Their latencies are local CPU timings, so they stay out of the latency charts.

Roadmap

d1-3B, d1-omni-600M, d3 edge, lite and nano, Kev 0.8B and both Naluri encoders leave the roadmap. The larger d3 and Kev sizes stay listed.

v1.9.0: A logo, live badges and community files

Released 2026-10-17, commit 1e58fbb.

The repository now reads like a mature open-source project. No runs, data or scores changed.

What's new

• Logo: the System One Bench mark and wordmark head the README, in light and dark versions (docs/assets/).
• Live badges: the site now publishes shields.io endpoint badges at systemonebench.dev/badge/models.json, text-leader.json and vision-leader.json, written from the results at every deploy. The README shows them with the latest release, the frozen-data check, licences and citation.
• Quick start and contents: the README opens with the six commands from clone to a scored run, and a list of every section.
• Community files: a code of conduct, a security policy, a problem report form next to the model request, CODEOWNERS and .editorconfig.
• Dependabot keeps the workflows' actions current. The Python packages stay pinned, since they build the frozen data.

v1.8.0: A page for every model, and head-to-head comparisons

Released 2026-10-17, commit 389c366.

Every model now has its own page, and every pair of ranked models has a comparison page. No runs, data or scores changed.

What's new

• Model pages at systemonebench.dev/models/<model>/, 20 in all: both indexes with their rank, calibration, latency and cost, then every text and image dataset with the model's accuracy, its rank, and the best score on the board. For example, Decider V1.1 27B (retired in round 8, so its page now ends in -retired) and Jev 1.14.
• Comparison pages at systemonebench.dev/compare/<a>-vs-<b>/, 120 in all, one for every pair of ranked models, with the better score marked on every line. For example, Decider V1.2 vs Jev 1.14 (this example pointed to Decider V1.1 until it retired in round 8, which removed its comparison pages).
• The leaderboard links each model's details to its page, and sitemap.xml lists every page so search engines can find them.
• The pages are generated from the results every time the site deploys, so they always match the leaderboard.

v1.7.0: Open licence, citation and link previews

Released 2026-10-17, commit 5971c39.

System One Bench is now openly licensed and citable, and shares well. No runs, data or scores changed.

What's new

• Licence: the code is Apache-2.0 and the results are CC BY 4.0, so anyone can reuse them with a link back. Cases and images keep their source datasets' licences.
• Citation: CITATION.cff, so GitHub offers "Cite this repository".
• Link previews: shared links to systemonebench.dev now show a title, description and card image on LinkedIn, X, Slack and Discord. The site also has a favicon, robots.txt and sitemap.xml.
• Request a model: a new issue form to ask for a run in public.
• Badge: model makers can link their model card to its results with a "Scored on System One Bench" badge (see the README).
• Plan: docs/plan.md lists what's next: per-model pages, a technical report, open-weight runs, and a one-command way to score your own model.

v1.6.0: Searchable site, one leader per track

Released 2026-10-15, commit ce33abf.

A redesign of systemonebench.dev and a pass that checks every claim on the site and in the README against the results files (#12). No runs, data or scores changed.

New on the site

• Search (press /) and maker filters, applied to every chart and table at once. The filter is kept in the URL, so a view like ?q=clef or ?maker=Perplexity can be shared.
• One leaderboard for both tracks: System One Index, English, Arabic and Vision Index side by side, sortable by any column. Click a model to see its run details and every dataset score with its rank.
• A leader for each track. The headline names Decider V1.1 27B as text leader (83.4, 0.7 ahead of d1) and Jev 1.14 as vision leader (85.1, 1.2 ahead of Jev 1.13), and says the text leader is 3rd on images. It's computed from the data, so it stays true as models are added.
• A header that stays on screen, section labels for each track, a light/dark switch, and a phone layout.

Corrected

• The "fastest 5" latency card ranked Laya first with a time measured locally on a Mac, which doesn't compare with API round trips. Latency charts now compare API models only.
• README: 7 open-weight roadmap entries (not 6), Decider V1.1's median latency 263 ms, and Jev 1.14 described as the fastest API model.

v1.5.0: Round 6, the vision track and frozen data

Released 2026-10-15, commit 4fd85a7.

An image benchmark for every model on OpenRouter's Decisions API that reads images, the benchmark data frozen so later models run on exactly the same bytes, and written rules for every run (#11). Plus the project's first outside contribution (#9).

Vision track

1,200 image cases across 6 datasets: the 200 SST-2 reviews and 200 Arabic tweets drawn as pictures (a like-for-like test of reading from pixels), Oxford-IIIT Pet, VQAv2 yes/no, CountBench, and RefCOCO for Box questions. The Vision Index is the mean accuracy over the first 5, kept apart from the System One Index.

Model	Vision Index	Calibration error	Median latency	Cost per 1,000 calls
Jev 1.14	85.1	2.9%	183 ms	$0.046
Jev 1.13	83.9	2.8%	192 ms	$0.046
Decider V1.1 27B	83.0	5.6%	302 ms	$0.009
GPT-6 Luna Decisions	81.6	5.3%	221 ms	$0.047
Mercury Decide	79.4	9.9%	305 ms	free tier
Clef	78.1	3.6%	502 ms	$0.138
Clef Flash	78.0	9.4%	429 ms	$0.022
Clef Omni	72.2	20.6%	274 ms	$0.086

• The two indexes rank models differently: Decider V1.1 27B leads text but is 3rd on images; Jev 1.14 leads images and is 8th on text.
• Reading English from a picture costs every model at most 7.5 points; reading Arabic costs 12.5 to 23.5.
• Box questions (Jev 1.14 only): 78.5% of boxes overlap the labelled one by at least half, at an average stated confidence of 48%.
• d1, Microsoft-Decision-1, Kev 4B, Tev1, Solar Decide, Solar Decide Flash and Drex reject images, so they have no vision score rather than a zero.

Frozen data

• data.lock.json holds a SHA-256 for every dataset and image; run_decisions.py and score.py refuse data that doesn't match.
• Every source is pinned to a fixed commit, and .github/workflows/data.yml rebuilds everything from scratch on data PRs and every Monday.
• The text cases are the same 3,250 every earlier release used.

Benchmark rules

CONTRIBUTING.md and a PR template checklist: frozen data, identical requests, failures count as wrong, latency from one machine, runs added without disturbing others, and only scores measured here on the site.

First outside contribution

Thanks to @jerichosiahaya for adding the Naluri encoders (279M and 308M) to the open-weights roadmap, with a serve.py that answers the same /v1/systemone request, so run_decisions.py --url runs them unchanged (#9). They're on the roadmap and get on the board once they've run here under the same rules.

v1.4.0: Round 5, Jev 1.14, and latency from one machine

Released 2026-10-15, commit b626193.

TypeSafe's Jev 1.14 preview on all 3,250 cases, and every API model re-timed from one machine so latency compares fairly (#10).

Jev 1.14

Model	Index	English	Arabic	Calibration error	Median latency	Cost per 1,000 calls
Jev 1.14	79.3	78.8	80.3	6.0%	136 ms	$0.028
Jev 1.13	79.0	78.5	79.9	6.1%	150 ms	$0.028

• 8th of 16 on the System One Index, 0.3 points above Jev 1.13: within run-to-run noise (a same-day re-run of Jev 1.13 moved single datasets by up to 1.5 points).
• One clear gain: customer-service decisions in typed-decisions, 83.6% against 77.4%.
• Same price and the same billed tokens as Jev 1.13. Every call succeeded.

Latency, re-timed

Every charted API model ran all 3,250 cases again from one cloud server, 8 calls in flight, with models from the same maker at different times. Only latency comes from these runs; accuracy, calibration and cost stay from each model's original run.

• Every model got faster, but unevenly: from the earlier machine the same calls took 1.5 to 3.5 times as long.
• The two Jevs are the fastest API models (136 and 150 ms); Liquid AI d1 (188 ms), Drex v1.5 and Clef Omni now come before GPT-6 Luna Decisions.
• set_latency.py copies latency from re-runs into results.json. Every latency chart carries a note on what the numbers include.

v1.3.0: Round 4, four more models, and vendor APIs

Released 2026-10-12, commit 91f4f28.

Four decision models that appeared on OpenRouter's Decisions API, on all 3,250 cases with 8 calls in flight (#7), and a runner that reaches models outside OpenRouter (#8).

Results

Model	Maker	Index	Calibration error	Median latency	Cost per 1,000 calls
Decider V1.1 27B	Perplexity	83.4	5.7%	678 ms	$0.009
Clef Omni	Cloudflare	79.7	3.4%	544 ms	$0.074
Drex v1.5	Nace AI	77.6	10.8%	608 ms	$0.010
Solar Decide Flash	Upstage	63.6	14.4%	908 ms	$0.042

• Decider V1.1 27B leads the System One Index and is the cheapest paid model. Perplexity retired Decider V1 (its slug returns 404), so V1.1 replaces it on the charts; the V1 run is kept for reference.
• Solar Decide Flash shares Solar Decide's 26-option limit and fails all 200 Banking77 cases.

Also in this release

• run_decisions.py --provider fastino|celeris|hanzo|meragpt calls each vendor's own API with the same request body, and --url reaches any compatible endpoint, including a local llama-server (#8).
• README: a section on running open-weight models locally (llama.cpp GGUF builds, Hugging Face system_one, kev.serve).
• The roadmap is rebuilt and grouped by what each model needs; scatter labels get leader lines so every dot is labelled.

v1.2.0: Round 3, Microsoft-Decision-1

Released 2026-10-12, commit bfe2557.

Microsoft-Decision-1 (released 2026-10-09, post-trained from Qwen3.5-9B) on all 3,250 cases through OpenRouter's Decisions API, 8 calls in flight. Every call succeeded (#6).

Results

• Index 82.0, 4th of 12, 1.3 points behind Perplexity Decider V1 27B. 3rd on the Arabic index (83.2).
• Calibration error 2.8%, second only to Cloudflare Clef (2.2%).
• $0.034 for the full run ($0.010 per 1,000 calls), level with Kev 4B, which scores 5.2 points lower.
• Most accurate model on SST-5 (62.5%), typed-decisions (79.0%) and Arabic MIRACL pairs (91.0%); weakest on Emotion (57.0%).
• Median latency 679 ms through OpenRouter, network included. Microsoft reports 85 ms p50 in its own test; only numbers measured here go on the board.

Microsoft gets its own color on the site and a lane in the live race.