V1.0.0
First release, and the first published run.
Konkord ranks LLMs on a task suite and then measures whether the automated judge behind that
ranking can be trusted. The ranking and the trust number are published together.
The finding
Three models answered 25 hand-written Python tasks. A fourth model judged every pair blind, in both
presentation orders. Then a human labelled all 75 pairs blind, seeing neither the model names nor
the judge's opinion.
The judge agreed with the human on 61.3% of them. That is 46 of 75, 95% CI 50.0% to 71.5%.
Cohen's kappa 0.370, where 0 is chance and 1 is perfect.
Split by matchup, the number moves a long way:
| Matchup | Agreement |
|---|---|
| gpt-5.4 vs gemini-2.5-flash-lite | 76.0% (19/25) |
| claude-haiku-4.5 vs gemini-2.5-flash-lite | 56.0% (14/25) |
| gpt-5.4 vs claude-haiku-4.5 | 52.0% (13/25) |
The judge could tell a strong model from a weak one. Between the two strong models it was at
chance: 52% on a two-way call is a coin flip.
None of that is visible on the leaderboard itself. The ranking separates all three models with
non-overlapping bootstrap intervals, and it is drawn correctly. But a confidence interval only
measures how much the judge's verdicts would wobble under resampling. It says nothing about whether
those verdicts are right. Judging more pairs would have tightened the interval around the same
error.
Two tasks reached 0% agreement. The judge also changed its answer on 20% of pairs when the two
answers were swapped, deciding on position rather than content.
The ranking
| # | Model | Rating | Win rate, 95% interval |
|---|---|---|---|
| 1 | gpt-5.4 | 3.429 | 85.0% (76.3 to 91.8) |
| 2 | openrouter/anthropic/claude-haiku-4.5 | 1.214 | 55.0% (43.2 to 67.1) |
| 3 | openrouter/google/gemini-2.5-flash-lite | 0.240 | 10.0% (4.8 to 16.7) |
Bradley-Terry ratings over 150 judged comparisons, with percentile bootstrap intervals over 1,000
seeded resamples. Judge: openrouter/mistralai/mistral-small-3.2-24b-instruct, chosen because no
ranked model shares its provider family. Generating the 75 answers cost $0.42.
What it does
| Command | Does |
|---|---|
konkord run |
Generate one output per (task x model), resumable and cached |
konkord judge |
Pairwise LLM-as-judge, both orderings, same-family judges refused |
konkord label |
Local blind labeller, no model names and no judge verdict shown |
konkord calibrate |
Agreement and kappa, broken down by task, pair and answer length |
konkord report |
Bradley-Terry ratings, bootstrap intervals, calibration block |
Bias controls that are enforced rather than documented: the judge never sees a model name, a judge
sharing a provider family with any ranked model is refused outright (routing prefixes are peeled
off first, so openrouter/anthropic/claude-opus-5 and claude-opus-5 count as the same family),
every pair is judged in both orders and disagreement between them is recorded as a tie, and a
response with no parseable verdict is kept as evidence rather than coerced into a winner.
Install
git clone https://github.com/rouzbeh-abadi/konkord
cd konkord
uv sync --all-extrasRequires Python 3.12. Model names are whatever litellm accepts; one OpenRouter key reaches most
vendors.
What it does not do
Nothing here is executed. There is no sandbox, so no answer is compiled, linted or run, and every
judgement is a reading of the code rather than a test of it. Suites can declare deterministic
checks and the loader validates them, but no runner consumes them yet.
Full results, the judge prompt verbatim, and every answer:
konkord.deadpixelstudio.io