v1.1.2

No behaviour change. This release exists to republish the PyPI project page, whose description is
baked into the uploaded distribution and cannot be edited without a new version.

The page carried instructions that only work from a clone, which is the wrong audience for the place
people land after pip install:

• uv run konkord ... assumed a uv project
• cp .env.example .env referenced a file that is not in an installed package
• --out site/results.<suite>.json referenced a directory that is not either

Installation is now its own section, pip install is the primary route, and the repository is
presented as where the worked example suites live rather than as the way in.

Nothing to upgrade for if you are already on 1.1.1.

v1.1.0: suite-defined judging rubrics

1.0.0 could only judge programming tasks: the rubric was a constant in the source. A suite now
carries its own criteria, so the harness works on any domain.

name: support_replies
rubric: |
  1. Accuracy. Does the reply follow the policy as stated, and address everything
     the customer actually asked?
  2. Tone. Given equal accuracy, prefer the reply the customer would rather receive.
answer_language: null   # prose, so render it as prose rather than as code

The response format and the instruction to ignore length stay in the tool's frame. A rubric free to
rewrite those would break verdict parsing and delete the bias controls with it, so one mentioning the
verdict token is refused.

Every verdict now records the prompt that produced it. A rubric edited mid-suite is refused rather
than averaged into one rating meaning two standards, and the site publishes the prompt that actually
ran instead of recomposing it from the suite file. The labeller shows the same rubric to the human
that the judge was given.

Second calibrated run

25 customer support tickets, same three models and judge as the code run, 75 blind human labels.

	Support replies	Python codegen
Agreement with a human	46.7%	61.3%
Order-flip rate	42.7%	20.0%
Agreement on self-consistent pairs	81.4%	76.7%

The judge is not worse at prose. On pairs where it gave the same verdict in both orderings it agrees
slightly more; the whole gap is that it contradicts itself twice as often, and a contradicted pair is
recorded as no winner. A harness judging each pair once would publish a confident ranking built on
those coin flips.

Breaking changes

• rubric is required in suite files. The old behaviour is konkord.models.DEFAULT_RUBRIC.
• The site reads one results file per suite, indexed by runs.json, not a single results.json.
• report refuses to build when a suite's verdicts span more than one judge prompt.
• The comparisons table gains a judge_prompt column, applied automatically on open.

konkord.deadpixelstudio.io

V1.0.0

First release, and the first published run.

Konkord ranks LLMs on a task suite and then measures whether the automated judge behind that
ranking can be trusted. The ranking and the trust number are published together.

The finding

Three models answered 25 hand-written Python tasks. A fourth model judged every pair blind, in both
presentation orders. Then a human labelled all 75 pairs blind, seeing neither the model names nor
the judge's opinion.

The judge agreed with the human on 61.3% of them. That is 46 of 75, 95% CI 50.0% to 71.5%.
Cohen's kappa 0.370, where 0 is chance and 1 is perfect.

Split by matchup, the number moves a long way:

Matchup	Agreement
gpt-5.4 vs gemini-2.5-flash-lite	76.0% (19/25)
claude-haiku-4.5 vs gemini-2.5-flash-lite	56.0% (14/25)
gpt-5.4 vs claude-haiku-4.5	52.0% (13/25)

The judge could tell a strong model from a weak one. Between the two strong models it was at
chance: 52% on a two-way call is a coin flip.

None of that is visible on the leaderboard itself. The ranking separates all three models with
non-overlapping bootstrap intervals, and it is drawn correctly. But a confidence interval only
measures how much the judge's verdicts would wobble under resampling. It says nothing about whether
those verdicts are right. Judging more pairs would have tightened the interval around the same
error.

Two tasks reached 0% agreement. The judge also changed its answer on 20% of pairs when the two
answers were swapped, deciding on position rather than content.

The ranking

#	Model	Rating	Win rate, 95% interval
1	gpt-5.4	3.429	85.0% (76.3 to 91.8)
2	openrouter/anthropic/claude-haiku-4.5	1.214	55.0% (43.2 to 67.1)
3	openrouter/google/gemini-2.5-flash-lite	0.240	10.0% (4.8 to 16.7)

Bradley-Terry ratings over 150 judged comparisons, with percentile bootstrap intervals over 1,000
seeded resamples. Judge: openrouter/mistralai/mistral-small-3.2-24b-instruct, chosen because no
ranked model shares its provider family. Generating the 75 answers cost $0.42.

What it does

Command	Does
konkord run	Generate one output per (task x model), resumable and cached
konkord judge	Pairwise LLM-as-judge, both orderings, same-family judges refused
konkord label	Local blind labeller, no model names and no judge verdict shown
konkord calibrate	Agreement and kappa, broken down by task, pair and answer length
konkord report	Bradley-Terry ratings, bootstrap intervals, calibration block

Bias controls that are enforced rather than documented: the judge never sees a model name, a judge
sharing a provider family with any ranked model is refused outright (routing prefixes are peeled
off first, so openrouter/anthropic/claude-opus-5 and claude-opus-5 count as the same family),
every pair is judged in both orders and disagreement between them is recorded as a tie, and a
response with no parseable verdict is kept as evidence rather than coerced into a winner.

Install

git clone https://github.com/rouzbeh-abadi/konkord
cd konkord
uv sync --all-extras

Requires Python 3.12. Model names are whatever litellm accepts; one OpenRouter key reaches most
vendors.

What it does not do

Nothing here is executed. There is no sandbox, so no answer is compiled, linted or run, and every
judgement is a reading of the code rather than a test of it. Suites can declare deterministic
checks and the loader validates them, but no runner consumes them yet.

Full results, the judge prompt verbatim, and every answer:
konkord.deadpixelstudio.io

v0.1.0: the whole pipeline, end to end

Konkord ranks LLMs on a task suite and measures whether its own automated judge
can be trusted. Every leaderboard it produces carries the judge-versus-human
agreement rate alongside it.

This is the first tagged version. The pipeline is complete and runnable end to end.

What works

Command	Does
konkord run	One output per (task × model), concurrent, resumable, every response cached on disk
konkord judge	Pairwise LLM-as-judge, every pair scored twice in both presentation orders
konkord label	Local Streamlit app for blind human labelling
konkord calibrate	Agreement rate, Cohen's kappa, breakdowns, failure gallery
konkord report	results.json with Bradley-Terry ratings and bootstrap confidence intervals

Ships with a 25-task Python code generation suite. Each task targets a specific
failure mode rather than testing whether a model can write Python at all.

The bias controls

These are the point of the project, not features around it:

• The judge sees "Answer 1" and "Answer 2", never a model name.
• Every pair is judged in both orderings. Disagreement between them is position
  bias, resolves to a tie, and the order-flip rate is reported as a diagnostic.
• A judge sharing a provider family with any ranked model is refused outright.
• Unparseable verdicts are retried once, then recorded with the raw response.
  They are never coerced into a winner.
• The labeller never sees the judge's verdict, so agreement is not suggestibility.
• Models whose confidence intervals overlap share a rank_group and must be
  rendered as tied.

What does not work yet

• konkord check, the deterministic graders in a Docker sandbox, is not
  implemented and exits non-zero.
• No successful model call has been made through this code yet. The failure
  paths are tested; the happy path against a live provider is not.
• No calibration number exists, so no leaderboard has been published from this.

Getting started

See the Quickstart in the README. Not on PyPI yet, so clone and uv sync --all-extras.

Engineering

Python 3.12, 178 tests, ruff and mypy --strict clean in CI. Tests never touch
a network: the model provider sits behind a protocol so the suite runs against a
fake completer in under a second. stats.py is verified against closed-form
inputs, since on a single pair the Bradley-Terry strength ratio equals the win
ratio exactly.