v0.1.0: the whole pipeline, end to end

Konkord ranks LLMs on a task suite and measures whether its own automated judge
can be trusted. Every leaderboard it produces carries the judge-versus-human
agreement rate alongside it.

This is the first tagged version. The pipeline is complete and runnable end to end.

What works

Command	Does
konkord run	One output per (task × model), concurrent, resumable, every response cached on disk
konkord judge	Pairwise LLM-as-judge, every pair scored twice in both presentation orders
konkord label	Local Streamlit app for blind human labelling
konkord calibrate	Agreement rate, Cohen's kappa, breakdowns, failure gallery
konkord report	results.json with Bradley-Terry ratings and bootstrap confidence intervals

Ships with a 25-task Python code generation suite. Each task targets a specific
failure mode rather than testing whether a model can write Python at all.

The bias controls

These are the point of the project, not features around it:

• The judge sees "Answer 1" and "Answer 2", never a model name.
• Every pair is judged in both orderings. Disagreement between them is position
  bias, resolves to a tie, and the order-flip rate is reported as a diagnostic.
• A judge sharing a provider family with any ranked model is refused outright.
• Unparseable verdicts are retried once, then recorded with the raw response.
  They are never coerced into a winner.
• The labeller never sees the judge's verdict, so agreement is not suggestibility.
• Models whose confidence intervals overlap share a rank_group and must be
  rendered as tied.

What does not work yet

• konkord check, the deterministic graders in a Docker sandbox, is not
  implemented and exits non-zero.
• No successful model call has been made through this code yet. The failure
  paths are tested; the happy path against a live provider is not.
• No calibration number exists, so no leaderboard has been published from this.

Getting started

See the Quickstart in the README. Not on PyPI yet, so clone and uv sync --all-extras.

Engineering

Python 3.12, 178 tests, ruff and mypy --strict clean in CI. Tests never touch
a network: the model provider sits behind a protocol so the suite runs against a
fake completer in under a second. stats.py is verified against closed-form
inputs, since on a single pair the Bradley-Terry strength ratio equals the win
ratio exactly.