CoderDrift is a Python package for auditing whether annotation disagreement changes over annotation order. It supports nominal and ordinal multi-rater data, leave-one-out peer comparisons, and declared external references.
The software keeps three kinds of evidence separate:
- Descriptive: agreement, workload, batch, and rolling profiles.
- Exploratory: change-point candidates that do not carry a primary claim.
- Calibrated primary: profile-compatible candidates that also satisfy eligibility, practical-effect, support, and multiplicity requirements.
CoderDrift does not diagnose fatigue, negligence, bias, or individual coder deterioration. Peer results are relative to the observed peer group, and an external reference is treated as fallible rather than ground truth.
CoderDrift supports Python 3.12 through 3.14 and uses a locked uv
environment.
git clone --branch v1.0.1 --depth 1 https://github.com/ecylmz/CoderDrift.git
cd CoderDrift
uv sync --frozen
uv run coderdrift versionThe version command should report CoderDrift 1.0.1.
For development against the latest main branch:
git clone https://github.com/ecylmz/CoderDrift.gitAll runtime dependencies are open source and CPU-only. The package has no telemetry, hosted service, or proprietary component.
Run the bundled nominal example:
uv run coderdrift validate configs/examples/nominal_descriptive.yml
uv run coderdrift audit configs/examples/nominal_descriptive.ymlRun the ordinal exploratory example:
uv run coderdrift audit configs/examples/ordinal_exploratory.ymlEach audit writes a deterministic bundle to the configured output_dir. The
bundle includes report.html, manifest.json, eligibility.csv, temporal
profiles, change candidates, multiplicity records, support regions, warnings,
and SVG figures. Read eligibility.csv before interpreting any candidate as
primary evidence.
Each row represents one coder-item-dimension decision. Required fields are
item_id, coder_id, label, and the configured ordering field. Within a
dimension, coder-item pairs and per-coder order values must be unique.
item_id,coder_id,label,sequence,dimension
i001,coder_a,accept,1,overall
i001,coder_b,reject,1,overallReference-channel audits additionally require one consistent reference value per item and an explicit declaration of the reference's provenance and expected fallibility. Full schema and configuration details are in the getting-started guide and methodology guide.
Version 1.0 contains three deliberately narrow profiles:
| Dataset role | Channel | Minimum sequence | Analysis family |
|---|---|---|---|
| MultiPref overall | peer | 250 | 64 |
| WebCrowd25K four-level labels | peer | 500 | 17 |
| WebCrowd25K binary labels | fallible reference | 250 | 28 |
These limits belong only to the named roles. A new dataset is descriptive or exploratory until it receives its own prospective calibration.
The repository contains:
- synthetic examples;
- a locked 192-cell, 19,200-panel validation benchmark;
- the three immutable built-in profiles;
- aggregate MultiPref and WebCrowd25K result tables;
- deterministic hashes and machine-readable provenance;
- tests and complete reproduction code.
Third-party raw datasets and record-level worker traces are not redistributed. Their provider terms continue to apply.
The public checkout is self-verifying even when third-party datasets are not present:
uv sync --frozen
uv run python scripts/reproduce_all.pyThis verifies the committed scientific artifacts and aggregate-result hashes,
regenerates the synthetic examples, and runs the test suite. If the lawful
third-party inputs are present in the documented local paths, the same command
also rebuilds the real-data audits and aggregate analyses. Use
--rerun-benchmark only when an explicit recomputation of all 19,200 benchmark
panels is required.
For development checks:
uv run ruff check .
uv run mypy src
uv run pytest -q
uv run mkdocs build --strict- Getting started
- Methodology and interpretation
- Calibration tutorial
- Data acquisition and governance
- Python API
Citation metadata is provided in CITATION.cff. Version 1.0.1 is
archived at doi:10.5281/zenodo.21855066.
CoderDrift is released under the BSD 3-Clause License.