Repository navigation
Releases
CrossSource v0.3 — 150-item CourtListener dataset with labelling protocol and inter-rater agreement
Second dataset release. 150 items (75 qa_citation, 75 citation_check) over 45 public US opinions from CourtListener (27 federal circuit, 18 state supreme court, 2018+), 15 legal topics, full provenance per item (opinion ID, URL, exact passage, cited authority).
What's new since v0.1 (25 items)
LABELING_PROTOCOL.mdv1.1: item types, source rules, label decision rules, rater brief, adjudication rule.AGREEMENT.md: two blind independent model raters, κ 0.98 (citation_check) / 0.84 (qa_citation); 30-item disagreement + audit queue adjudicated by the maintainer with a third independent model rater (27/30 matched).- Harness runs v0.3 end to end. Sanity run on a 30-item sample:
claude-sonnet-5-5qa precision 0.919 / recall 0.718,citation_checkaccuracy 1.00;gpt-5-mini0.763 / 0.718, 0.93. Total cost $4.83. - Validator: 0 problems.
Limits
- Labels are model-drafted and model-checked, not human-labelled item by item;
citation_checkagreement is near ceiling partly because items are constructed and raters share a model family. precedential_statusisnot_checkedfor all 45 opinions (CourtListener exposes no treatment flags); no item relies on an abrogated or overruled opinion as far as checked.- Sanity-run sample is 13 + 15 items, not a full benchmark run.
Cite the version you used: concept DOI 10.5281/zenodo.21969577.
CrossSource v0.1.0 — eval-driven RAG citation-quality methodology
Add author byline linking to zoebnomi.com
CrossSource v0.1
First public release. 25 human-authored golden questions over 22 public court opinions; four scoring dimensions with a citation-failure taxonomy; LLM judge blind-validated at 100% human agreement. Baseline vs. citation-discipline configs: precision 0.981 → 0.994, faithfulness 1.000, recall 0.760 (retrieval-capped). See README for reproduction.
Full Changelog: https://github.com/zoeb-nomi/crosssource/commits/v0.1