Skip to content

CrossSource v0.3 — 150-item CourtListener dataset with labelling protocol and inter-rater agreement

Latest

Choose a tag to compare

@zoeb-nomi zoeb-nomi released this 20 Oct 08:08
90f10be

Second dataset release. 150 items (75 qa_citation, 75 citation_check) over 45 public US opinions from CourtListener (27 federal circuit, 18 state supreme court, 2018+), 15 legal topics, full provenance per item (opinion ID, URL, exact passage, cited authority).

What's new since v0.1 (25 items)

  • LABELING_PROTOCOL.md v1.1: item types, source rules, label decision rules, rater brief, adjudication rule.
  • AGREEMENT.md: two blind independent model raters, κ 0.98 (citation_check) / 0.84 (qa_citation); 30-item disagreement + audit queue adjudicated by the maintainer with a third independent model rater (27/30 matched).
  • Harness runs v0.3 end to end. Sanity run on a 30-item sample: claude-sonnet-5-5 qa precision 0.919 / recall 0.718, citation_check accuracy 1.00; gpt-5-mini 0.763 / 0.718, 0.93. Total cost $4.83.
  • Validator: 0 problems.

Limits

  • Labels are model-drafted and model-checked, not human-labelled item by item; citation_check agreement is near ceiling partly because items are constructed and raters share a model family.
  • precedential_status is not_checked for all 45 opinions (CourtListener exposes no treatment flags); no item relies on an abrogated or overruled opinion as far as checked.
  • Sanity-run sample is 13 + 15 items, not a full benchmark run.

Cite the version you used: concept DOI 10.5281/zenodo.21969577.