Benchmarks

This workspace contains two Kaggle Benchmarks submission scaffolds for the Measuring Progress Toward AGI hackathon.

RecordBound

RecordBound is the active benchmark-and-training platform for the Executive Functions track, with a secondary Metacognition analysis lens. It is built for model developers and eval teams who want to measure whether a role-conditioned or fine-tuned model stays governed by the record when advocacy pressure, ambiguity, and late evidence pull it toward cherry-picking, overclaiming, fake citation, or weak belief updating.

The legal setting is the pressure environment, not the product target. RecordBound is not intended to provide legal advice.

Kaggle Submission Assets

• public benchmark bundle: output/recordbound_platform/benchmark_package.zip
• judge-facing validation bundle: output/recordbound_platform/validation_package.zip
• Kaggle writeup draft: recordbound_writeup.md
• card and thumbnail image: assets/recordbound_thumbnail.png

RecordBound Platform Highlights

• split-aware release surfaces:
  • recordbound_train
  • recordbound_dev
  • recordbound_public_eval
  • recordbound_private_holdout
• default platform release target: 10,000+ cases across splits
• current public benchmark surface remains compatible with the original role-task wiring
• training-oriented supervision layers:
  • reference structured outputs for each role
  • faithful vs distorted preference pairs
  • auxiliary rationales for non-scoring training use
  • citation-grounding targets
• provenance and contamination metadata:
  • case_source_type
  • authoring_mode
  • holdout_status
  • generator_family
  • contamination_risk
• stress-profile metadata:
  • role pressure severity
  • contradiction density
  • update severity
  • distractor salience
  • evidence ambiguity
• validation and trust artifacts:
  • split manifest
  • schema and leakage checks
  • baseline separation summary
  • role-pressure widening diagnostics
  • package contents report

Current Content Highlights

• 500-case legacy single-surface dataset still available for backward compatibility and refreshed to cover all 12 active dispute domains
• 12 active dispute domains:
  • contract
  • employment
  • property_damage
  • vendor_dispute
  • workplace_incident
  • landlord_tenant
  • personal_injury
  • consumer_fraud
  • insurance_coverage
  • construction_delay
  • product_liability
  • data_breach
• 6 evaluated roles:
  • plaintiff_counsel
  • defendant_counsel
  • mediator
  • judge
  • investigator
  • truth_reviewer
• richer evidence mix across:
  • documents
  • digital_metadata
  • eyewitness
  • expert_testimony
  • video/audio
  • physical_evidence
• light admissibility layer with symbolic classifications
• multiple surface realizations for case files, evidence packets, and submissions to reduce template overfitting
• added task families for:
  • evidence ranking
  • timeline reconstruction
  • post-update revision
  • missing-evidence requests
  • admissibility audit
  • contradiction spotting
• explicit penalties for:
  • hallucinated or unsupported facts
  • hidden adverse evidence
  • failure to update
  • role-based distortion
  • fake evidence citation
• missing-evidence task now uses mixed candidate pools with distractors instead of gold-only request IDs

Key files

• recordbound/core.py: generator, gold labels, scorers, and cross-role integrity logic
• recordbound/generate_dataset.py: dataset builder
• recordbound/release.py: split-aware platform release generator, packaging, and validation reports
• recordbound/kaggle_benchmark.py: Kaggle Benchmarks tasks and benchmark bundle
• recordbound/baselines.py: scripted local baselines for role tasks and new v3 task families
• tests/test_recordbound.py: local validation suite
• recordbound_writeup.md: writeup draft
• recordbound_research_prompt.md: repo-grounded judge-facing audit prompt for deep research passes

Generate the dataset

python3 -m recordbound.generate_dataset

Default output:

data/recordbound_v3.csv

Generate the split-aware platform release

python3 -m recordbound.release generate --total-cases 10000

Default output root:

output/recordbound_platform

Validate a release

python3 -m recordbound.release validate --release-root output/recordbound_platform

Run local validation

python3 -m unittest tests.test_recordbound

Current public task bundle

The exported RecordBound Kaggle benchmark now uses only the seven core tasks below for the headline score. The six supporting probes remain available as standalone diagnostic tasks and under a separate RecordBound Diagnostics benchmark so capability isolation stays aligned with the writeup.

• RecordBound: Plaintiff Counsel
• RecordBound: Defendant Counsel
• RecordBound: Mediator
• RecordBound: Judge
• RecordBound: Investigator
• RecordBound: Truth Reviewer
• RecordBound: Cross-Role Integrity
• RecordBound: Evidence Ranking
• RecordBound: Timeline Reconstruction
• RecordBound: Post-Update Revision
• RecordBound: Missing-Evidence Request
• RecordBound: Admissibility Audit
• RecordBound: Contradiction Spotting

BoundarySense

BoundarySense remains in the repo as the earlier metacognition benchmark scaffold.

Key files:

• boundarysense/core.py
• boundarysense/generate_dataset.py
• boundarysense/kaggle_benchmark.py
• writeup.md