Reality Grading Loop

Reality Grading Loop

A system that grades its own homework learns to write confident nonsense. This loop takes the red pen out of the model's hands and gives it to reality.

The problem

If a learning loop rewards the model's self-reported success, it trains the model to look successful rather than to be honest — a confident false "done" is rewarded like a true one, so the system drifts toward plausible fabrication. This loop rewards only outcomes that reality confirms.

The core move — stake a falsifiable hypothesis

A finished mission does not emit "success." It emits a hypothesis: a concrete, checkable prediction, e.g. "the report exists at docs/audit/report.md, has a 'Findings' section, ≥ 2 KB" — something that can be proven false by looking. Vague claims can't be graded; hypotheses can.

The core discipline — Evidence Normalization

A provider's assertion is not evidence. Evidence is what the filesystem and the ledger say.

The judge never reads the model's claim. It reads reality: does the file exist? does it have the claimed properties? do the ledger records corroborate it? The model's "I did it" is treated with the same suspicion as any unverified claim. This is the mechanism behind never fabricate: a fabricated result cannot survive grading.

The loop

flowchart TB
    M[Mission completes] --> H[Stake HYPOTHESIS<br/>falsifiable · judge-versioned]
    H --> W[Wait for horizon<br/>the Outcome Clock]
    W --> J[Judge grades vs EVIDENCE<br/>filesystem + ledger]
    J --> V{Verdict}
    V -->|confirmed| EP[Validated Episode<br/>immutable: hash + judge version]
    V -->|refuted| BR[Belief Revision<br/>lower the belief's confidence]
    V -->|abstain| ND["'no data yet' — honest unknown"]

Loading

• confirmed → an immutable Validated Episode (artifact hash + judge version at stake time).
• refuted → a Belief Revision attributed to Reality (outcome) — corrigibility, mechanised.
• abstain → an honest "no data yet", never a fake zero.

Versioned judges

The judge will get smarter (rg-1 grades by existence; rg-2 by content quality). Two rules keep history honest: the judge version is pinned into each hypothesis at stake time (history is immutable), and a judge upgrade carries a Migration Note through the governance ceremony — never a bare version bump, because changing how "success" is graded is a governed act.

The honesty dividend

In the reference lineage, the first live mission failed three times before it succeeded — and on those failures it staked zero hypotheses. The system refused to claim a success it did not have. That restraint, not the eventual completion, is the real victory: the loop optimises for telling the truth, not looking good — the property that compounds over years.

---

📖 Full document: docs/05 · 🗳️ Decision: ADR-0003 · 🧩 Code: reality_grading.py → Next: Governance and Constitution