Skip to content

Provenance and Audit

Jump to bottom
Allen Byrd edited this page Jun 2, 2026 · 1 revision

Provenance & Audit — the tamper-evident decision trail

Every decision RegRails' engine makes can be written to an append-only, hash-chained log. A single pass over that log proves the entire decision history is intact — or pinpoints exactly where it was altered. This is the auditability story: not "trust the tool," but "re-derive the hashes yourself."

Source of truth: src/regrails/audit.py. The interactive verifier lives at the /provenance web route; the CLI equivalent is regrails audit verify.

Two layers of audit

audit.py provides two distinct mechanisms — keep them separate:

Layer Function What it is
1. Event log emit_event(action, payload, sink) An append-only JSONL stream of named actions (EventAction), each {ts, action, payload}. Mirrors Evidentia's EventAction(str, Enum) pattern. This records that things happened (a rule was encoded, a faithfulness check passed, the guardrail blocked).
2. Decision provenance append_decision(...) / verify_chain(...) A tamper-evident, hash-chained log of GuardrailDecision records. This is the cryptographic part: each record commits to the one before it, so the history cannot be silently edited.

The event log answers "what happened, in order." The decision chain answers "has this record of decisions been altered since it was written." This page is mostly about the second.

The EventAction enum

The auditable actions are a closed, string-valued enum (class EventAction(str, Enum)), grouped by subsystem — encoding (RULE_ENCODED, SECTION_LOADED), faithfulness (FAITHFULNESS_CHECK_PASSED / _FAILED), research, guardrail outcomes (GUARDRAIL_ALLOWED, GUARDRAIL_BLOCKED, the four escalation/insufficient/out-of-scope variants), provenance (DECISION_RECORDED, AUDIT_CHAIN_VERIFIED, AUDIT_CHAIN_BROKEN), and LLM (LLM_CALL_SUCCEEDED / _FAILED / LLM_FALLBACK_USED). Using a string-valued enum keeps the on-disk action names stable and greppable.

How the hash chain works

For each decision record:

record_hash = SHA-256( prev_hash + canonical(decision) )

Four steps (the same four the /provenance page walks a visitor through):

  1. Canonicalize. The decision object is serialized deterministically — json.dumps(decision, sort_keys=True, separators=(",", ":"), ensure_ascii=True) — so the same decision produces the same bytes every time, regardless of key order. (_canonical() in audit.py.)
  2. Link. Compute record_hash = SHA-256(prev_hash + canonical(decision)). The new hash folds in the previous record's hash, so each record commits to its entire predecessor chain.
  3. Append. The first record points at a genesis hash of 64 zeros (GENESIS_HASH = "0" * 64); every record after it points at its predecessor's record_hash. The on-disk record is {"prev_hash": ..., "record_hash": ..., "decision": {...}}.
  4. Verify. Re-derive each hash and confirm every link matches.

Because every link folds in the one before it, editing, reordering, inserting, or deleting any record silently rewrites every hash downstream — so a single linear pass detects tampering and points at the first broken link.

Honest scope. This is a keyless integrity chain, not a signed one. It proves a log has not been altered relative to itself after writing; it does not, on its own, bind the log to an authenticated author the way a cryptographic signature would. audit.py's own docstring states this directly: it "echoes the cryptographic evidence-binding idea in Evidentia (Sigstore-signed artifacts) at POC scale: no keys, just a verifiable chain." For who-signed-it assurance at the package level, see Releases & Safeguards (PEP 740 + SLSA provenance on the PyPI artifacts).

verify_chain — what it actually checks

verify_chain(path) -> (ok: bool, problems: list[str]) walks the file line by line and, for every record in order, checks two things:

  1. Linkage — the record's prev_hash equals the prior record's record_hash (or GENESIS_HASH for the first record). A mismatch yields a prev_hash linkage broken problem naming the record.
  2. Content integrity — the stored record_hash equals a freshly recomputed SHA-256(prev_hash + canonical(decision)). A mismatch yields a record_hash mismatch — decision content was altered after writing problem.

It also flags invalid JSON per record and reports an empty chain as a problem. It returns ok = True only when the problems list is empty. The function is pure and offline — no keys, no network, no model.

append_decision is the writer: it reads the last record's hash via _last_record_hash, computes the new record_hash, and appends the {prev_hash, record_hash, decision} record as one JSONL line, returning the new hash.

The tamper demo: edit, insert, delete all break the chain

The chain catches all three classes of tampering, because each changes either a record's own bytes or the predecessor a later record commits to:

Tampering Why it breaks
Edit a stored decision (even one character) The recomputed record_hash no longer matches the stored one → content-integrity failure at that record.
Insert a record mid-chain Its prev_hash won't match the prior record_hash, and the following record's prev_hash won't match the inserted record's hash → linkage failure.
Delete a record The next record's prev_hash points at a record_hash that is no longer present → linkage failure cascading downstream.

Reproduce it from the CLI

The committed demo chain is verifiable straight from a clone, no API key required (this is also a CI gate — see Releases & Safeguards):

# Intact chain → reports verified
uv run regrails audit verify demo/recorded-runs/decisions.chain.jsonl

# Tamper a COPY (flip a byte / drop a line), then re-verify → reports broken,
# naming the first record whose hash no longer matches.

The same regrails.audit code path backs both the CLI and the web verifier, so the two agree by construction.

The /provenance web verifier

The /provenance route makes tamper-evidence tangible in the browser:

  1. Explains the chain using the live sample's own hashes (a small chain visualization showing each record's prev → hash links, with the genesis record shown as genesis (00…00)).
  2. Loads a real 3-record sample chain (/data/provenance-sample.jsonl) into an editable textarea. The three records model an allow (FERPA, low), an escalate_human_review (Title IV, high — a loan-default eligibility question), and a block (FERPA, medium).
  3. Verifies server-side via the /api/verify serverless function — which runs the same regrails.audit code the CLI uses — and renders an intact/green or broken/red panel with the engine's actual problems list.
  4. "Tamper a record" button flips exactly one hex character inside the second record's record_hash, leaving everything else byte-identical, so a re-verify fails mid-chain and the visitor feels the chain catch the edit. A "Restore sample" button resets it.

Two deliberate security/correctness properties of the page itself:

  • The SPA never recomputes hashes client-side. The textarea content is POSTed verbatim to the server verifier; the authoritative check is always server-side. (This avoids a "marking your own homework" trust gap where the browser could be made to claim a broken chain is intact.)
  • Untrusted log text only ever reaches the DOM as React text children (never innerHTML), so a pasted-in malicious log can't inject script into the page.

Why this design

The thesis RegRails inherits from Evidentia: an AI-in-the-loop governance system is only trustworthy if its decision trail is verifiable by a third party without trusting the tool that produced it. A hash chain delivers exactly that property at proof-of-concept scale — anyone with the log file and a SHA-256 implementation can re-derive every link and detect any after-the-fact edit, with no access to RegRails, no keys, and no network. That is the auditability the role charter asks for ("auditability, AI-in-the-loop governance"), made concrete and runnable.

See also

  • Benchmark — the held-out study; note that the guarded arm's decisions are the deterministic outputs this chain records.
  • Methodology & Limitations — where the keyless-chain scope sits relative to faithfulness and legal correctness.
  • Releases & Safeguards — the signed-artifact layer (PEP 740 + SLSA) that complements this keyless decision chain.

Clone this wiki locally