Repository navigation
Methodology and Limitations
This page is written to be checked, not believed. Every number it states is regenerated by a committed command, and every limitation is stated plainly. The canonical document is
docs/METHODOLOGY.md; the coverage matrix it references isdocs/COVERAGE.md.
RegRails is a proof-of-concept. It is not legal advice, not a compliance certification, and not production software. Nothing in the repository should be relied on to make a real disclosure, eligibility, or aid decision about a real student.
RegRails encodes selected provisions of two U.S. higher-education frameworks into machine-readable rules and wires them into an AI advisor as a deny-by-default, risk-tiered guardrail that is consulted before the model answers:
- FERPA — 34 CFR Part 99, Subpart D (education-record disclosure).
- Title IV — a subset of 34 CFR Part 668: Satisfactory Academic Progress (§ 668.34) and student eligibility (§ 668.32).
| It is | It is not |
|---|---|
| A worked example of the policy-as-code pattern for AI-in-the-loop governance | A statement that any institution using it is "compliant" |
| A deterministic decision engine with a verbatim-text faithfulness check | A substitute for review by a FERPA or financial-aid officer |
| A hash-chained decision log (see Provenance & Audit) | A system that "prevents violations" |
| A published rule-to-scenario coverage matrix with its gaps visible | A guarantee that the legal reading of any encoded rule is correct |
It produces auditable guardrails and escalation paths. Whether the encoded reading of a rule is legally correct is outside what the tooling can establish — see The faithfulness gate below.
The encoding covers eight CFR sections / 37 rules. This table is what is actually present in data/encoded/ — exhaustive, not illustrative:
| Framework | Encoded sections | Rules encoded |
|---|---|---|
| FERPA (Part 99, Subpart D) | §§ 99.30, 99.31, 99.32, 99.33, 99.36, 99.37 | 23 |
| Title IV (Part 668, subset) | §§ 668.34 (SAP), 668.32 (eligibility) | 14 |
| Total | 8 sections | 37 |
The boundary matters more than the coverage, so the in-scope CFR that is deliberately NOT encoded is stated explicitly in METHODOLOGY.md — for example, most of § 99.31's roughly-a-dozen no-consent disclosure exceptions are not encoded (8 of the prongs are), and sections that are referenced by encoded rules but not themselves modeled (§ 99.34, § 99.35, § 99.38, § 99.39; § 668.35 reinstatement-after-default) are named. The bundled CFR source files even say so in their headers (data/cfr/title-iv-subset.txt: "This is a SUBSET … it is not the complete Part 668").
RegRails uses three separate scenario sets for three different jobs. A reader skimming the repo might see "12," "22," and "24" and suspect an inconsistency; here is why they are intentionally different:
| Corpus | Size | Location | Purpose |
|---|---|---|---|
| Demo | 12 scenarios |
demo/queries.yaml + demo/recorded-runs/
|
A two-framework walkthrough (python -m regrails.demo --all) showing the engine + the LLM renderer end to end. |
| Golden (rule-design) | 22 labeled scenarios (12 FERPA + 10 Title IV; one Title IV case is an out-of-scope control) |
tests/golden/ferpa.jsonl, tests/golden/title_iv.jsonl
|
The labeled fixtures the rules are designed and regression-tested against; each pins (outcome, risk_tier, human_gate, citations). |
| Held-out benchmark | 24 scenarios | bench/scenarios.heldout.jsonl |
The independent evaluation set, authored without reading the rules and committed before any model ran (see Benchmark). |
The held-out 24 is deliberately distinct from the rule-design 22 — that separation is the whole point of a held-out evaluation (you cannot measure generalization on the data you tuned on), and EVAL.md states it explicitly. The demo 12 is a third thing again: a narrated walkthrough, not a labeled test set. Three corpora, three jobs, no contradiction.
The 22 golden scenarios were labeled by a single person (the project author), reasoning from the regulatory text and the committed research snapshots, without independent review by a FERPA or financial-aid practitioner. Each expected outcome is pinned to its controlling CFR citation (e.g. the loan-default scenario is labeled escalate_human_review / high / human_gate=true citing 34-CFR-668.32(g)(1)) — that citation is the provenance: it names the clause that justifies the label.
The honest limitation, stated in METHODOLOGY.md §3: these are one defensible reading of the cited clauses, not an adjudicated ground truth. A second qualified reviewer could disagree with a label, and that disagreement would be a legitimate finding, not a bug. The benchmark's own ho-07 / ho-08 disagreements are a live example of exactly this tension.
This is the line RegRails is most careful about, and it is repeated in README.md, METHODOLOGY.md §5/§7, and the /methodology web route.
regrails check faithfulness (src/regrails/faithfulness.py) verifies verbatim text, and only that. For each rule it checks the source_quote against the bundled CFR text for the matching section. A rule passes if the quote is a character-for-character substring of the bundled text OR its token coverage (|quote ∩ section| / |quote|) is at least 0.85. Combined with a SHA-256 over each bundled section, this gives a traceable chain: public CFR text → bundle → encoded source_quote. The repo reports 37/37 rules pass this gate.
The words attributed to a section are actually in that section — the encoding cannot silently paraphrase or fabricate regulatory text.
It says nothing about whether the semantic encoding is a correct legal reading. The gate does not check whether the chosen rule_type, the triggers that route a query to the rule, or the assigned risk_tier are right. A rule can quote the statute perfectly and still map to the wrong outcome, and the faithfulness gate will pass it. Establishing semantic correctness requires review by an institutional FERPA or financial-aid officer; the faithfulness gate is a necessary check, not a sufficient one.
This distinction is the difference between "we did not make up the law" (which RegRails proves) and "we read the law correctly" (which RegRails explicitly does not claim to prove).
regrails coverage report builds a rule→scenario traceability matrix from the golden corpus and writes it to COVERAGE.md. The headline:
37 encoded rules; 31 are exercised by at least one of 22 golden scenarios; the remaining 6 rules have no scenario yet.
The six uncovered rules are listed, not hidden:
| Gap rule ID | Section | What it covers |
|---|---|---|
FERPA-99.30-3 |
§ 99.30 | electronic-consent validity |
FERPA-99.31-A3 |
§ 99.31 | audit / evaluation exception |
FERPA-99.31-A4 |
§ 99.31 | financial-aid exception |
FERPA-99.31-A9 |
§ 99.31 | judicial order / subpoena |
FERPA-99.33-A2 |
§ 99.33 | purpose-limitation on received PII |
FERPA-99.36-B1 |
§ 99.36 | disciplinary-action information in records |
A rule with no scenario means the engine's behavior on it is unverified by any example, so its routing should be treated as unproven until a scenario exists. The reason to publish the gaps is precisely that a coverage matrix showing only the covered rules would overstate what has been checked — the opposite of the discipline this project is trying to demonstrate.
Each decision carries a risk tier derived from reversibility and blast radius, which drives a routing stance:
- low / reversible → may be automated (explaining a rule, an out-of-scope question, an aggregate de-identified release).
- medium → automate only with a cited obligation or a quick check (consent defect, directory opt-out state, recordkeeping duty).
-
high / irreversible → mandatory human gate. Loss of aid eligibility, loan default, a yes/no eligibility determination — the engine refuses to let the AI answer and routes to a human (the financial-aid office). In the Title IV encoding, § 668.34(a)(7) (SAP termination) and § 668.32(g)(1) (default) carry the
hightier and force the gate.
Stated honestly in METHODOLOGY.md §6: this is a design stance, not a guarantee. Tiers are assigned by the author per rule; the engine enforces the stance deterministically, but it cannot promise every high-stakes situation has been classified as high. A miscategorized rule would route a serious case as medium — and the gate is only as good as the tiering, which is itself unreviewed (same caveat as label provenance above).
From METHODOLOGY.md §7, because a guardrail that hides its failure modes is worse than none:
- Verbatim faithfulness ≠ semantic correctness. A rule can quote the CFR exactly and still encode the wrong legal conclusion; the faithfulness gate will pass it.
-
Deterministic tag-matching can mis-route novel phrasing. Routing depends on a fixed set of
triggersand the structured fields of aConsultationRequest. A real query phrased outside those tags can land on the wrong cascade — or on the deny-by-default path — for the wrong reason. -
The LLM renderer can still mis-phrase a correct decision. The engine decides deterministically before any model call; the LLM only renders the reply. A correct
block/escalatedecision can still be worded misleadingly. The decision is reproducible without the model; the prose is not guaranteed faithful to it. - Subset coverage. Only eight sections are encoded and only 31 of 37 rules have a golden scenario. A situation governed by an unencoded section, or an encoded-but-untested rule, has no verified behavior here.
RegRails can export its rules and decisions in standards-aligned formats, and it is careful about what "aligned" means:
-
OSCAL —
regrails export oscalemits an OSCAL 1.1.2-shaped catalog of the 37 rules. The/exportsweb route states this plainly: the output is structurally aligned, NOT NIST-validated against the official OSCAL schema. "Shaped" is the honest word — it is the right structure and version string, not a certified-conformant document. -
SARIF —
regrails decide --format sarifemits a genuine OASIS SARIF v2.1.0 document (version2.1.0, with the SARIF 2.1.0$schema), suitable for a CI security gate.
The distinction is deliberate: claiming "OSCAL-validated" would be an overclaim the tooling cannot back; "OSCAL-shaped" is exactly what was built.
Every number on this page is regenerated by the following, from a fresh clone (uv sync --extra dev first). None require an API key:
# Faithfulness: each rule's source_quote is verbatim CFR (substring OR coverage >= 0.85)
uv run regrails check faithfulness
# Coverage: rebuild the rule -> scenario matrix and the covered/total/gap counts
uv run regrails coverage report
# Full test suite (golden corpus + tamper-detection)
uv run pytest -q
# Hash-chained decision provenance: recompute the chain and detect any edit/insert/delete
uv run regrails audit verify demo/recorded-runs/decisions.chain.jsonlThe encoded rules live in data/encoded/, the bundled verbatim CFR text in data/cfr/, the labeled scenarios in tests/golden/, and the research snapshots in research/snapshots/. COVERAGE.md is the generated output of the second command and the authoritative source for the coverage numbers.
- Benchmark — the held-out study and the honest ho-07 / ho-08 disagreements.
- Provenance & Audit — the hash-chained decision trail and its keyless scope.
- Releases & Safeguards — how the package is published and what is checked before and after a tag.