feat(tools): Virgin.root <-> .ut corpus sync verifier (warn-only) (#675)

[jsavin]

Summary

Adds tools/verify_virgin_root_sync.py (+ thin .sh wrapper) that detects content drift between databases/Virgin.root and the .ut corpus at usertalk_scripts/Frontier.root/**/*.ut. Fixes #675.

The .ut corpus exists because git can't diff binary .root files. Both are tracked; nothing previously enforced sync between them.

What landed

• tools/verify_virgin_root_sync.py — the verifier. Long-lived frontier-cli --lock-opened-roots --protocol --skip-startup session; per-.ut script/eval of string(<dotted.path>); normalize kernel output; byte-compare against on-disk .ut. Modes: default (incremental — staged .ut files from git diff --cached), --full (entire corpus, 3058 in-scope files), --paths FILE ... (explicit list). --rewrite flag overwrites drifted .ut files with kernel-canonical form (opt-in cleanup tool).
• tools/verify_virgin_root_sync.sh — thin shell wrapper exec'ing the .py (matches the rest of tools/*.sh).
• Makefile (new top-level) — proxy targets: build/unit/integration/verify-odb-sync/help. Existing per-area Makefiles in tests/ and frontier-cli/ continue to own the real logic.
• tests/Makefile — adds verify-odb-sync proxy + test-odb-sync target wired into test-integration.
• tests/verify_odb_sync_test.sh — positive (known-good file matches) + negative (synthetic drift via --corpus-root sandbox is detected, exit 1) regression tests.
• tools/hooks/pre-commit-integration-tests — HOOK_VERSION 8 -> 9; adds a sync-check section. Warn-only (prints drift to stderr, does NOT block the commit). Rationale below.
• Docs — CLAUDE.md Quick Reference lists the three invocation paths; docs/ODB_SCRIPT_EDITING.md updates the ODB: pre-commit + CI verification that Virgin.root and .ut corpus stay in sync #675 entry.

Normalization pipeline

The kernel emits MacRoman bytes inside JSON (NOT valid UTF-8 in the presence of high-bit chars), CR line endings, « (MacRoman 0xC7) as the single-line comment marker, and outline structure markers around all-comments subtrees. The corpus is UTF-8, LF, // comments. The verifier applies:

1. Decode JSON escapes (\r \t \n \" \\ \/ \uXXXX)
2. MacRoman → UTF-8 (via Python's mac_roman codec)
3. « → //; » → empty
4. CR → LF
5. Strip outline structure markers around all-comment subtrees (matches langstripstructuremarkers install behavior)
6. Trim trailing LF symmetrically

Why warn-only, not gating

The corpus on develop carries ~374 of 3058 files (12.2%) of drift that predates this verifier — primarily from the original tools/convert_ut_encoding.sh pass (commit fa468dde0) decoding MacRoman-encoded source bytes as Windows-1252, which produced systematic miscoding (MacRoman 0xD5 = ’ U+2019 was read as Windows-1252 Õ and re-encoded as UTF-8 Õ). The kernel still emits the correct MacRoman characters.

Reviewer (gate) feedback initially diagnosed some of this as my structure-marker regex over-stripping. Diagnostic finding: the regex is NOT firing on the cited files; the drift is genuine kernel-vs-disk lexical difference (the .ut export tool moved close-braces from end-of-body-line to start-of-next-comment-line, inserting semicolons). The lexical difference is consistent and reproducible across all 374 affected files. Re-engineering the historical export's transformations is out of scope for this PR.

Promoting from warn-only to hard gate requires a corpus cleanup pass (drift-by-rewrite using verify_virgin_root_sync.sh --rewrite --full then inspecting the diff). Tracked for follow-up. The verifier landing today still provides value as a diagnostic + as --rewrite infrastructure for the cleanup.

/gate review findings (all addressed)

P1 #1 (bar-raiser, structure-marker regex over-strip) — diagnostically incorrect. Verified the regex does NOT fire on the cited files; drift there is real kernel-vs-disk lexical difference (see normalization-pipeline rationale above). No code change; documented in commit 4d836aba6 so future readers don't try to re-fix it.

P1 #2 (security, --rewrite writes attacker-controlled bytes to attacker-controlled paths) — Fixed in 4d836aba6. New _path_within() helper using os.path.commonpath. --rewrite refuses paths that don't resolve under --corpus-root (reason rewrite refused: outside corpus root). Mitigates CWE-22 / CWE-73.

P1 #3 (bar-raiser, ProtocolSession has no read timeout) — Fixed in 4d836aba6. New _readline_with_timeout() using selectors + os.read on the fd; 30s default, overridable via FRONTIER_VERIFIER_TIMEOUT env. On timeout the subprocess is killed and RuntimeError raised; caller surfaces exit code 2 (infra error).

P1 #4 (security, pre-commit hook word-splits paths) — Fixed in 4d836aba6. Hook now reads staged paths NUL-delimited via git diff -z + while read -d '' into a bash array, then expands as "${UT_STAGED_PATHS[@]}". Eliminates the silent-bypass primitive for paths with spaces or shell globs.

P2 #3 (bar-raiser, pre-commit silently skips when frontier-cli unbuilt) — Fixed in 4d836aba6. Hook now prints a yellow NOTE: "skipping ODB sync check (frontier-cli not built; run \make build`)"`.

P2 #4 (bar-raiser, parts[0] not quoted), P2 #5 (bar-raiser, VALUE_RE field-order coupling), P2 #6 (security, segment validation completeness) — P2 #8 (control-byte rejection) addressed; P2 #4 and P2 #5 deferred (current corpus has no parts[0] that needs quoting, and the VALUE_RE order assumption is documented in the class docstring).

Issue cleanup

• Edited ODB: pre-commit + CI verification that Virgin.root and .ut corpus stay in sync #675 to drop the "CI invokes it" exit-criterion bullet (no build CI exists in this repo; only the server-monitor cron and Claude bot). Reattach when build CI gets stood up.

Test plan

• Unit tests: ./tools/run_headless_tests.sh — 483/483 pass
• Integration: cd tests && make test-integration — 2154 pass, 18 baseline failures preserved (12 html., 6 tcp.), no new regressions
• Verifier regression test (positive + negative): 2/2 pass
• ./tools/verify_virgin_root_sync.sh --paths <known-good>.ut returns 0
• ./tools/verify_virgin_root_sync.sh --corpus-root <sandbox> --paths <tampered>.ut --rewrite refuses outside corpus root
• make verify-odb-sync runs at repo root
• cd tests && make verify-odb-sync runs from tests/
• Pre-commit hook syntax OK (bash -n)
• make help lists targets
• Verifier as gate (deferred until corpus cleanup follow-up)

[jsavin]

/gate Review Round 2 @ dc1633af5

Verdict: PASS WITH NOTES → cleaned to PASS in this commit

Round 2 was triggered by the fix-pass commit 4d836aba6. Both bar-raiser and security re-reviewed.

Tests

• unit: 483/483 pass (cached at dc1633af5)
• integration: 2154/2379 pass; 18 baseline failures preserved (12 html., 6 tcp., identical to develop baseline)
• verifier regression: 2/2 pass

Reviewers

• bar-raiser: ran (always) — verdict APPROVE, all 4 round-1 findings closed, 3 new P2s flagged
• security: ran (always) — verdict CLEAN, all 4 round-1 findings closed, zero new findings
• concurrency: skipped (auto: no triggers matched)
• swiftui: skipped (excluded)

Round 1 findings — verification

Finding	Status
P1 #1 (bar-raiser): structure-marker regex over-strip	Diagnostically false. The regex does NOT fire on the cited files. Drift is genuine kernel-vs-disk lexical difference. Re-confirmed by bar-raiser in round 2.
P1 #2 (security): --rewrite containment	Closed (4d836aba6) — _path_within() + resolve. Round 2 confirms.
P1 #3 (bar-raiser): no read timeout	Closed (4d836aba6) — _readline_with_timeout() + persistent _read_buf. Round 2 confirms.
P1 #4 (security): hook word-splitting	Closed (4d836aba6) — NUL-delimited paths + bash array. Round 2 confirms.
P2 #3 (bar-raiser): hook silent skip	Closed (4d836aba6) — yellow NOTE printed. Round 2 confirms.
P2 #8 (security): segment validation	Closed (4d836aba6) — control bytes rejected; ] permitted for legitimate corpus paths. Round 2 confirms.

Round 2 new findings

#	Sev	Finding	Status
1	P2	Three em-dashes in docstrings (project rule is ASCII-only in code comments)	Fixed dc1633af5
2	P2	_path_within() is purely lexical; safe today but invites future misuse	Fixed dc1633af5 — resolves internally now
3	P2	No unit tests for _path_within / _quote_segment control-byte rejection / _readline_with_timeout	Deferred — shell regression test covers happy paths; adding pytest unit tests is a separate follow-up

Bar-raiser on warn-only landing

12.2% baseline drift (374/3058) is genuine pre-existing lexical corruption from the Windows-1252-as-MacRoman conversion era, not anything this PR can responsibly fix in-band. Shipping warn-only with a documented gate-flip predicate is the right call. Blocking on corpus cleanup would gate visibility behind a multi-week task and provides no incremental safety — reviewers already see the warning. APPROVE the warn-only position.

Security on the fix pass

All four round-1 findings are correctly closed. No new findings at >=70% confidence. Recommendation: merge-ready from the security side.

Final state after dc1633af5

• 0 P0
• 0 P1
• 1 P2 deferred (unit tests for new helpers — visible follow-up)

Branch is ready to merge.

[jsavin]

/auto Summary

What went well:

• Discovery during smoke-testing surfaced two critical normalization gaps (kernel emits MacRoman bytes inside JSON, not UTF-8; string(@adrScript) returns the path string, not script body) BEFORE writing the verifier. Switching to string(<dotted.path>) and Python+regex-on-raw-bytes (instead of jq+json.loads) was the right pivot.
• /gate round 1 caught real P1s (path-traversal on --rewrite, no read timeout, hook word-splitting). All addressed in one fix commit; round 2 confirmed clean.
• Bar-raiser's round-1 P1 Repo Modernization: 64‑bit/ARM + DB Header Rev; Headless/Portable Runtime; Headless EFP Routing for file.* and Tests #1 (structure-marker regex over-strip) was diagnostically false — verified by running normalize with/without step 5 across all 4 cited files; the drift is genuine kernel-vs-disk lexical difference. Pushing back with evidence was correct.

What could improve:

• The 374-file historical drift (12.2% of corpus) is genuine pre-existing damage from the original convert_ut_encoding.sh pass (Windows-1252 assumption on MacRoman source). Should be re-cleaned via ./tools/verify_virgin_root_sync.sh --rewrite --full in a follow-up PR before flipping the hook from warn-only to hard gate.
• Phase 7 issue-filing was blocked by the auto-classifier (read as scope escalation). The two follow-ups (corpus cleanup, unit tests for new helpers) need to be filed manually.
• ExitWorktree wedge at Phase 6 happened again (n=3 of 3 in Frontier worktrees now: PR fix(tcp): gate variant verbs on stream state (#666) #669 chained, docs(tcp): document statusStream's full sentinel set + remove dead switch arm (#657) #670 chained, feat(tools): Virgin.root <-> .ut corpus sync verifier (warn-only) (#675) #679 single-shot). The SKILL.md's "single-shot has predictive value" claim is now contradicted; worth updating to reflect that Frontier-worktree Phase 6 is effectively guaranteed to wedge.

Recommendations:

• Update ~/.claude/skills/auto/SKILL.md Phase 0 step 7 note: for Frontier specifically, plan on session-handoff merge regardless of chain vs single-shot. n=3 / 3.
• File follow-up for corpus cleanup pass (--rewrite --full then inspect diff, commit, flip warn-only to hard-gate).
• File follow-up for unit tests covering _path_within, _quote_segment control-byte rejection, _readline_with_timeout (round-2 P2 docs: UTF-8 transition plan #3 deferred).

🤖 Generated with Claude Code /auto