feat(hooks): factual-claim scan in end-of-turn Stop hook (autonomous-self-audit §11.1 interim) - #96
Merged
Merged
Conversation
…self-audit §11.1 interim) Implements Option (ii) — the E-checklist Stop-gate interim from the merged autonomous-self-audit R-phase (#95 §10.5/§11.1). Extends the live Stop hook with a deterministic scan that enumerates factual-claim shapes (numeric counts, file:line citations, negative-existence) in the last assistant turn and appends an item-specific re-verify instruction — targeting the at-write-time factual class (incidents #1 "4+ files", #5 ":30", #2 negative-existence) that the generic recap nudge does not force. Fires on claim-PRESENCE regardless of turn length (covers short turns the long_text gate skips). No LLM, no external call (no-paid-llm-in-ci). Honest limit: raises salience, does not structurally force compliance. Also: portable last-assistant-line extraction (grep|tail, was BSD-only tail -r) so the companion test can run on Linux/CI. New behaviour test tests/hooks/eot-claim-scan.test.sh (6 sub-tests incl. mutation) registered in audit-self.yml — the Q-D recursive backstop: the drift-catcher hook is itself caught by a test if it silently breaks. Prior-art: prior-art-evaluations.md#20 (Claude Code hooks API, ADOPT) — extends existing Stop-hook substrate, no new surface. Agent Verifier (REFERENCE/ADAPT per #95 research-patch §5.1) — session-executed verifier pattern; this auto-fires it at the AI's own claims. No new dependency.
artyhoo
added a commit
that referenced
this pull request
May 21, 2026
…empirical research (#97) Empirical-validation arm for Q1 of the autonomous-self-audit line (#95 §10 Q-C, round-3 §3.5): does the live claim-scan (#96) change behaviour, or is compliance voluntary-and-equal? Ran a deterministic no-LLM scorer over 337 real eval-unaware transcripts (266 sessions / 1537 claim-turns) + a controlled planted-error probe on 5 naive sub-agents. H0 NOT rejected (treatment arm N~0, eval-aware, cross-model). Substantive re-frame of Q1: baseline groundedness already ~0.74-0.81 (small headroom); the binding weakness is the DETECTOR — recall ~0.43 on natural numeric phrasings (misses "6 discipline rules", "266 sessions"; missed the one organic failure case), precision ~0.20-0.25 (cry-wolf). Compliance-when-delivered CAN be positive (probe1 corrected 6->8). Pre-registered decision-gate (a/b/c/d) + 4 maintainer decision-needed items. Sibling scorer committed as a reproducibility artefact (NOT CI-wired; promotion to tests/ is Q-E2-gated). principle-10 green; self-application (§7) caught this patch's own headline numbers in the recall blind spot. No mechanism implemented; no rule codified; no hook/settings edit; no PR on a mechanism. The 5 sub-agents were measurement subjects, not implementers. Prior-art: skipped — research patch only, no new capability; eval methodology + pilot, no mechanism adopted
This was referenced May 21, 2026
artyhoo
added a commit
that referenced
this pull request
May 21, 2026
Lands the maintainer's settings.json safety-hardening that was stranded uncommitted on the superseded chore/ssot-karpathy-skills-ref branch (the only non-superseded residue there — every other file already on main via #89/#92/#93/ #94/#96/#97/#98). Committed verbatim. On-thesis: this is edit-time (earliest-channel) enforcement of project safety rules that previously lived only as memory/prose — "no git reset hard ever", no force-push, no skip-verify. The harness now blocks them before they run: - destructive git: force/mirror push, push to main/master, skip-verify commit, hard reset, checkout discard, branch force-delete, filter-branch, update-ref - destructive fs: recursive delete, shred, dd, mkfs, sudo - pipe-to-shell: curl-to-shell, wget-to-shell; raw netcat/socat - secret exfil: gh secret set/delete, gh auth logout, reads of credential dirs (ssh / aws / gnupg / gh config) and dotenv files - self-protection: .git/hooks, .husky, settings.json (Edit/Write deny) Also: portable $CLAUDE_PROJECT_DIR hook paths + ask-question-reminder.sh permission. Prior-art: skipped — harness config hardening (permission deny-list), no new capability code or dependency
5 tasks
artyhoo
added a commit
that referenced
this pull request
May 21, 2026
…#106) Append + in-place refinement of the living history book (v1 technical +86 lines, v2 literary +78). Narrates post-#93 events: the end-of-turn hook saga (#81 broken via systemMessage → #89 reason-channel fix), #94 AskUserQuestion hook, principles 11/12/13 landing, the #96/#97 claim-scan experiment (H0 not rejected), #90 recommendation-gate, and the niche-among-companions chapter. Colophon extended (chapters III–XII, patches to 2026-05-21, PR #85–#99). Preserves continuation work that was stranded uncommitted on the superseded chore/ssot-karpathy branch. Prior-art: skipped — narrative doc continuation, no new capability or dependency.
7 tasks
artyhoo
added a commit
that referenced
this pull request
May 31, 2026
…rdicts for tracing-knowledge-lineages / when-stuck / preserving-productive-tensions + 5 SSOT rows (#92-#96) (#321) Single-session R-phase per .claude/orchestrator-prompts/companion-capability-survey/ kickoff.md + companion-capability-survey-meta-launch/kickoff.md. Surveys 11 Superpowers/Superset/aif-handoff capabilities not previously in SSOT against the /meta-orchestrator pipeline stages. Net additions: - docs/meta-factory/research-patches/2026-06-01-companion-capability-survey.md - prior-art-evaluations.md rows #92-#96 (4 ADOPT VOCABULARY + 1 REJECT) - #55 Last reviewed bumped to 2026-06-01 (writing-skills TDD-for-docs; verdict unchanged at ADAPT) - #86 Trigger to revisit sharpened (Superset Automations + MCP start_agent_session are a separate surface from the worktree-substrate #86 originally rejected) T-CCS-B fired: kickoff §2 claimed 'Superset cron — DeepWiki found NONE'; this survey verified Automations + RRule DO exist per docs.superset.sh/automations. Slack-agent precedent for Superset was the other §2 seed claim — NOT confirmed against source; dropped INCONCLUSIVE. No code changes. No new deps. Verdicts are recommendations per reviewer-discipline.md §2 — maintainer decides per-row adoption. Prior-art: skipped — research-patch + SSOT additive rows + Last-reviewed/Trigger sharpening on #55/#86. Not a capability commit per CLAUDE.md definition (no new explicit package.json dep; no ≥50 LOC under new packages/core/<dir>/; no ≥80 LOC under packages/). All edits are append-only or §3-step-1 in-place updates to prior-art-evaluations.md. Co-authored-by: Mastra Code (anthropic/claude-opus-4-7) <noreply@mastra.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Builds Option (ii) from the merged autonomous-self-audit R-phase (#95, §10.5/§11.1): the E-checklist Stop-gate interim. Extends the live
end-of-turn-reminder.shStop hook with a deterministic factual-claim scan.Why
The research (#95 §3.3, §4.2) showed the live Stop hook prompts a recap but does not force re-verification of the at-write-time factual class — and it skips short turns entirely, which is exactly where incident #1 ("4+ files") lived. This adds item-specific salience at turn-end for that class.
How
4 files,10 tests), file:line citations (foo.ts:42), negative-existence (no production tool exists).Recursive backstop (Q-D from #95)
tests/hooks/eot-claim-scan.test.sh— 6 sub-tests (3 positive enumeration, 1 silence, 1 valid-JSON, 1 mutation that breaks the numeric regex and asserts enumeration stops). Registered inaudit-self.yml. The drift-catcher hook is now itself caught by a test if it silently breaks.Side fix
Portable last-assistant-line extraction (
grep | tailinstead of BSD-onlytail -r) so the companion test runs on Linux/CI. Equivalent on macOS where the hook actually fires.Notes for review (live hook — merge is yours)
.claude/hooks/+tests/, not underpackages/);Prior-art:trailer included anyway (SSOT Wave 0.5: revise §13.21 L3 closure plan + self-review #20 ADOPT + Agent Verifier REFERENCE/ADAPT)..claude/hooks/,tests/,.github/).hook-stub-completeness✅,audit-ai-docs9/0 ✅,eot-claim-scan6/0 ✅.§1.7 Forward-check applied
@cc-only-rationale(internal CC dev-tooling);spec:pointer added to the research patch.§1.7 Backward-check applied
.claude/hooks/*.sh: onlyend-of-turn-reminder.shis a Stop-event recap hook;ask-question-reminder.sh(feat(hooks): PreToolUse:AskUserQuestion fork-challenge reminder #94) is PreToolUse (different event), no overlap. No other hook needs the claim-scan.