docs(open-questions): add §13.34 autonomous self-audit triggering trigger - #53
Merged
Hidden character warning
The head ref may contain hidden characters: "add-\u00a713.34-autonomous-self-audit"
Conversation
…gger Codifies post-Wave-10 research trigger for autonomous self-audit mechanism research. Surfaced during PR #52 dialogue (memory-to-docs codification) — maintainer question «разве это не я ловлю а каждый слой сам?» exposed structural gap: AI agent does not autonomously re-verify own claims; catches happen only via external triggers (user prompts, reviewer sessions, CI hooks, principle tests). Empirical evidence: PR #51 + PR #52 accumulated 9+ file:line/count slips caught across 3 review rounds. Cost: 3 review rounds for 10-commit PR. Moves the «not forgotten» tracking from gitignored .claude/orchestrator-prompts/autonomous-self-audit-research/research-prompt.md into git-tracked open-questions.md §13.x — periodic trigger-sweep (phase-research-coverage.md §1.6) ensures structural follow-up after Wave 10 closure. Prior-art: skipped — entry addition to open-questions register, no new capability or rule semantics; trigger condition for post-Wave-10 research session (mechanism design out of scope for this commit).
Initial §13.34 entry quoted the project goal verbatim («AI agents can't silently bypass undocumented conventions»). audit-ai-docs.sh probe D5 greps the repo for that canonical phrase and requires each occurrence to be enrolled in DOWNSTREAM_DOCS or test-infrastructure exempt list. open-questions.md is in neither — D5 raised orphan violation, failing pre-push hook. Paraphrased the citation to reference README.md by link instead of verbatim quote. Audit now 9/9 green on this branch. Recursive-irony note: PR adding §13.34 trigger about «AI doesn't self-verify own claims» got caught by D5 verifying canonical-phrase enrollment. Discipline chain caught the slip before push — exact behavior §13.34 research will study. Prior-art: skipped — fix commit on previous commit's bug; no new capability, no rule semantics.
artyhoo
added a commit
that referenced
this pull request
May 21, 2026
…s eval (Q-E1…Q-E4) (#98) * fix(hooks): claim-scan detector recall+precision (Q-E4) The instruction-compliance-empirical R-phase (#97) measured the live claim-scan detector: recall ~0.43 on natural numeric phrasings (the adjacency-only regex missed "6 discipline rules", "11 distinct principles" — and the one organic failure case in the pilot), precision ~0.20-0.25 (over-fired on numbers/paths inside fenced code, blockquotes, markdown link targets — cry-wolf). Maintainer decision 2026-05-21 (Q-E1+Q-E4): fix the detector. - recall: numeric regex now allows <=2 intervening tokens between number and count-noun. - precision: scan runs over a cleaned copy (fenced code + blockquotes + markdown link targets stripped); inline `code` kept so genuine file:line citations fire. Validated on 340 real transcripts (tests/eval baseline): recall +~37% claim-turns detected, precision -~105 false-positive claim-turns. eot-claim-scan.test.sh extended to 9 sub-tests (recall intervening-token + 2 precision cases + updated mutation). Noun-list extension (sessions/agents/...) deliberately out of scope. Prior-art: skipped — improves existing claim-scan detector (recall+precision), no new capability or dependency * feat(eval): session-bound claim-groundedness scorer (Q-E2) Promotes the instruction-compliance-empirical R-phase scorer (#97) to a committed session-bound tool (maintainer decision Q-E2, 2026-05-21: fund the longitudinal eval). Deterministic, no-LLM: reads the operator's own CC transcripts and measures the ungrounded-claim rate (claim present, no verification tool action in-cycle). NOT a CI gate — needs the operator's private transcripts + no-paid-llm-in-ci; it is operator-run (like template-audit / audit-self session-bound tooling). No workflow globs tests/eval/. Regexes + cleaning kept identical to the live hook so the eval measures the same surface. Includes README (doc-authority header + the §13.34 >=+15pp gate) and a committed baseline snapshot (2026-05-21) the gate compares future windows against. SSOT #53 records the build-vs-reuse verdict (REFERENCE promptfoo/Inspect/METR + BUILD thin CC-jsonl scorer; T16 problem-class check in rationale). Prior-art: prior-art-evaluations.md#53 (eval harness — BUILD; promptfoo/Inspect host the grading half via echo-provider/re-scoring but neither hosts the no-paid-LLM generation half; thin CC-jsonl scorer built, upstreams REFERENCE'd). Under tests/ not packages/ — not a capability-gate commit. * docs: wire §13.34 ≥+15pp promotion gate + record Q-E decisions (Q-E1/Q-E3) open-questions.md §13.34: the claim-scan interim graduates from interim -> kept-mechanism only on an empirical ≥+15pp grounded-lift over the committed tests/eval baseline (>=50 fired instances), else stays interim / candidate for removal or the H10-pair target. Clean A/B DEFERRED to longitudinal accrual (Q-E1) — ~no claim-scan-active data exists yet. research-patches/2026-05-21-instruction-compliance-empirical.md: appended §8.5 recording the maintainer's explicit answers to Q-E1..Q-E4 (reviewer-discipline §2 step 3 closure; append to the patch that surfaced the options, mirroring #95 §10.5). Implementation landed in this same PR's other commits. docs(research-patches) is §1.7-allowlisted; open-questions edit extends an existing armed entry (no new rule introduced).
artyhoo
added a commit
that referenced
this pull request
Jun 16, 2026
…551 regression-prevention) Phase 1 of shipped-artifact-liveness-gap (research PR #566, Option C phased, scope (i)). The 6 agents themselves were already fixed on staging by #571; this adds the DETERMINISTIC GATE #571 did not — preventing any future non-canonical tool name from shipping (the form-check that would have caught #551 at pre-push/CI). - packages/core/principles/21-shipped-agent-tools-valid.test.ts (9 tests, paired-negative arm) - prior-art-evaluations.md #121 (BUILD; own liveness family with #114/#115) Built autonomously via aif-handoff (task 8cae6079); agent edits dropped as redundant with #571. Prior-art: prior-art-evaluations.md#121 (BUILD - shipped-agent tools:-name allow-list gate; no upstream validator; own liveness family with #114/#115; T16 form-check distinct from #53/#115 behavioural eval).
artyhoo
added a commit
that referenced
this pull request
Jun 16, 2026
…551 regression-prevention) Phase 1 of shipped-artifact-liveness-gap (research PR #566, Option C phased, scope (i)). The 6 agents were already fixed on staging by #571; this adds the DETERMINISTIC GATE #571 did not — preventing any future non-canonical tool name from shipping. - packages/core/principles/21-shipped-agent-tools-valid.test.ts (9 tests, paired-negative) - prior-art-evaluations.md #121 (BUILD; own liveness family with #114/#115) Built autonomously via aif-handoff (task 8cae6079); agent edits dropped as redundant with #571. §1.7: forward-check applied — no-paid-llm-in-ci compliant (deterministic vitest, zero API call), build-first-reuse BUILD #121 (no upstream validator), T16 form-check not behaviour-check; backward-check sweep — #571 already made the 6 agents canonical so this adds only the missing regression gate and supersedes nothing. Evidence: packages/core/principles/21-shipped-agent-tools-valid.test.ts:46 Prior-art: prior-art-evaluations.md#121 (BUILD - shipped-agent tools:-name allow-list gate; no upstream validator; own liveness family with #114/#115; T16 form-check distinct from #53/#115 behavioural eval).
artyhoo
added a commit
that referenced
this pull request
Jun 16, 2026
…551 regression-prevention) (#576) Phase 1 of shipped-artifact-liveness-gap (research PR #566, Option C phased, scope (i)). The 6 agents were already fixed on staging by #571; this adds the DETERMINISTIC GATE #571 did not — preventing any future non-canonical tool name from shipping. - packages/core/principles/21-shipped-agent-tools-valid.test.ts (9 tests, paired-negative) - prior-art-evaluations.md #121 (BUILD; own liveness family with #114/#115) Built autonomously via aif-handoff (task 8cae6079); agent edits dropped as redundant with #571. §1.7: forward-check applied — no-paid-llm-in-ci compliant (deterministic vitest, zero API call), build-first-reuse BUILD #121 (no upstream validator), T16 form-check not behaviour-check; backward-check sweep — #571 already made the 6 agents canonical so this adds only the missing regression gate and supersedes nothing. Evidence: packages/core/principles/21-shipped-agent-tools-valid.test.ts:46 Prior-art: prior-art-evaluations.md#121 (BUILD - shipped-agent tools:-name allow-list gate; no upstream validator; own liveness family with #114/#115; T16 form-check distinct from #53/#115 behavioural eval).
artyhoo
added a commit
that referenced
this pull request
Aug 16, 2026
… confounding measured) (#1392) Authors the S4 dispatch input fresh (kickoff-s1/s2/s3 §8/§5 chain) and flips the router's S4 row to READY. Substantive content beyond restating design §4/§5: - C0 re-measured on the adjudicated corpus: 0.733 accuracy, MATERIAL-miss 0.319, scored subset 131/151 (86.8%, above D-K8's ~70% floor). This falsifies the spec's own r2 NEW-M4 pre-read («roughly 0.8-0.9») on that pre-read's own stated falsifier; §3.3 carries the number, the command, and a re-derive-or-PARK rule. - §3.6 names a confounding design §5b.1 covers only generically: C1 shares rubric and model family with the S2 cold rater whose labels survive on 122/151 rows. Mandates a secondary breakdown on the 29-row (25 C0-scored) advisor-override slice, with its n≈25 power caveat as part of the requirement, not a footnote. - §7 deliberately does NOT copy the S2 contract shape — S2 closed with `#contract-that-cannot-fail` open against its own lines 2 and 4. Line 1 is arms A-F over S4's own deliverables; the S0 frozen-record probe is absorbed into arm F (blob-hash immutability of every population/S0/S2/S3 master), which S4 can break. - Arm E closes W-7 for S4's own report: report numbers are recomputed from the artifacts and must match the prose. - Destination probes re-run 2026-08-16: `gh` IS authenticated in the container now, so S2's `cannot-reach-gh` park is NOT a standing fact; promptfoo 0.122.0 reachable; aif base clone STALE at 3a10c71. Table carries probe + date per destination-environment-verification §1b. - Domain traps T-TK4-A..D (truth edited to fit the bench, confounded agreement as accuracy, bar rebuilt after seeing the candidate, promptfoo adopted by name — the last aimed at SSOT #53's existing BUILD verdict for a neighbouring problem class). Prior-art: skipped — dispatch-input authoring only; no dependency, module, or capability is added by this commit. The stage's own capability commit (promptfoo devDependency + Prior-art: trailer + SSOT #250) is specified in kickoff-s4.md §3.1 and lands with S4's implementation, not here. Co-authored-by: Test <test@example.com>
Merged
artyhoo
added a commit
that referenced
this pull request
Aug 16, 2026
* feat(triage-kernel-v2): S4 capability — promptfoo devDep + SSOT row #250 Stage S4 bench runner (kickoff-s4 §3.1, design D-K3 ratified ADOPT). promptfoo@0.122.0 pinned as devDependency — matrix/caching runner over an exec:-shaped provider shelling `claude -p --model sonnet` for C1 (151 per-row) and C2 (41 grouped) vs s3-final.csv. devDependency only; never enters CI (no-paid-llm-in-ci.md, design §6). SSOT #250 carries the T16 problem-class check vs #53: same tool name, different problem class (generation-then-grade here vs grade-only-pre-existing-transcripts there), so the verdicts differ legitimately. Prior-art: prior-art-evaluations.md#250 (ADOPT — S4 bench runner; T16 class check vs #53 in rationale.) Prior-art: prior-art-evaluations.md#53 (BUILD — different problem class: transcript scoring without generation; verdict stands.) * triage-kernel-v2 S4: bench mechanics — promptfoo exec runner, sha-gated bench input, scorer + 8 fail-closed arms Runner: promptfoo@0.122.0 over an exec provider shelling `claude -p --model sonnet` (model pinned by name inside provider.mjs — a swap is a PARK, kickoff-s4 §3.4). Bench input built from the frozen substrate via the frozen buildPayload (judges see rubric + context + finding only), stamped with its SHA-256 and re-derived by arm H. Scorer: per-axis stats (copied frozen choose/mcnemar/kappa), two-leg class gate, confounding slice, C2 delta/cost split, --join for s4-bench.csv, and arms A-H (blindness differential + token rescan, enums, subset honesty, report-number reconciliation, substrate blob-hash immutability at 7425346, SSOT #250 wiring, judge provenance). One recorded parse-refinement incident in run.mjs parseC2Group doc comment (frozen search-parse semantics; positional keys still reject). * triage-kernel-v2 S4: results — C1/C2 judge artifacts, s4-bench.csv join, bench report + corpus README §S4 Verdict lines (scorer output, arms A-H green): class n=131 — no layer beats C0 (C1 0.687 p=0.4514 miss 0.351; C2 0.710 p=0.7608 miss 0.266; both DOES-NOT-SHIP, C0 0.733 stays the bar). Layer n=151 — both candidates beat the 0.530 majority bar (C1 0.662 p=0.0012; C2 0.642 p=0.0076). Whose n=151 — judgment-only, not corpus-validated (C1 0.848, C2 0.854 below the degenerate 0.901 bar). Confounding slice measured (n=25, insufficient power, reported never gated). C2 = delta over C1 (+0.023, inside ±9pp MDD) at 41 vs 151 calls. Spec's C0 pre-read falsification and the C2 parse-refinement incident recorded in the report. * fix(triage-kernel-v2): S4 round-1 fidelity rework — arm E reconciles report PROSE, directional leg 1 Cold fidelity audit round 1 (FIDELITY: REVISE vs 1bbff50) — five items, no judge re-run, no answer-key touch, no artifact regeneration: 1. MAJOR arm E: NUM gains the report-stated keys computed from the existing artifacts (class PABAK per candidate, layer multiclass kappa per candidate, discordant b/c pairs for every McNemar comparison), and arm E now reconciles the report PROSE in addition to the s4-numbers block (block stays the canonical carrier). Boundary- anchored match (?<![\d.])v(?!\d|\.\d) so '25' cannot match inside '0.325'/'25.5' while a sentence-ending period still matches. Falsified live: a one-digit prose edit REDs arm E; restored, green. 2. MINOR arm A(a): per-axis denominators captured inside the per-axis loops from the iterated arrays and asserted against the §3.6 populations (class=131 on the scored subset, layer=whose=151 on labelable) — a loop switched to the other population is RED. 3. MINOR leg 1 made DIRECTIONAL (PASS requires p<0.05 AND c>b), same rule applied to the layer beat phrase; method note added to the report. No verdict changed: both class candidates failed leg 1 on p alone. 4. DOC the C2 incident passage now states the actual finding — kickoff §3.5 specified no C2 output contract; the frozen S2/C1 search semantics (triage-s0-run.mjs:59) were adopted and are ratified; PARK-shaped note recorded for S5. No behaviour change. 5. NOTE regex-divergence recorded (s0-run [a-z]+ vs bench TRIPLE_RE [a-z-]+ layer pattern); neither regex changed. §7 gates green: s4 --check 8/8 arms, s3-agreement --check, 9-arg s2-labels-check, markdownlint clean (226 lines). Prior-art: skipped — gate-strength fix + documentation on existing S4 surfaces, no new capability. * fix(triage-kernel-v2): add <!-- scope: --> annotation to the S4 bench research patch Principle 10 (packages/core/principles/10-research-patch-annotation.test.ts) requires every file in docs/meta-factory/research-patches/ to carry a machine-parseable <!-- scope:<slug> --> comment on its FIRST line; the S4 bench patch shipped with the H1 heading there, so vitest-principles went RED in the full local CI sweep (branch-introduced — the file is new in this branch). Slug triage-kernel-v2-s4-bench, matching the sibling 2026-08-11-triage-kernel-v2-s0-probe.md shape. Content-neutral: two lines prepended, no prose touched, so arm E of triage-s4-score.mjs is unaffected. Prior-art: skipped — one-line annotation fix for an existing principle gate, no new capability. --------- Co-authored-by: Test <test@example.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds §13.34 — Autonomous self-audit triggering layer (post-Wave-10 research) as armed trigger in
docs/meta-factory/open-questions.md. Codifies the structural gap surfaced during PR #52 dialogue: AI agent does not autonomously re-verify its own claims at write time; all catches in PR #51 + #52 review chain required external triggers (user prompts, reviewer sessions, CI hooks, principle 10 hook). Empirical cost: 3 review rounds for 10-commit PR #52, 9+ file:line/count/annotation slips caught.Moves the «won't be forgotten» tracking from gitignored
.claude/orchestrator-prompts/autonomous-self-audit-research/research-prompt.mdinto git-trackedopen-questions.md §13.x— periodic trigger-sweep perphase-research-coverage.md §1.6ensures structural follow-up after Wave 10 closure.Changes
no-paid-llm-in-ci.md), trigger condition «Wave 10 closure», first-action pointer to research prompt, scope warning re: friction budget + turtles-all-the-way-down, recursive self-application gate, cross-refs.audit-ai-docs.sh:154CANON_PHRASE) raised orphan violation becauseopen-questions.mdis not in DOWNSTREAM_DOCS enrolled list. Recursive irony: PR adding the trigger about «AI doesn't self-verify own claims» got caught by an audit verifying canonical-phrase enrollment — discipline chain working as designed.Prior-art consult
Prior-art: skippedtrailer on both commits (rationale ≥20 chars)Test plan
bash packages/core/audit-self/audit-ai-docs.test.sh— 9 pass / 0 fail after fix commitbash .husky/pre-push— clean on branch HEADopen-questions.mdstill parses — 23 §13.x entries detected via grep§1.7 Skipped: open-questions.md is not in discipline-self-check.yml path filter and not in s17_is_discipline_introducing file glob; entry addition does not introduce or extend rule semantics, only registers an armed trigger condition for post-Wave-10 future work. No rule formation here.