docs(research-patches): instruction-compliance vs structural-forcing empirical research - #97
Merged
Merged
Conversation
…empirical research Empirical-validation arm for Q1 of the autonomous-self-audit line (#95 §10 Q-C, round-3 §3.5): does the live claim-scan (#96) change behaviour, or is compliance voluntary-and-equal? Ran a deterministic no-LLM scorer over 337 real eval-unaware transcripts (266 sessions / 1537 claim-turns) + a controlled planted-error probe on 5 naive sub-agents. H0 NOT rejected (treatment arm N~0, eval-aware, cross-model). Substantive re-frame of Q1: baseline groundedness already ~0.74-0.81 (small headroom); the binding weakness is the DETECTOR — recall ~0.43 on natural numeric phrasings (misses "6 discipline rules", "266 sessions"; missed the one organic failure case), precision ~0.20-0.25 (cry-wolf). Compliance-when-delivered CAN be positive (probe1 corrected 6->8). Pre-registered decision-gate (a/b/c/d) + 4 maintainer decision-needed items. Sibling scorer committed as a reproducibility artefact (NOT CI-wired; promotion to tests/ is Q-E2-gated). principle-10 green; self-application (§7) caught this patch's own headline numbers in the recall blind spot. No mechanism implemented; no rule codified; no hook/settings edit; no PR on a mechanism. The 5 sub-agents were measurement subjects, not implementers. Prior-art: skipped — research patch only, no new capability; eval methodology + pilot, no mechanism adopted
artyhoo
added a commit
that referenced
this pull request
May 21, 2026
…s eval (Q-E1…Q-E4) (#98) * fix(hooks): claim-scan detector recall+precision (Q-E4) The instruction-compliance-empirical R-phase (#97) measured the live claim-scan detector: recall ~0.43 on natural numeric phrasings (the adjacency-only regex missed "6 discipline rules", "11 distinct principles" — and the one organic failure case in the pilot), precision ~0.20-0.25 (over-fired on numbers/paths inside fenced code, blockquotes, markdown link targets — cry-wolf). Maintainer decision 2026-05-21 (Q-E1+Q-E4): fix the detector. - recall: numeric regex now allows <=2 intervening tokens between number and count-noun. - precision: scan runs over a cleaned copy (fenced code + blockquotes + markdown link targets stripped); inline `code` kept so genuine file:line citations fire. Validated on 340 real transcripts (tests/eval baseline): recall +~37% claim-turns detected, precision -~105 false-positive claim-turns. eot-claim-scan.test.sh extended to 9 sub-tests (recall intervening-token + 2 precision cases + updated mutation). Noun-list extension (sessions/agents/...) deliberately out of scope. Prior-art: skipped — improves existing claim-scan detector (recall+precision), no new capability or dependency * feat(eval): session-bound claim-groundedness scorer (Q-E2) Promotes the instruction-compliance-empirical R-phase scorer (#97) to a committed session-bound tool (maintainer decision Q-E2, 2026-05-21: fund the longitudinal eval). Deterministic, no-LLM: reads the operator's own CC transcripts and measures the ungrounded-claim rate (claim present, no verification tool action in-cycle). NOT a CI gate — needs the operator's private transcripts + no-paid-llm-in-ci; it is operator-run (like template-audit / audit-self session-bound tooling). No workflow globs tests/eval/. Regexes + cleaning kept identical to the live hook so the eval measures the same surface. Includes README (doc-authority header + the §13.34 >=+15pp gate) and a committed baseline snapshot (2026-05-21) the gate compares future windows against. SSOT #53 records the build-vs-reuse verdict (REFERENCE promptfoo/Inspect/METR + BUILD thin CC-jsonl scorer; T16 problem-class check in rationale). Prior-art: prior-art-evaluations.md#53 (eval harness — BUILD; promptfoo/Inspect host the grading half via echo-provider/re-scoring but neither hosts the no-paid-LLM generation half; thin CC-jsonl scorer built, upstreams REFERENCE'd). Under tests/ not packages/ — not a capability-gate commit. * docs: wire §13.34 ≥+15pp promotion gate + record Q-E decisions (Q-E1/Q-E3) open-questions.md §13.34: the claim-scan interim graduates from interim -> kept-mechanism only on an empirical ≥+15pp grounded-lift over the committed tests/eval baseline (>=50 fired instances), else stays interim / candidate for removal or the H10-pair target. Clean A/B DEFERRED to longitudinal accrual (Q-E1) — ~no claim-scan-active data exists yet. research-patches/2026-05-21-instruction-compliance-empirical.md: appended §8.5 recording the maintainer's explicit answers to Q-E1..Q-E4 (reviewer-discipline §2 step 3 closure; append to the patch that surfaced the options, mirroring #95 §10.5). Implementation landed in this same PR's other commits. docs(research-patches) is §1.7-allowlisted; open-questions edit extends an existing armed entry (no new rule introduced).
artyhoo
added a commit
that referenced
this pull request
May 21, 2026
Lands the maintainer's settings.json safety-hardening that was stranded uncommitted on the superseded chore/ssot-karpathy-skills-ref branch (the only non-superseded residue there — every other file already on main via #89/#92/#93/ #94/#96/#97/#98). Committed verbatim. On-thesis: this is edit-time (earliest-channel) enforcement of project safety rules that previously lived only as memory/prose — "no git reset hard ever", no force-push, no skip-verify. The harness now blocks them before they run: - destructive git: force/mirror push, push to main/master, skip-verify commit, hard reset, checkout discard, branch force-delete, filter-branch, update-ref - destructive fs: recursive delete, shred, dd, mkfs, sudo - pipe-to-shell: curl-to-shell, wget-to-shell; raw netcat/socat - secret exfil: gh secret set/delete, gh auth logout, reads of credential dirs (ssh / aws / gnupg / gh config) and dotenv files - self-protection: .git/hooks, .husky, settings.json (Edit/Write deny) Also: portable $CLAUDE_PROJECT_DIR hook paths + ask-question-reminder.sh permission. Prior-art: skipped — harness config hardening (permission deny-list), no new capability code or dependency
5 tasks
artyhoo
added a commit
that referenced
this pull request
May 21, 2026
…#106) Append + in-place refinement of the living history book (v1 technical +86 lines, v2 literary +78). Narrates post-#93 events: the end-of-turn hook saga (#81 broken via systemMessage → #89 reason-channel fix), #94 AskUserQuestion hook, principles 11/12/13 landing, the #96/#97 claim-scan experiment (H0 not rejected), #90 recommendation-gate, and the niche-among-companions chapter. Colophon extended (chapters III–XII, patches to 2026-05-21, PR #85–#99). Preserves continuation work that was stranded uncommitted on the superseded chore/ssot-karpathy branch. Prior-art: skipped — narrative doc continuation, no new capability or dependency.
4 tasks
artyhoo
added a commit
that referenced
this pull request
May 21, 2026
…ap (D6 → ship H1) (#117) Implements the recommendation-gate D6 decision (maintainer, 2026-05-21): ship H1 (turn-start reminder), reject H2 (67% FP rate = cry-wolf, erodes compliance per the #97 finding), defer H10 (verdict-as-tool-call) to a separate research R-phase. H1 = one-line addition to the existing UserPromptSubmit hook (reuse, not a new hook): the session-bootstrap digest now injects a recommendation-discipline checklist before each turn. It is a context reminder, NOT structural enforcement — honest about being instruction-compliance-dependent (0% FP, 0 build cost, no theatre claim). Evidence base: recommendation-gate-iterative round-3 dry-run + round-5 §5.1 (branch research/recommendation-gate-iterative). Prior-art: skipped — reuses existing hook (SSOT #20, Claude Code hooks API, ADOPT); no new capability or dependency, single-line context-injection addition.
artyhoo
added a commit
that referenced
this pull request
May 21, 2026
…13.36 to closed Records the maintainer D6 decision (2026-05-21) as §13.39: H1 turn-start reminder shipped (#117), H2 keyword-scanner rejected (67% FP cry-wolf per #97), H10 verdict-as-tool-call deferred to a research R-phase (kickoff scaffolded; ARMED). Cross-refs §13.34 (kept separate per round-5 §5.3). Adding §13.39 pushed open-questions.md over the 500-line pre-commit budget, so the already-RESOLVED §13.36 (TDD-for-Skills → shipped as principle 15 this session) is migrated to closed-questions.md per the append-only archive convention, anchor preserved. Net: open-questions back under budget (496 lines). Prior-art: skipped — tracker entries recording existing decisions + archive migration, no new capability or dependency.
5 tasks
artyhoo
added a commit
that referenced
this pull request
May 21, 2026
…13.36 to closed (#118) Records the maintainer D6 decision (2026-05-21) as §13.39: H1 turn-start reminder shipped (#117), H2 keyword-scanner rejected (67% FP cry-wolf per #97), H10 verdict-as-tool-call deferred to a research R-phase (kickoff scaffolded; ARMED). Cross-refs §13.34 (kept separate per round-5 §5.3). Adding §13.39 pushed open-questions.md over the 500-line pre-commit budget, so the already-RESOLVED §13.36 (TDD-for-Skills → shipped as principle 15 this session) is migrated to closed-questions.md per the append-only archive convention, anchor preserved. Net: open-questions back under budget (496 lines). Prior-art: skipped — tracker entries recording existing decisions + archive migration, no new capability or dependency.
5 tasks
artyhoo
added a commit
that referenced
this pull request
May 22, 2026
…cation (#161) Codifies 5 stage-0 memory conventions (DN-4 gaps #20/#23/#25/#26/#27): - phase-research-coverage.md §1.12 — lead with reasoned recommendation (#27); act when best path clear, reserve questions for genuine forks (#20) - ai-laziness-traps.md T17/T18/T19 — preserve-before-destructive (#25), preserve-unique-residue (#26), own-cold-QA-before-handoff (#23) - tracker rows → CODIFIED (#27 prose-only; mechanization deferred per #97/#98) Channel-selection: judgment rules → reuse 2 already-always-on rules, not new files (#always-on-bloat). §1.7 self-review at research-patches/2026-05-22-dn4-round2-codification.md. Prior-art: skipped — prose discipline codification of existing memory conventions, no new capability or dependency
artyhoo
added a commit
that referenced
this pull request
May 31, 2026
… DN #11=A) (#322) Registers the convergence-aware auto-review loop (closure_first + AGENT_MAX_REVIEW_ITERATIONS) already used verbatim in generated kickoff §4c. Resolves DECISION-NEEDED #11 from PR #321 (maintainer chose Option A). Append-only; no existing row edited. Prior-art: skipped — append-only SSOT row registering an already-adopted vocabulary; no new capability, no dependency, no code change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Empirical-validation arm for Q1 of the autonomous-self-audit line (#95 §10 Q-C, round-3 §3.5): does the live claim-scan (#96) actually change AI behaviour, or is compliance voluntary-and-equal with/without it (theatre)?
Single deliverable: research-patch
docs/meta-factory/research-patches/2026-05-21-instruction-compliance-empirical.md(197 lines, principle-10 green) + a sibling reproducibility scorer (*.scorer.py, NOT CI-wired).Method (no-paid-LLM, deterministic)
Headline findings
Decision-needed (maintainer /
/orchestrator— reviewer does not pick)Pre-registered gate (a keep / b remove / c fast-track H10 / d defer) + Q-E1…Q-E4 in §8/§9. Recommendation in the patch: DEFER the clean Q1b A/B to longitudinal accrual + fix the detector (recall/precision) as the cheap high-value move; bare-removal defensible on headroom, fast-track-H10 premature (H0 un-tested).
Scope
No mechanism implemented; no rule codified; no hook/
settings.json/CI edit; no PR on a mechanism. Connects to open-questions §13.34 (Q-E3: should its promotion trigger depend on this result?).§1.7: N/A —
docs(research-patches)is allowlisted in.husky/pre-push.