Skip to content

Recall-failure fixes: live-validate gate in /pr-create + fixture-realism in hook-authoring - #309

Merged
evansenter merged 3 commits into
mainfrom
improve/live-validate-and-fixture-realism
May 21, 2026
Merged

Recall-failure fixes: live-validate gate in /pr-create + fixture-realism in hook-authoring#309
evansenter merged 3 commits into
mainfrom
improve/live-validate-and-fixture-realism

Conversation

@evansenter

Copy link
Copy Markdown
Owner

Summary

Two surgical edits surfaced by improve-workflow reflection on the recent three-PR Stop-hook enforcement arc (#303#307#308):

  1. home/.claude/commands/pr-create.md — new step 4 (path-gated live-validation prompt). When git diff --name-only main...HEAD matches home/.claude/hooks/, plugins/, or settings.json, surface a live-validate question before opening the PR. Five lines of prose.
  2. home/.claude/skills/hook-authoring/SKILL.md — fixture-realism nudge in the Testing section. Sample one real ~/.claude/projects/*/<id>.jsonl message before writing regex fixtures, instead of hand-writing the expected format. Three lines of prose.

Why these specifically

The three-PR arc had a recurring failure mode: memory entries existed for "live-validate runtime PRs before merge" but didn't fire when I needed them, because at PR-creation time I was thinking "the regex fix is done" rather than "is the live-validation gap applicable here." Same pattern across rounds 1 and 2:

The diagnosis: memory alone is recall-dependent; the rule needs to fire deterministically when the path-match condition holds. Hence the harness-level check in /pr-create rather than more prose in CLAUDE.md.

Cost discipline

Both edits respect the cost-per-invocation rule shipped in #306:

  • pr-create.md gets +5 lines of prose, paid every /pr-create invocation. The check is path-gated so the AskUserQuestion only fires when relevant — most invocations just read the condition and move on.
  • hook-authoring SKILL.md auto-loads only when the model detects hook-authoring context. +3 lines of prose, bounded cost.

Avoided: the broader /work enforcement-gap classifier (event #4029) which had similar motivation but a fuzzier trigger and would have taxed every /work invocation. Kept that one in memory + event bus.

Companion memory

feedback_behavioral_guard_acceptance.md (in personal memory, not in this repo): for hook/lint/policy PRs, acceptance = deliberate violation in a fresh session. Replay against a transcript without a fresh violation proves nothing. Referenced from the /pr-create step.

Test plan

  • make check passes (29/29; doc-only changes, no shell or test impact)
  • shellcheck would be a no-op (no shell code changed)
  • Behavioral validation: next time I make a hook/settings/plugin PR, the live-validate prompt should surface naturally. Confidence is high because the check is git diff --name-only + AskUserQuestion — deterministic.

🤖 Generated with Claude Code

Two harness-level recall fixes for lessons memory alone failed to fire:

1. /pr-create: new step 4 gates on `git diff --name-only` matching
   home/.claude/{hooks,plugins,settings.json}. Surfaces a live-validation
   prompt before PR creation — for behavioral guards (hook/lint/policy
   that BLOCKS something), CI fixtures can't prove the guard is wired
   in and reachable. References the new memory.

2. hook-authoring SKILL.md: "Fixture realism" nudge in the Testing
   section. Sample real transcript content before writing regex; don't
   hand-write fixtures from a mental model.

Both surfaced by improve-workflow reflection on the three-PR Stop-hook
arc (#303#307#308) where each round shipped with insufficient
empirical validation. Skill prose stays tight (~3-5 lines each) to
respect the cost-per-invocation rule from PR #306.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
claude[bot]
claude Bot previously approved these changes May 18, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review by Claude Code (claude-review.md). Doc-only PR; no Critical or Important issues. Three Suggestion-level findings below — non-blocking. Note: harness blocked file writes needed for inline-comment JSON, so findings are consolidated here with file:line anchors.

Verdict: APPROVE

Suggestion 1 — home/.claude/skills/hook-authoring/SKILL.md:209 — Dangling reference. The new line says 'See hooks/README.md Gotchas for the empirical case study (regex shipped without backtick support...)' but home/.claude/hooks/README.md Gotchas (line 266+) actually covers jq multiline regex anchoring, no-jq test PATH setup, and Stop hook transcript-writer races — none of them is the backtick case study. The backtick-wrapped-decorator narrative actually lives inline in home/.claude/hooks/enforce-insight-publish.sh:60-68. Either point the reader there (See enforce-insight-publish.sh:60-68 for the empirical case study) or add a short Gotchas entry in hooks/README.md so the reference resolves.

Suggestion 2 — home/.claude/commands/pr-create.md:49 — The home/.claude/plugins/ arm of the path-gate is dead today: that directory doesn't exist in the dotfiles repo (plugins are referenced from settings.json enabledPlugins, not vendored as files). Either drop home/.claude/plugins/ from the list to keep the gate honest, or add a brief inline note that it's a forward-looking hook for if/when plugins get vendored locally.

Suggestion 3 — home/.claude/commands/pr-create.md:49 — The trigger is described in prose ('if git diff --name-only main...HEAD matches ...') rather than as a concrete pattern, which leaves the match semantics to the model and could drift between invocations. A one-liner like 'git diff --name-only main...HEAD | grep -qE ^home/[.]claude/(hooks/|settings[.]json$)' would make the trigger deterministic and align with the harness-level check framing called out in the PR description (and is cheap — grep -q exits early).

- pr-create.md: replace prose trigger with deterministic `grep -qE`
  one-liner, and drop the dead `home/.claude/plugins/` arm (plugins
  aren't vendored in this repo; they're referenced via enabledPlugins
  in settings.json).
- hook-authoring SKILL.md: fix dangling reference. The backtick case
  study lives inline in enforce-insight-publish.sh, not in hooks/README.md
  Gotchas (which covers jq multiline, PATH-stubbing, and the transcript
  race — not backticks). Point at the script with its inline comment.

Verified the new regex via dry-run: matches `home/.claude/hooks/foo.sh`
and `home/.claude/settings.json`, correctly does NOT match
`home/.claude/skills/` or `home/.claude/commands/`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@evansenter

Copy link
Copy Markdown
Owner Author

Feedback Addressed

Implemented

  • [Suggestion] Add dark mode theme switching for btop and iTerm2 #1 — Dangling reference in hook-authoring/SKILL.md fixed. Now points at enforce-insight-publish.sh (with the inline comment + regex), which is where the backtick case study actually lives. The hooks/README.md Gotchas section covers jq-multiline / PATH-stubbing / transcript-race, not backticks.
  • [Suggestion] Add Claude Code GitHub Workflow #2 — Dropped the dead home/.claude/plugins/ arm from pr-create.md. Plugins live in enabledPlugins in settings.json, not vendored as files.
  • [Suggestion] Refactor sync to use home/ directory #3 — Replaced the prose trigger ("if git diff matches...") with a deterministic grep -qE one-liner. Verified via dry-run: matches home/.claude/hooks/foo.sh + home/.claude/settings.json, correctly skips skills/ and commands/.

claude[bot]
claude Bot previously approved these changes May 21, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review. All three prior Suggestion-level findings (dangling reference, dead plugins arm, prose-only trigger) are correctly addressed. One new Suggestion: in home/.claude/skills/hook-authoring/SKILL.md:209, the new line contains 9 backticks (odd), so at least one inline code span is unmatched. The regex around match(...) renders awkwardly on GitHub because an inner backtick splits it into two spans plus a literal backtick. Easy fixes: wrap the whole regex in double-backticks so the inner backtick becomes a literal, or refer to the call by location (See the match call at enforce-insight-publish.sh:87 and the inline comment above it). APPROVE - one Suggestion, non-blocking. Automated review by Claude Code.

PR #309 round-2 review caught that my dangling-reference fix introduced
an odd backtick count (9) on hook-authoring/SKILL.md:209 — the inlined
regex contained a literal backtick that closed an inline code span
prematurely.

Drop the inlined regex from the prose entirely; the file pointer
itself is the durable reference, and the script's inline comment has
the regex + narrative anyway.

Backtick count is now even (6, three pairs).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@evansenter

Copy link
Copy Markdown
Owner Author

Feedback Addressed (round 2)

Implemented

  • [Suggestion] refactor: Make git pull opt-in and skip unchanged LaunchAgent reload #4 — Fixed odd-backtick markdown rendering issue. Dropped the inlined regex from the prose (it was the source of the unmatched backtick); the script pointer alone is durable and the inline comment in the script has the regex anyway. Line now has 6 backticks (3 clean pairs).

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review by Claude Code (claude-review.md). Doc-only PR, round 3 review. All four prior Suggestion-level findings (dangling reference, dead plugins arm, prose-only trigger, odd-backtick markdown) are correctly resolved. Verified: enforce-insight-publish.sh exists with the backtick-aware regex comment at lines 61-87 — the SKILL.md pointer resolves cleanly; the grep -qE pattern in pr-create.md correctly matches home/.claude/hooks/* and home/.claude/settings.json (exact), correctly skips settings.local.json, skills/, and commands/; no leftover plugins/ arm; SKILL.md backtick count is balanced. No Critical, Important, or Suggestion-level issues. Verdict: APPROVE.

@evansenter
evansenter merged commit 1cbef0c into main May 21, 2026
6 checks passed
@evansenter
evansenter deleted the improve/live-validate-and-fixture-realism branch May 21, 2026 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant