Docs: bring the audit tutorial and CATALOG up to what audit 0.2.0 actually does - #35
Merged
Conversation
…ually does The tutorial is the procedure for run 2, and run 2 is the run that tests resume — so it was the one document that could not be left describing the pre-#34 behaviour. Step 4 taught the single-block resume rule that #34 replaced: read findings.md, resume at the lowest number with no entry. It now describes partitioning issues.json by state and resuming each side independently, and names the new failure mode — resuming the open set correctly while re-doing the closed set from scratch. The same step quoted the three-form citation list that doctrine §2 has outgrown, without the reachability rule that is the whole point of defect 1. Two review checks are strengthened where run 1 showed them to be weak. Check 3 passed a citation that pointed at an unmerged branch, so it now says to resolve each sampled citation on the ref the header names. Check 5 held in the report and broke in the checkpoint, so it now looks at findings.md as well as the output. CATALOG records the two severity-1 fixes. Its central caveat is unchanged and still true: resumability is untested, because run 1 was never interrupted. Repo-level files only — no plugin directory touched, so no version bump. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 7, 2026
There was a problem hiding this comment.
Pull request overview
Updates repository-level documentation to reflect the behavior and expectations of audit as of 0.2.0 (post-#34), especially around phase-2 resumability/checkpointing and the “citations must resolve on the named ref” rule.
Changes:
- Update the audit tutorial’s Step 4 to describe per-state resuming (
## Open/## Closed) and the ref-resolvable citation requirement. - Strengthen two post-run review checks in the tutorial (resolve sampled citations on the named ref; verify closed-side checkpointing via
findings.md). - Update
CATALOG.mdto note the two severity-1 fixes shipped in audit 0.2.0.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| docs/tutorial-run-an-audit.md | Updates Step 4 and review checks to match audit 0.2.0’s resume/checkpoint and citation-resolution behavior. |
| CATALOG.md | Notes that the two severity-1 defects addressed in #34 are fixed as of audit 0.2.0. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…f the citation list Copilot's three comments on #35, all on text this PR introduced. Step 4 said the closed pass "has never been checkpointed at all" two sentences before crediting #34 with checkpointing it. True of run 1, false of main. The point is that it has never been *exercised*, which is what it now says. The doctrine §2 link was hard-coded to blob/main while line 7 of the same file links doctrine relatively, and developing-skills says to keep relative links relative because CI checks they resolve. The AGENTS.md absolute-URL rule governs links out of a plugin directory, since an installed plugin ships alone and cannot resolve `../`; docs/ never ships alone, so relative is right here. Review check 3 enumerated citation forms and omitted tags. Rather than sync what had become the fourth copy of that list, the row no longer enumerates at all — the same fix applied to doctrine §1 on #34, and what line 7 means by "points, never restates". The list at line 79 is kept deliberately: it is the teaching sentence and it links to §2 in the same breath. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-through on #34. The tutorial is the procedure for run 2, and run 2 is the run that tests resume — so it is the one document that could not be left describing the pre-#34 behaviour.
Step 4 taught the resume rule that #34 replaced. It said to read
findings.mdand resume at the lowest number with no entry, which is exactly the single-block assumption defect 2 was about. It now describes partitioningissues.jsonby state and resuming each side independently, and it names the new failure mode worth watching for: resuming the open set correctly while re-doing the closed set from scratch. It also suggests interrupting during the closed pass if the chance comes, since that is the half that has never been checkpointed at all.The same step quoted doctrine's old three-form citation list —
file:line, a merged PR, or a tag — with no reachability rule, which is the drift Copilot caught inside doctrine on #34 surviving one file over. It now names the fourth form and, more importantly, the requirement that whatever is cited resolves on the ref the audit named.Two review checks are strengthened where run 1 showed them weak. Check 3 passed a citation pointing at an unmerged branch, so it now says to resolve each sampled citation on the ref the header names, with
git merge-base --is-ancestorfor commits. Check 5 held in the report and broke in the checkpoint, so it now looks atfindings.mdas well as the output — closed entries reaching the catalog with no## Closedblock is precisely how run 1 passed that check in its report while failing it in its log.CATALOG records the two severity-1 fixes. Its central caveat is unchanged and still true: resumability is untested, because run 1 was never interrupted.
reviews/audit-run-action-translation-2026-07-28.mdis deliberately untouched — it is a dated record of what the doctrine said at the time, and correcting it would destroy the evidence that the defects were real.Repo-level files only, so no plugin directory is touched and nothing bumps;
check-version-bump.pyagrees.🤖 Generated with Claude Code