feat: shared test coverage audit across plan/ship/review (v0.9.5.0)#259
Open
feat: shared test coverage audit across plan/ship/review (v0.9.5.0)#259
Conversation
DRY extraction of the test coverage audit methodology into a shared generator function with three explicit placeholders: - TEST_COVERAGE_AUDIT_PLAN (plan-eng-review) - TEST_COVERAGE_AUDIT_SHIP (ship) - TEST_COVERAGE_AUDIT_REVIEW (review) Shared across all modes: codepath tracing, ASCII diagram format, quality scoring rubric, E2E test decision matrix, regression rule, and test framework detection via CLAUDE.md. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the thin 6-line Section 3 test review with the full shared
methodology via {{TEST_COVERAGE_AUDIT_PLAN}}. Plan mode now:
- Traces every codepath with full ASCII diagrams
- Adds missing tests to the plan (not just "check for tests")
- Writes test plan artifact for /qa consumption
- Includes E2E/eval recommendations and regression detection
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace 135 lines of inline Step 3.4 methodology with
{{TEST_COVERAGE_AUDIT_SHIP}}. Functionally identical output plus:
- E2E test decision matrix (marks paths needing E2E vs unit)
- Eval recommendations for LLM prompt changes
- Regression detection iron rule
- Test framework detection via CLAUDE.md first
- Test plan artifact for /qa consumption
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add codepath tracing to the pre-landing review via
{{TEST_COVERAGE_AUDIT_REVIEW}}. Review mode:
- Produces ASCII coverage diagram (same methodology as plan/ship)
- Generates tests for gaps via Fix-First (ASK user)
- Subsumes Pass 2 "Test Gaps" checklist category
- Gaps are INFORMATIONAL findings
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
10 new tests verifying the three TEST_COVERAGE_AUDIT placeholders: - All modes share: codepath tracing, E2E matrix, regression rule - Plan mode: adds to plan + artifact, no ship-specific content - Ship mode: auto-generates + before/after count + coverage summary - Review mode: Fix-First ASK + INFORMATIONAL, no artifact - Regression guard: ship SKILL.md preserves all key phrases Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Extract billing.ts fixture into coverage-audit-fixture.ts (DRY) - Refactor ship-coverage-audit E2E to use shared fixture - Add review-coverage-audit E2E for Step 4.75 - Update touchfiles: both E2Es depend on shared fixture Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The coverage audit E2E tests (ship + review) were only asserting exitReason === 'success' and readCalls > 0 — they passed even if the agent produced no coverage diagram. Add assertion that the output contains either GAP or TESTED markers. Found during /review. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codex adversarial review caught that plan-eng-review was inheriting "git diff origin/<base>...HEAD" from the shared resolver, but plan mode reviews a plan document, not a code diff. Plan mode now says: "Trace every codepath in the plan" and "Read the plan document." Ship and review modes keep the git diff instruction. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…e-catalog # Conflicts: # scripts/gen-skill-docs.ts # test/gen-skill-docs.test.ts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
{{TEST_COVERAGE_AUDIT}}shared resolver with 3 mode-specific placeholders (plan/ship/review)/reviewStep 4.75 for codepath tracing during pre-landing reviewTest Coverage
Tests: 417 → 451 (+34 from main merge)
Pre-Landing Review
No issues found. All auto-fixable items addressed in prior commits.
Reviews
Test plan
🤖 Generated with Claude Code