Skip to content

pr-review: severity-regex false-blocks on zero-count summary lines (e.g. πŸ”΄ Critical (BLOCKING): 0)Β #269

@cbeaulieu-gt

Description

@cbeaulieu-gt

Symptom

claude-pr-review/quality-gate fails on reviews that report zero findings in a findings-summary block. Example reproducer: glitchwerks/claude-configs#564 β€” approved, no real blockers, but the gate posted failure β€” 2 severity marker(s) matched.

Failing lines from the #564 review body

**Findings:**
- πŸ”΄ Critical (BLOCKING): 0
- 🟑 High-Priority (MAJOR): 0

The shadow gate (claude-pr-review/quality-gate-shadow) also posted agree:blocking on the same PR β€” strongly suggesting the synthesis fallback (which uses the same prose-regex to fabricate a marker when the persona drops it) miscounted these as 2 critical findings.

Root cause

pr-review/action.yml's pre-filter (currently at line 479, also lines 386–392 of the synthesis step and the shared regex in lib/severity-regex.sh):

grep -vE 'βœ…|\*\*(FIXED|ADDRESSED|RESOLVED)\*\*\s*$'

This covers two prior resolution-narration patterns:

But it misses a third pattern: zero-or-numeric-count summary lines, where the persona reports counts as part of a findings tally rather than a per-finding callout. These lines carry a severity token (πŸ”΄ Critical (BLOCKING), 🟑 High-Priority (MAJOR), **MAJOR**, etc.) but the value is a number, not a finding.

Both the authoritative grep at line 479 AND the synthesis fallback (lines 469–518) count these as load-bearing severity hits.

Proposed fix

Extend the pre-filter to drop lines where the severity token is followed by : <number> at end of line:

grep -vE 'βœ…|\*\*(FIXED|ADDRESSED|RESOLVED)\*\*\s*$|:\s*[0-9]+\s*$'

A tighter alternative is :\s*0\s*$ (only suppress zero-counts and let non-zero counts still potentially gate), but a non-zero summary line is also not itself a finding β€” the actual line items are β€” so the broader form is structurally cleaner.

Scope

All three callsites must stay in sync (the YAML comment at line 477 already flags this):

  1. pr-review/action.yml line 479 β€” authoritative quality-gate step
  2. pr-review/action.yml lines 469–518 β€” synthesis fallback (fabricates marker from prose when persona drops it)
  3. pr-review/lib/severity-regex.sh β€” shared regex source

Add a unit test under pr-review/tests/marker-cases/ (or wherever the existing pre-filter cases live) using the #564 reproducer string.

Why this matters beyond Phase 2

Per the locked plan (docs/superpowers/plans/2026-05-06-quality-gate-structured-marker.md, issue #185), Phase 4 retires the prose-regex as the authoritative gate. So this looks like a transient Phase 2 bug β€” but it isn't:

  1. Today (Phase 2): the prose-regex IS the authoritative gate, and is directly false-blocking real PRs.
  2. Cutover gate: the Phase 2β†’3 criterion is β‰₯20 reviews across β‰₯5 PRs, zero disagreement, zero marker-missing. The synthesis fallback uses the SAME buggy regex to fabricate a marker when the persona drops one, then the shadow gate compares prose-result vs marker-result for the agree:/disagree: label. A buggy regex inflates disagreement counts on both branches, which blocks the very cutover that would retire the regex.
  3. Post-cutover (Phase 4): marker-missing becomes fail-closed, but the synthesis step runs before the gate. Synthesis fabricates a marker via the prose-regex, so the buggy regex still survives in the synthesis afterlife whenever the persona drops the marker.

Fixing this is therefore on the critical path to closing #185.

Acceptance criteria

  • Pre-filter regex updated at all three callsites listed under "Scope" (sync verified).
  • New marker-case fixture under pr-review/tests/marker-cases/ containing the #564 reproducer; existing harness asserts the filter drops the zero-count lines.
  • Re-run review on claude-configs#564 (or a fresh equivalent PR) confirms quality-gate is now success and shadow posts agree:clean.

Related

πŸ€– Generated by Claude Code on behalf of @cbeaulieu-gt

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions