fix(pr-review): require attribution on every finding - #86
Merged
Conversation
There was a problem hiding this comment.
{
"verdict": "approve",
"summary": "PR adds a MANDATORY block to `commands/pr-review.md` Step 5 requiring every finding to open with a bold file reference and carry an inline rule tag, plus a corresponding `fix:` CHANGELOG entry. All judgment-rule triggers (changelog rules, agent-cmd/command-thin) are satisfied — the command addition is procedural documentation, not leaked business logic; the CHANGELOG bullet uses the correct conventional prefix and sits correctly under ## Unreleased. No violations found.",
"comments": [],
"concerns_addressed": [
"correctness: commands/pr-review.md — the anti-example `**Inconsistent due_date choice**` is correctly quoted; first bold = summary phrase is correctly identified as wrong",
"correctness: commands/pr-review.md — the three accepted shapes (**`path:LINE`**, **`path`**, **`path:START-END`**) are documented as specified; consistency with harvester implementation is an implementation detail beyond rule scope",
"tests: commands/pr-review.md — the new finding format is a command output specification, not new code logic; no test gap introduced by this change"
]
}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the blocker measured by the review benchmark:
/coding:pr-reviewnon-deterministically emits findings carrying neither a file reference nor a rule tag, so they cannot be keyed, matched or scored.The measurement
Three benchmark runs over the 5-PR dev fixture (
codingv0.35.4, sonnet/medium/short), review cache cleared between each:6 of 15 attempts (40%) produced no row, rejected by the harvester's
UNATTRIBUTABLE FINDINGgate. No PR fails consistently —quant#109failed, succeeded, failed. The missingness is not ignorable: it correlates with the runs that found the most (thegithub-pr-review-agent#11rejection happened on the run where it produced six items), so averaging the survivors understates variance invisibly.Why the fix belongs here and not in the harvester
Relaxing the harvester to accept body-only findings would restore exactly the silent acceptance that produced defects D2, D4 and D7 in the benchmark runner — a layer emitting a plausible wrong number instead of refusing.
bench/README.md§ "Reading review output" is the contract; the command simply never asked for what that contract requires. Step 5 mandated the three severity headers and described what kinds of issue belong under each, but said nothing about the shape of an individual finding item.The change
commands/pr-review.mdStep 5 only. Fifteen lines, no rule content touched, no harvester change.- **Inconsistent due_date choice**is wrong;- **task/x.yaml:24** — inconsistent due_date choiceis right.**`path:LINE`**,**`path`**,**`path:START-END`**— the four shapes the harvester actually parses.**Notes:**block, never as a bullet inside a severity section.Verification
Two live runs against the modified command over the full 5-PR fixture:
10 of 10 findings carry a real path; zero rejections across 10 PR attempts. Run A also exercised the range shape (
.github/workflows/ci.yml:27-34) and a deep path (task/recurring-schedules/prod/plan-next-week.yaml:24).Run A exposed a residual gap that run B closed: a finding led with a bold summary phrase rather than a path, satisfying "bold run first" while remaining unattributable. That is why the instruction now requires the bold run to be the reference, with the failing shape quoted verbatim.
Not evidenced:
rule_idis 0/10 —grep '(rule:'finds no tag in any capture. Every finding in these runs was a CI/YAML/Makefile issue with no matching entry inrules/index.json, so omission may be correct per the instruction. Separating "correctly omitted" from "ignored" needs a run over a PR where code rules actually apply, and is deliberately not claimed here.make precommit30/30, 103 tests. Note these prove nothing about this change — no test reads the command's instructions; the live runs are the evidence.