What happened
Two patterns of false-positive or miscalibrated review findings emerged on PR #3489:
-
Build tag ordering: qodo-for-conforma[bot] flagged build tag placement in cmd/validate/image_test.go and internal/image/validate_test.go, recommending //go:build be placed as the first line. The author dismissed both findings: "Every test file in the repo puts the license header before the build tag. This file follows the same convention." This is an intentional repo-wide convention.
-
Documentation gaps on security fixes: The fullsend review agent issued REQUEST_CHANGES verdicts in 6 of 10 review cycles for documentation gaps (THREAT_MODEL.md missing --vsa-public-key, DESIGN.md not describing new validation steps). Both human reviewers considered these non-blocking for a security fix and merged the PR with the agent's REQUEST_CHANGES outstanding. The team's workflow is to merge security fixes quickly and address documentation in follow-up work.
What could go better
Both findings are technically correct observations but were miscalibrated for this repo's conventions:
-
The build-tag false positive is a straightforward convention gap — the repo consistently puts license headers before build tags, but no AGENTS.md guidance documents this. Review agents (both fullsend and qodo) that check Go conventions will flag it unless told otherwise.
-
The documentation severity miscalibration is more impactful. Six REQUEST_CHANGES verdicts for doc gaps on a security fix creates friction and teaches authors to ignore agent verdicts. The conforma/cli team treats doc updates as follow-up work for security PRs, but the review agent has no signal about this workflow preference.
Confidence is high for both patterns — the build-tag convention is visible across the entire repo, and the security-fix-then-document workflow is a common pattern in security-sensitive projects. conforma/cli already has a pattern of adding AGENTS.md guidance to prevent false positives (issues #3503, #3458, #3508, #3451, #3389).
Proposed change
Add two sections to AGENTS.md:
1. Go file header convention:
## Go file header convention
Go source files in this repository place the SPDX license header comment
before the `//go:build` tag. This is the established convention across
all Go files — do not flag build tag placement as a style violation.
2. Security fix review expectations:
## Security fix review expectations
Security bug fixes and vulnerability mitigations (PRs labeled `bug` +
`Possible security concern`, or referencing security-related Jira tickets
like EC-1842) should not be blocked on documentation updates.
Documentation gaps in files like `THREAT_MODEL.md`, `DESIGN.md`, and
user-facing docs should be flagged as informational comments (not
blocking change requests) when the PR's primary purpose is a security
fix. Authors are expected to create follow-up issues or PRs for
documentation updates after the security fix is merged.
These additions follow the established pattern in this repo of documenting conventions to improve agent review accuracy.
Validation criteria
- Future review runs on Go files in conforma/cli should not flag license-header-before-build-tag ordering as a violation. 2. On the next 3 security-fix PRs (labeled
Possible security concern) where documentation gaps exist but the code changes are correct: the review agent should flag doc gaps at COMMENT severity rather than producing REQUEST_CHANGES verdicts based solely on documentation findings.
Generated by retro agent from #3489
What happened
Two patterns of false-positive or miscalibrated review findings emerged on PR #3489:
Build tag ordering: qodo-for-conforma[bot] flagged build tag placement in
cmd/validate/image_test.goandinternal/image/validate_test.go, recommending//go:buildbe placed as the first line. The author dismissed both findings: "Every test file in the repo puts the license header before the build tag. This file follows the same convention." This is an intentional repo-wide convention.Documentation gaps on security fixes: The fullsend review agent issued REQUEST_CHANGES verdicts in 6 of 10 review cycles for documentation gaps (
THREAT_MODEL.mdmissing--vsa-public-key,DESIGN.mdnot describing new validation steps). Both human reviewers considered these non-blocking for a security fix and merged the PR with the agent's REQUEST_CHANGES outstanding. The team's workflow is to merge security fixes quickly and address documentation in follow-up work.What could go better
Both findings are technically correct observations but were miscalibrated for this repo's conventions:
The build-tag false positive is a straightforward convention gap — the repo consistently puts license headers before build tags, but no AGENTS.md guidance documents this. Review agents (both fullsend and qodo) that check Go conventions will flag it unless told otherwise.
The documentation severity miscalibration is more impactful. Six REQUEST_CHANGES verdicts for doc gaps on a security fix creates friction and teaches authors to ignore agent verdicts. The conforma/cli team treats doc updates as follow-up work for security PRs, but the review agent has no signal about this workflow preference.
Confidence is high for both patterns — the build-tag convention is visible across the entire repo, and the security-fix-then-document workflow is a common pattern in security-sensitive projects. conforma/cli already has a pattern of adding AGENTS.md guidance to prevent false positives (issues #3503, #3458, #3508, #3451, #3389).
Proposed change
Add two sections to
AGENTS.md:1. Go file header convention:
2. Security fix review expectations:
These additions follow the established pattern in this repo of documenting conventions to improve agent review accuracy.
Validation criteria
Possible security concern) where documentation gaps exist but the code changes are correct: the review agent should flag doc gaps at COMMENT severity rather than producing REQUEST_CHANGES verdicts based solely on documentation findings.Generated by retro agent from #3489