Classify review preambles as needing nothing from the author - #245
Merged
trask merged 1 commit intoAug 12, 2026
Merged
Conversation
A review summary that only says where the review's comments came from or that the author may push back on them was ambiguous enough that the classifier's fail-safe kept the pull request with its author, blocking it on a note that asks for nothing. The eval baseline already recorded this shape as flaky. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Clarifies that review-only preambles require no author action, preventing incorrect dashboard routing.
Changes:
- Updates classifier guidance for review preambles.
- Adjudicates a previously flaky evaluation case.
- Documents the routing rationale.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
RATIONALE.md |
Documents preamble handling. |
reviewer_feedback_cases.json |
Adds the adjudicated expected label. |
classification.py |
Extends the reviewer-feedback prompt. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Pull request dashboard statusMerged · refreshed 2026-08-12 13:53 UTC Status above doesn't look right?
|
maryliag
approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A review summary like
More lightly filtered AI-generated feedback, push back freelywas routing the pull request back to its author, as it did on opentelemetry-java-instrumentation#18995. The preamble asks for nothing itself — the review's inline comments are already independent dashboard items — but the wording reads enough like a request that the reviewer-feedback classifier's ambiguity fail-safe (When you cannot tell, answer author_action) kept it with the author. The eval baseline had already caught the shape:pr-review-4830006514is recorded flaky at 4/5no_author_action, and near-identical text on the same PR classified both ways on different refreshes.REVIEWER_FEEDBACK_PROMPT_TEMPLATEnow names the case, so it no longer depends on the fail-safe:Classification cache keys hash the prompt text, so existing cached verdicts for these items are invalidated automatically and reclassified on the next refresh.
pr-review-4830006514is now adjudicatedno_author_action, joiningpr-review-4811760271("AI-generated review"), so the scorer measures this behavior instead of leaving it as unlabelled flakiness.Measured against
gpt-5.4-miniover the two recorded preamble cases, three synthetic ones, and a 60-case sample of stable baseline cases, 3 trials each, old prompt versus new:no_author_actionin every trialauthor_actionin every trialpr-review-4380645795("I like where this is going, thank you for working on this! Added minor comments/questions.") moves tono_author_action. The old prompt was itself unstable on it, and the baseline already records the same shape — "Just a couple more minor comments. Looks pretty good!" — asno_author_action, so this is more consistent than the label it replaces. The PR still waits on the author through the inline threads that summary introduces.A narrower variant that dropped the "some nits below" example was tried and rejected: it left 11 of 65 cases answered inconsistently.