Code review and review summaries and inline review threads and terminology and

[JGronholz]

🏷️ Discussion Type

Bug

💬 Feature/Topic Area

Copilot Code Review

Body

🎯🟢 Address unresolved defects, not comments

"Comments" about defects are secondary artifacts:

• This comment
• That comment
• The summaries
• The threads
• Caveats
• Addendums
• ....

Side-step API quirks by focusing your "natural language" on what matters:

• unresolved defects are what needs attention.
• All of them, everywhere.

@Copilot Find the unresolved defects and fix them.

• Which unresolved defects?

Without hints, the agent must find them:

• With MCP tools
• By manually searching in code that hasn't changed since...

Code_review identifies most defects and many defect classes:

• Make your @Copilot Fix it. prompt pro-active, and pre-emptive.

Sometimes it's better if you don't provide line numbers:

• Be vague, as specifically as possible.
• You only need to be accurate.
• If you are too precise, copilot will "execute," rather than "interpret."

🎯🟢 By default, agents identify and fix defect classes, one instance at a time.

Here, then here, then here...

Both sides of the review loop are guilty.
🛠️ Fix: When a defect is found, scan adjacent/related code for the same and adjacent/related defect classes.

When code_review does this:

• All instances of a defect class are found in the same turn.

When session agents do this:

• All instances of a defect class are fixed in the same turn.

Don't rely on the review cycle's turn boundary.

⏱️ Great Wall Of Instructions (GWOI)

• Instructions about [entry-level programming] must be injected into every single session, or the review churn never ends.
• Instructions like that are bricks in the GWOI.
• In turn, the GWOI is responsible for regression, and waste ($$$$$$$$$$).

🛠️ Fix: native skills

• Same quantity of instructions
• Loaded one "brick" at a time
• JIT

SKILL.md is a router:

• Everything else is payload
• The payload can be enormous
• While the router remains thin

What is a "document"?
What are "instructions"?

Who is the "target audience" of the content that copilot generates:

• In which context/environment/scope?
• With which capabilities/permissions?
• How much "prose" is too much?

MCP helps.

Copilot needs some basic/more governance regarding instructions/skills:

• Copilot should not create "skill-like documents" in locations other than .github/skills.
• Github Copilot should always RTFM when working on Github and/or Copilot.
  • (Even if all other MCP is disabled?)
• Github Copilot embarrasses Github and/or Copilot with these kinds of mistakes.

These need to be clearly distinguishable, to Copilot, at all times:

• Instructions (including copilot-instructions.md)
• Documents
• Native skills

⏱️😵‍💫 Dots

🔴🟠 - Blocking unresolved defects currently exist ❌
🔵 - "Concerns" exist. This PR has the potential to create unresolved defects in adjacent code. ❌🔍⚠️
🟢 - Non-blocking unresolved defects may exist ⚠️
(This has majorly improved.)

🔴🟠 - Easy! Start a session. @Copilot Fix it.
🟢 - Easy! Just be smart. Read the parent issue(s) to see the big picture.
🔵 - FML..... <---- Not worth trying to decipher.... "Chat with Copilot" instead

🛠️🎯🟢❤️ "Chat with Copilot"

• The psuedo-sudo psuedo-stateful chatbot is very good at detecting when Copilot is being stupid/useless.
• Can take turns in-between review-cycle turns
• The PR agents have similar scope with more-restrictive permissions.
• The PR agents don't use that scope the same way the chatbot does.
• All chatbot does to @Copilot Fix it: 🔵 is read.
• Chatbot builds the closure matrix, and produces the exact @Copilot Fix the [remaining unresolved defects]. prompt.

This causes copilot to address "findings that can't be tied to specific lines of code":

• Copilot will seek-and-destroy, touching lines that are not attached to specific findings.
• Touching those lines breaks stalemates with the reviewer.

cloud_agent is chatbot, code_review, and PR/session/coding_agent.

• With the same model, chatbot behaves "smarter" (by default).

When Gemini 3.0 (pro with extended reasoning) first came out, I used it for this same purpose:

• My Gemini chats were living "tracker issues" that could be made aware of each other.
• Gemini's superior context window held the entire master plan without losing a single detail.
• I was just the go-between, manually copy-and-pasting Gemini's replies into Issues/PR's.

Need to find a clever way to integrate chatbot's superior "vision" into the PR review agents.

⏱️ Issues

Agents are already highly-integrated with PR's, but seem to have few capabilities regarding Issues.

• Tracker issues represent the plan.
• If nobody reads them, what's the point?
• If they aren't maintainable, they become stale/equally-useless.

The plan (issue), if there is one, is more important than the implementation details (PR).

• The tracker should be very detailed. Assigned when all sub-issues are closed, to validate the big-picture.
• The sub-issues tracking implementation details should cover one concern and be extremely detailed.
• PR's can be easily scrapped and "tried again" after hardening/strengthening the issue

⏱️ When I assign copilot to complicated slices of an EPIC tracker:

• The resulting PR descriptions are underwhelming.

• The prompt, that generates the PR description, that drives the PR progress, should be extremely detailed.

• These don't always get used (with issues or PRs)

• These don't always get updated

• When all of these are checked and all the work is done:

  • The entire description (completion checklist) gets replaced by the completion summary.
  • The checkboxes the user has been monitoring have vanished.
  • The latest review summary is 🔵 Good luck..

😵‍💫 Imprecise/missing language:

• What is a "comment"?
  Everything copilot posts to a PR is labeled:

  • "Copilot commented" (cloud_agent)
    Copilot's replies to the PR/prompt.
  • "Copilot left a comment" (code_review)
    "Copilot review overview"
    Which everyone calls a "summary"

  Replies to review threads are shown graphically attached to the thread:

  • The thread doesn't become "resolved" until next turn
  • And a manual page reload

• 🔵 What is a "concern"?
  Be specific.
  Summarize individual concerns, individually.
  What do I tell cloud agent to do, next turn?
  Give me the prompt.
  Session agents can't do this.

These terms need to get nailed down, so that copilot always produces what the user (meant) to ask for.

⏱️😵‍💫 When assigning copilot to issues, every PR automatically "Fixes: [Parent Issue]"

• This should not be invariant.
• Each PR is at least "related to" the issue.
• Multiple PR's may be required to close one issue.
• Agents do not have consistent write-access to issue bodies or PR "descriptions".
• code_review complains about things that cloud_agent can't fix from within its' ephemeral box

It currently appears that only the chatbot has the scope/permissions to fix Issue/PR misalignment.

• Which may be an intentional circuit-breaker

The initial post to a PR is the PR summary/description/plan:

• User considers it a "PR summary" and calls it that.
• It is a "comment" on a PR.
• The back-end calls it something else.
• Copilot parrots the user.

"Overview" + "Edit description" + "Fixes", at the same time, in the same place:

😵‍💫 Meanwhile, it's a "comment" on the discussion.

⏱️😵‍💫 Manual page reloading / second-class-feeling web UI

Multiple things that should get timely updates do not:

• When copilot finishes a session and replies to the PR, it is immediately visible
• Same for code_review
• When copilot replies to inline threads, manual reload is often needed
• When code_review has finished reviewing, the re-review indicator never updates

When a session agent has finished working and replied to the PR:

• The session continues to spin for many turns
• The session agent might make multiple pushes/comments
• "looking done", the user triggers code_review
• The coding session finishes 5 minutes later.
• The 5-minute review session finishes 2 minutes after that.
• There is some kind of "lag" between work-finished, end-of-session, replied-to-pr, and/or web-ui-updates.
• Note: Since nothing is being done, spinning wastes time, but not credits.

The same "lag" applies to review sessions:

• Spinning, and spinning, and spinning, while the work is already done.
• Note: Even though nothing is being done, spinning wastes time, and actions minutes.

1-5 minutes per session, per side, per review-turn, per PR, per...:

• It adds up.

These are mostly nit/minor, but the symptoms might all be pointing at/toward the same problem.

⏱️ "Something" is wrong in the review (metadata?) handling:

• code_review "loses" findings and refuses to "outdate" them
• or whatever mechanism handles resolution status...
• churn

⏱️😵‍💫 Review summaries

Complaints about "findings that can't be tied to specific lines of code":

• Are actually complaints about the review summaries.
• Non-inline-able findings:
  • Turn 1: Get summarized
  • Turn 2: Get collected
  • Turn 2: Get summarized
  • Turn 56: Get summarized....

Previous summaries don't get "linked" or "copied forward:"

• "Summary - Turn 1 : ABC"
• "Summary - Turn 2 : ABC + XYZ"

"Remaining unresolved defects" becomes blurred by the most-recent one-liner:

• "Summary - Turns 1-20, 32-44, 46, 48-50 : AXYCZ"
• B will be rediscovered later ("in code that hasn't changed since...")
• Just when you thought you were done 🤡

What is a "hidden review summary"?

• If you were an agent, you wouldn't "find" these unless you were explicitly told to look for (and report) them.

I discovered the existence of "hidden" review summaries with chatbots help.

⏱️😵‍💫 Those one-liners lose the most important details at the worst times:

🔵 - Fix the remaining unresolved conflicts (0 open findings) <--- Which ones?
🔵 - Sensitive, security <---- Possibly intentional - parent issues offering explanation are ignored
🔵 - Cross-domain, scope creep <---- Possibly intentional - parent issues offering explanation are ignored
🔵 - PR says Y, Z happened <---- copilot/user ID10T error - parent issues offering explanation are ignored
🔵 - Pre-merge acceptance criteria includes post-merge artifacts <---- copilot/user ID10T error
🔵 - Stubs/placeholders detected (PR implements one slice of an epic Issue) <---- parent issues offering explanation are ignored

These are all separate issues and should be kept separate.
🔵 is already ambiguous by nature.

🛠️🎯🟢 Closure matrix

Instead of a natural-language summary, emit a closure matrix in PR replies:

• The agent must correlate the entire PR history, every turn.
• The latest closure matrix always contains all currently unresolved defects
• Which is what matters for merge.

Like so:

• Collect all findings (from summaries, inline threads, metadata, everywhere)
• Determine resolution status
• Omit resolved/outdated
• Emit "closure matrix" in review summary:
  [ comment_id -> root cause -> fix location -> regression test(s) -> status ]

The closure matrix:

• Ensures convergence
• Establishes confidence
• Reduces churn/waste/cost
• Avoids regressions
• Is a "review summary"
• Is authoritative
• Is deterministically produced
• Is too wide (have to use scroll bars to read status column)

⏱️😵‍💫 The only way to resolve a review thread is for a human to click the "Resolve conversation" button

Otherwise:

• Turn 1: The finding gets created
• Turn 2: The finding gets addressed
• Turn 2: The finding gets replied
• Turn 3: The reply gets seen
• Turn 3: The code gets checked
• Turn 3: The problem suddenly vanished
• Turn 3: The finding became "outdated"

You must occasionally resolve threads manually because [technical/backend] causes code_review to "misplace" the conversation.

• After many rounds of cloud_agent insisting it is finished, code_review won't budge
• User discovers each finding is addressed, validated, replied, and extremely "outdated", yet remains "unresolved"

After all findings have been manually resolved:

• Request re-review
• 🟢

⏱️😵‍💫 Claim-to-evidence map

• This is another thing I learned about from Github Copilot
• While debugging Github Copilot

Github can't fix Copilot MCP and docs.
Github can't fix Amazon MCP and docs.
Github can fix Github MCP and docs.

Every claim in the Github MCP and docs should be exhaustively verified.

• You will find things...

Some of the claims made by X, affect the ones made by Y, affect the ones made by Z.

• Although Y and Z aren't the responsibility of X, trying to make X use Y or Z will surface problems with them all.
• X is probably Github, Y is probably Copilot, and Z is not "officially" your problem. ❤️

🛠️🎯🟢 The solutions:

• Demand a closure matrix in PR replies

• "Chat with Copilot"

  • Can produce the closure matrix

• Use this existing feature 🤡
  Users unaware of this believe the only way to find comments is by scrolling/digging.
  This does not locate findings that can't be tied to specific lines of code.

• PR's are "chats" (stateless LLM using a context window to mimic statefulness)
  Encourage agents to be verbose about what they're doing "out loud."
  Every post to the history becomes context for the next turn (code_review).
  Don't assume agents "just know" what each other are doing.
  Agents are all stateless, between stateful-looking sessions.

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐