GitHub Copilot summarization is too simplistic, losing crucial semantics

[garretwilson]

Conversation summarization seems to be worse in v1.107, or maybe I'm imagining things. It seems to be just losing things that any real summarization would have kept. It's almost as bad as truncation now.

Here is just one example. This is happening frequently, especially in a long-running architectural discussion, especially with multiple refactorings along the way.

In this example, the things it mentioned were really recent, after the last summarization. The things it had forgotten were before the last summarization. (And the part about deferring caching until later, and merely adding a TODO in its placed, I had told it over and over, several times over several days.)

This sort of thing seems to be happening significantly more after v1.07, but I can't swear to it. In any case it is now happening over and over. I don't know if you're truncating or if you're just telling some the LLM to "make a summary", but either way is inadequate for a world-class tool used for enterprise software development. The LLM needs to keep a running semantically weighted summary, continually aggregating the decisions made. A robust summarization strategy would include the following:

• Provide a way to designate an LLM just for summarization (Specify GitHub Copilot LLM specifically for summarizing conversation history #282207).
• Ask the LLM to analyze and categorize sections of the conversation, e.g. "brainstorming", "deep dive", "conclusion", "plan", "summary". (Those are off the top of my head as I'm writing this.) Examples:
  • Some parts of the conversation may go back and forth for ages on a deep-dive on some trivial detail. The summarization need only note that this deep-dive discussion happened, and simply preserve its conclusion.
  • Anything that is a "conclusion" should have a much higher weight than "brainstorming", and should only be lightly summarized, e.g. removing redundancy and wordsmithing for conciseness.
  • Anything that has been categorized as a "summary" should be maintained almost literally and carried forward. (After all, if the LLM provided a summary in the conversation, a resummarization will likely lose fidelity, like using JPEG compression on an already-compressed JPEG image.)
• You might use a cheap LLM to discover, demarcate, and categorize the sections; while using a more robust LLM to make the actual summarizations.
• The initial part of the conversation, as well as the most recent part of the conversation, should both have higher weight to be maintained in the summary.
• GitHub Copilot needs a facility for a user to flag a response as important, so that it is kept with utmost fidelity in summarizations.

All that is just off the top of my head. I'm sure if you sit down and brainstorm with team members you can come up with something much more sophisticated. The current implementation is honestly not adequate.

Version: 1.107.0 (user setup)
Commit: 618725e67565b290ba4da6fe2d29f8fa1d4e3622
Date: 2025-12-10T07:43:47.883Z
Electron: 39.2.3
ElectronBuildId: 12895514
Chromium: 142.0.7444.175
Node.js: 22.21.1
V8: 14.2.231.21-electron.0
OS: Windows_NT x64 10.0.26200

[garretwilson]

Or just allow a pluggable conversation summarization strategy, and I'll write my own conversation summarization implementation. (Or hire me and I'll help out with both the design and implementation.)

[garretwilson]

This is so troublesome that several times a day I'm having to copy-and-paste in a running summary of the LLM's last-produced design summary, ask the LLM to update that design summary based upon our latest discussions, and then save that updated summary somewhere, because the automatic "conversation summarization" seems to basically discard most of it. This manual maintenance of a running, continually updated summary is something that I would expect the background "conversation summarization" to do for me. Yes, it's complex. Yes, it requires some plan and knowledge of semantics. Yes, the summarization needs to be design just as intricately and robustly as the LLM agent capabilities.

What I'm saying is that when I tell the agent to do something, it goes and makes a plan and updates my code. That has to be complex and multi-step, or otherwise it will come up with some simplistic code. The same approach needs to happen with summarization, because maintaining context is just as important as implementing a design—in fact the latter depends on the former. In other words, planning to summarize must be an elaborate thing just like planning to implement what I asked it to do, because further iterations of the implementation depend on a running, accurate context.

Yes, it's complex, intricate functionality. Must it be done? Also yes.

[garretwilson]

Let me give you a very typical and relevant example. I just had a second LLM review the design and implementation we've been working on for a couple of weeks—a pretty large effort with deep design discussions. The review picked up a subtle bug about "empty versus absent" semantics in the framework and their implementation on one platform. In designing the fix, I needed to refer to conversation we had already had off and on about related design issues.

Of course the GitHub Copilot conversation summarization had long removed all those intricate details. But they are still in the Markdown export of the chat (see #272076 — why is so fundamental functionality ignored?). So I just now had to spend like half an hour painstakingly searching and picking through the past conversation (days and days and days of conversation) to copy and paste relevant snippets into the prompt to bring the LLM back up to speed.

But that sort of problem is what the LLM itself is supposed to solve! I shouldn't have to do that! For the LLM, something like that is trivial! It already does that when updating code, for example; when making a change it:

1. Searches in the codebase for search terms.
2. Captures the text before and after the finds to provide context.

Trivial.

And that's the point I was trying to make earlier: management of conversation history should be a first-class functionality, just like code-editing, because effective code editing requires an accurate context!! That's it! Undeniable. But GitHub Copilot does not currently treat conversation history management as a first-class citizen. It just does some simplistic "summarization" and that is that. And the entire coding experience suffers greatly because of it.

In this case for example, imagine that I had a command named #history. If the LLM sees that in the prompt, here is what it does:

1. Summarizes the current prompt to extract relevant semantic search terms.
2. Searches through the conversation history (the API should expose this to the LLM) to find those keywords.
3. Grabs conversation segments before and after the search hits to provide context.
4. Creates a summary of all those segments.
5. Inserts that summary into the prompt behind the scenes.

This would provide an enormous help! Note that this #history search capability is in essence conversation history RAG, and is complementary, not a replacement, to good conversation history management.

I'm pointing out ways that the conversation history management is substantially under-cooked and anemic; but I'm also suggesting constructive ways forward.

[garretwilson]

Here's another example. Try it yourself. Start a conversation with something like "We are going to implement ticket FOO-123. Here is the description. … We will discuss and then implement each piece."

Simple enough. The problem is after a few days, every time you ask the LLM to generate a Git message with the ticket number, the LLM will say, "I'm sorry, I don't know the ticket number," or it will make one up. I'll remind it again, but by the next day it will have forgotten it again. But the ticket number is fundamental to the work that was going on!

You might say, "you can't expect the LLM to remember the ticket number after so many summarizations of the conversation; how does it know which things to remember?"

But I do expect that. That's exactly my point. If the summarization algorithm isn't retaining that crucial information, then it's some naive, simplistic algorithm—or more to the point, the summarization algorithm is not a first-class citizen like the agentic code editing is! If summarization were a first-class citizen (and it should be!), there would be a whole infrastructure with back-room discussions about what should be summarized.

For example, maybe one question is to ask the LLM (or another fast, cheap LLM) at the first summarization: "Is there any information in this conversation to be summarized that represents information that will be continually relevant through the rest of this conversation (e.g. a ticket number)?" You take those things and put them in a special place. Then on each summarization you ask, "is there any more information that needs to be added to the list of those things tracked throughout the conversation?" etc.

See, I'm proposing building out a whole framework for intelligent, agentic conversation summary. It's a big job. It has to be done for real-world, enterprise-level coding. If you ignore the summarization and think it can be a second-class citizen with a naive algorithm ("Hey, LLM, make a smaller summary of this."), there will be competing projects that take maintaining the LLM context seriously.

I would write this framework for you except 1) you haven't hired me, and 2) I have so much other work to do. But hopefully you see where I'm going with the concept.

[garretwilson]

Here's a good example (as I mentioned elsewhere) which bears repeating. If summarization happens in the middle of implementatoin, it the LLM will lose crucial details of what we had just planned, sending the LLM off into another direction midstream! This is an extremely poor—not to mention unproductive—user experience.

Here is the LLM's explanation:

… The conversation was summarized mid-implementation. I only have the summary's description of what was done, not my actual memory of doing it or the reasoning at each step. The summary told me:

• "Updated RecordIntrospector to use type resolution for generic records"
• "Added @implNote explaining raw type fail-fast behavior"
• Created test files

But I don't have the actual context of why I made certain choices (like putting the @implNote in isSufficientFor() instead of RecordSchema). That decision may have made sense in the moment for some reason I articulated, or it may have been a mistake—I can't tell from the summary.

This is why I asked you to help me understand where things went wrong. I can see the current state of the code, but I've lost the thread of reasoning that led there.

And this is something we had discussed in the prompt right before the implementation!! If the summarization is so naive that it throws a way important details of the plan that was the impetus for the currently ongoing work, midstream, then the summarization algorithm is not robust and enterprise-level.

[garretwilson]

The current conversation summarization implementation breaks the planning process.

1. The LLM proposes a plan involving large sections A and B.
2. We discuss details about section B.
3. Conversation summarization happens.
4. I ask the LLM to update the plan based upon what we discussed.
5. The LLM only remembers about the B part which we discussed; it has already forgotten about section A.

This is broken. This is not an intricate corner case or the case of forgetting something that happened weeks ago in the conversion. This is a failure to carry through essential parts of ongoing planning that just happened before the summarization.

Takeaways

• Summarization must be intelligent (as I have been laying out in this ticket), and realize which parts are important and need to be carried through.
• Part of the algorithm may need to go as far as to carry over the previous couple of prompts almost verbatim, only summarizing for more density (e.g. smaller words, reducing redundancy and tool work, etc.). This should be part of the intelligence.

Workaround

Now every time I see the dreaded "summarizing conversation", if in the middle of planning I literally have to convert the conversation to Markdown and copy+paste the previous prompts into the conversation so as not to lose context. This workaround is defeating the purpose of summarization—but summarization as currently implemented is defeating the purpose of the conversation!

The GitHub Copilot team really needs to assign a team member just on this functionality. I stress summarization should be approached as a first-class citizen, just like agentic coding because it is the fundamental context for agentic coding.

[roblourens]

You don't need to convince me that there's an issue with the approach! I've felt the same way since we first implemented it. I can think of a dozen different strategies, and you describe some possibilities here, but yes we need to fund the time to actually implement and more importantly do the evals to figure out what actually works in which scenarios with which models.

[garretwilson]

You don't need to convince me that there's an issue with the approach!

I imagined you probably understood that there was a problem. The information I'm providing is meant to be constructive toward finding a solution:

• When designing a solution for an issue as complex as this one, you need many anecdotes from real user experience in order to explore the contours of the problem and determine semantically what it should do. I'm providing some of that user data.
• The solution will be complex and may involve new ontological constructs or processes. I'm drawing on my vast experience in this sort of design to provide you with some ideas. If I were on your team, this is the sort of thing I would be drawing out on a board (in a physical room or virtually).

… yes we need to fund the time to actually implement and more importantly do the evals …

I don't know whether "fund" was a typo, but it works either way. 😉 Just let me know where to apply.