LLM CLI: add auto-compacting#11297
Conversation
vito
force-pushed
the
llm-compact
branch
2 times, most recently
from
6f3452d to
c89b89a
Compare
| break | ||
| } | ||
|
|
||
| // Check if we need to compact in-between steps |
There was a problem hiding this comment.
It seems, from reading the code, that we call autoCompact while HasPrompt could still true, so Compact + WithoutMessageHistory() would wipe the tool-output user message that the next Step is supposed to send no ?
Basically, we seem to have chosen to allow compaction in-between steps, which could be a destroying operation ?
What was the impact on the agent’s ability from your experience?
To test, I dropped the compaction threshold to 0.1 (even 0.0 but this led to an infinite loop ahah): it seemed fine-ish on small vibe feeling tasks
There was a problem hiding this comment.
Well - you have to interrupt to auto-compact at some point, otherwise the agent will just keep going and going until it thinks it's done, and the point of auto-compacting is to keep it from blowing out the context window while it's working away.
Basically, we seem to have chosen to allow compaction in-between steps, which could be a destroying operation ?
The tool results are fed to the compacting agent, since that works by just appending the compacting prompt to the message history so far. So they're not really dropped; they're fed in with the rest of the history, and then the entire history is reset to the summarized prompt.
What was the impact on the agent’s ability from your experience?
It's been decent; compacting will always be lossy, and there have been moments where I wanted to disable it, but overall it's helped a lot for long-running sessions, and acts as a nice defensive mechanism for when the agent gets caught down a bad path and needs to take a step back.
Signed-off-by: Alex Suraci <alex@dagger.io>
These are useful building blocks for implementing support for summarizing/compacting message histories, without coupling the core to a specific prompt. Signed-off-by: Alex Suraci <alex@dagger.io>
Signed-off-by: Alex Suraci <alex@dagger.io>
Signed-off-by: Alex Suraci <alex@dagger.io>
Signed-off-by: Alex Suraci <alex@dagger.io>
Signed-off-by: Alex Suraci <alex@dagger.io>
Feels like this is generically useful - Doug was implementing this for EditFile, and now doesn't need to. Signed-off-by: Alex Suraci <alex@dagger.io>
Signed-off-by: Alex Suraci <alex@dagger.io>
* mark AsPatch output as verbose so we can opt to show the whole thing when it's reasonably sized Signed-off-by: Alex Suraci <alex@dagger.io>
I've observed LLMs tend to run batches of tool calls that involve 1. a change to make, and 2. a test to run assuming the change is made. Right now when that happens we guarantee that the test runs WITHOUT the changes, which confuses the LLM. Instead we should do all the changes and _then_ the read-only calls, which will read the freshest state. Signed-off-by: Alex Suraci <alex@dagger.io>
Signed-off-by: Alex Suraci <alex@dagger.io>
Signed-off-by: Alex Suraci <alex@dagger.io>
Signed-off-by: Alex Suraci <alex@dagger.io>
Signed-off-by: Alex Suraci <alex@dagger.io>
2 participants
Adds two pretty low-level APIs to
LLMfor clearing either system prompts or user messages. The compaction dance uses both: first we swap out the system prompt, since the compacting agent is single-purpose, and then we swap out the user messages with the prompt returned by the compacting agent.The compacting agent is implemented in the CLI at the moment. An earlier iteration had this directly implemented by the API (
LLM.compact) but this setup gives us a bit more flexibility without requiring us to maintain a compacting prompt in the core API.