Intelligent Context Management for LLM APIs

[nikzasel]

Problem: Sending full LLM context (e.g., 1M tokens) with every request is inefficient and costly, especially for minor interactions.

Proposed Solution: Implement intelligent, user-configurable context management when the context window is full, near capacity, or a manual limit is set.

Key Features:

1) Manual Context Size Limit:
Allow users to set a maximum token limit for the context, overriding the model's default.

2) Configurable Truncation Strategies (when context is full/near capacity):

• Clear Oldest: Automatically remove tokens from the beginning of the context.
• Summarize/Condense Middle: Intelligently reduce less critical or redundant parts of the context (e.g., via internal summarization/compression).
• Integrated LLM Rewriting/Condensing:
  • Leverage existing text condensation capabilities.
  • Provide an option to trigger rewriting/condensing based on a configurable percentage of context capacity, not just when full.

This will significantly improve efficiency and cost-effectiveness for LLM interactions.

[Juice10]

There is an automatic context compression setting you can turn on which should help with this and maybe do what you are hoping to achieve @nikzasel:

[crizCraig]

Thanks @nikzasel and @Juice10 - this is awesome!

We've worked super hard on this exact problem and have released a tool that automatically manages context for Kilo code.

More details:
https://www.reddit.com/r/kilocode/comments/1mph0o3/63m_tokens_sent_with_only_137k_context/

Do you think it would be useful to integrate a memory provider hook so that any model can have intelligent context management?

[juzweb]

Crazy Kiro, sending 20k of cotext on an empty project, and with a prompt of How are you :(