AGENT PROMPT AND CONTEXT BLOAT

[jhs88]

ROADMAP.md mentions

Agent prompt/context bloat. Agent mode is too heavy for smaller local models: tool schemas, skills, memory, documents, and instructions can eat the context before the user request really starts. We need slimmer prompts, better tool selection, smaller default tool sets, and clearer guidance for models with 4k/8k/16k context windows.

I have lots of experience with this and have lots of great examples using pi & opencode configs. I run small models at home and have gotten pretty good at it. I thought id start this discussion as a place for anecdotal notes on what I've found to be helpful. I don't know if they transfer but these things have helped me a lot.

[Michiel-VandeVelde]

If you have real examples from small local models, pi setups, or opencode configs, please share them here. Especially interested in what actually helped reduce prompt/context bloat: smaller tool sets, shorter system prompts, config patterns, tool-selection rules, memory/document limits, or anything model-size specific.

[jhs88]

Ive been trying to do a write up about all this stuff for a while now. Just haven't gotten around to it. I have a config on my profile if ur curious for pi.

[jhs88]

If you want to chat more about this i'd def be down. I have lots of stuff I've figured out over the last few years

[ferhimedamine]

Context bloat for small models is a retrieval problem disguised as a prompting problem. The issue is not that tool schemas and memory are too large — it is that EVERYTHING gets injected regardless of relevance to the current query.

The fix: externalize memory and retrieve only what is relevant. Instead of injecting 15 tools, 3 skills, 200 lines of memory, and 5 documents into every prompt, the agent queries an external memory service for the top-N facts relevant to the current user request, and only THOSE get injected. For a 4K context window, that might be 500 tokens of focused context instead of 3,000 tokens of everything.

For tool selection specifically: store tool descriptions as memories with semantic embeddings. At query time, retrieve the 3-5 tools most relevant to the user request rather than injecting all 50. Small models perform dramatically better when given 3 relevant tools vs 50 irrelevant ones.

The hybrid retrieval approach (BM25 keyword match + vector similarity) is critical for small models because it catches both exact tool names and semantic matches without requiring the model to do the filtering itself.

Focused retrieval pattern (query → top-N relevant memories only): https://github.com/Dakera-AI/dakera-py/blob/main/examples/hybrid_search.py

[Searaphim]

@ferhimedamine While their system is still buggy, isn't this already what Odysseus is doing?
That's the impression I get reading this https://github.com/pewdiepie-archdaemon/odysseus/issues/3934

Or is your proposal somehow different from how typical Tool-RAG behaves?

[alteixeira20]

I think the Tool-RAG direction is part of the answer, but I would not treat it as the whole prompt-bloat problem.

There are a few related surfaces getting mixed together here:

1. Tool selection - retrieve or expose only the tools likely to matter for the current request.
2. Memory/document retrieval - inject only relevant facts/docs instead of broad context.
3. Prompt assembly budget - measure what the final assembled agent prompt actually contains.
4. Just-in-time guidance - avoid loading detailed tool/skill/MCP instructions before the task path is known.
5. Small-model compatibility - keep the final prompt shape simple enough that 4K/8K/16K local models can still follow it reliably.

So even if Odysseus already has a Tool-RAG-like path, I think the next useful step is measurement rather than assuming the current behavior is enough. We need to know where the assembled prompt tokens are going: base rules, builtin tools, MCP descriptions, skills, memory, documents, date/user context, integration-specific additions, etc.

That is why I think the measurement-first work in #2750 is the right near-term path. Once there is a section-level token breakdown, it becomes much easier to tell whether Tool-RAG is doing enough, where it is not being applied, and which prompt surfaces should be slimmed or loaded just-in-time.

In short: Tool-RAG is probably one mechanism, but the broader goal should be a measurable prompt-budget system that proves smaller models are only seeing the minimum relevant surface for the current task.