Optimizing Token Burn Without Model Downgrades

[purvanshjoshi]

Discussion: Optimizing Token Burn Without Model Downgrades

As AutoMaintainer scales, the token usage across the LangGraph AI agents has become a significant bottleneck, especially when relying on the free tiers of APIs like Groq (which enforces a strict 100,000 Tokens Per Day limit).

Currently, the agents consume a large amount of tokens because:

1. The Architect Node passes massive amounts of AST context to the LLM.
2. The Agent Memory retains full tool execution history in the context window.
3. The Implementer and Maintainer pass full file contents back and forth during reviews.

While the easiest solution is to downgrade simple agents to llama-3.1-8b-instant or remove the AST dump completely, we want to explore optimization strategies that preserve the 70B model's reasoning capabilities and the current data flow.

Proposed Advanced Optimizations

Here are three advanced strategies to optimize token burn without downgrading the AI:

1. LLM Prompt Caching (Zero-Token Replay)

Currently, if an agent queries the same file or runs the same logic across different pipeline iterations, the API is hit multiple times.

• Solution: Implement langchain_core.globals.set_llm_cache using SQLiteCache or an in-memory cache.
• Impact: Duplicate prompts will hit the cache instead of Groq, costing 0 tokens and executing instantly.

2. Code Minification & Whitespace Stripping

LLM tokenizers are highly sensitive to whitespace and formatting. A Python script with heavy indentation and docstrings costs significantly more tokens than a minified version, even though the semantic logic is identical.

• Solution: Inject a pre-processing layer that programmatically strips out trailing whitespace, blank lines, and unnecessary indentation from source code and READMEs before injecting them into the LLM context.
• Impact: Reduces structural token cost by roughly 15-20% per file analyzed.

3. Prompt Compression (LLMLingua)

Passing massive 15,000-character AST dumps burns tokens on stop words and redundant syntax.

• Solution: Integrate a prompt compressor like LLMLingua. This mathematically removes stop words (like the, and, is) and compresses the prompt structure.
• Impact: Acts like a .zip file for text, reducing the size of the prompt by up to 30-40% without losing semantic meaning before it reaches Groq.

---

Request for Comments

Which of these optimization paths should we prioritize for the next sprint? Caching and Whitespace Stripping offer the highest ROI for the lowest architectural complexity. Please leave your thoughts below!

[purvanshjoshi]

@archittmittal reviews and new ways to do are welcome