Pre-inference context optimization: reducing prompt tokens before GPU

[tcconnally]

The Problem

Every token in your prompt costs GPU inference time. LLM agents routinely send 5,000-50,000+ tokens of context (system prompts, tool definitions, memory, skills, conversation history) even though only 10-20% of that context is relevant to the current task.

The Approach: Pre-Inference Context Optimization

What if we could reduce tokens before the prompt ever reaches the inference engine? Context optimization sits between the agent and the inference call:

Agent -> Context Engine -> Optimized Prompt -> TensorRT-LLM -> Response

The context engine can:

1. Prune irrelevant sections: Remove tool definitions the agent won't need
2. Compress redundant information: Merge duplicate context blocks
3. Prioritize by relevance: Sort context by task relevance, truncate lowest-priority
4. Token-budget awareness: Target a specific token budget, dynamically adjust

Real-World Numbers

From running Perseus (a live context engine for AI agents) in production:

• Average context reduction: 40-60%
• Without any quality loss on standard benchmarks

Combined Effect with TensorRT-LLM

If TensorRT-LLM gives you 2x inference throughput at the GPU level, and context optimization gives you 2x reduction in prompt tokens, the combined effect is multiplicative: you're effectively serving 4x the requests per GPU.

Reference Implementation

Perseus (https://perseus.observer / https://github.com/Perseus-Computing-LLC/perseus) implements live context rendering that only includes what's relevant to the current task, rather than dumping static pre-authored prompts.

Discussion Questions

1. Has anyone experimented with token reduction before inference?
2. Would a context-optimization middleware layer in the Triton pipeline make sense?
3. What metrics matter most: TTFT reduction or total throughput?