Your retrieval budget may be based on a context window you are not actually being served

[behrnt-slatgng]

Disclosure up front: I build Grunz, a hosted chat + coding agent on open-weight models. Nothing to sell here — this is a retrieval-sizing problem I keep hitting and I suspect it affects more LlamaIndex users than realise it.

The number your retrieval budget is based on is probably wrong

Chunk size, top-k and reranking all get tuned against an assumed context budget. That budget is normally taken from the model card.

The model card number is a property of the weights. The number you actually get is a property of whoever serves them — a KV-cache budget the operator traded off against concurrency. Across the hosted endpoints I've tested, most serve around 32K regardless of what the card claims. A few reach 256K. I have not found one actually serving the 1M figures that show up in release posts.

So if you tuned top-k against an assumed 128K and you're being served 32K, you are silently over-retrieving. And the failure is not an error — it's truncation. Your carefully reranked chunks get cut, usually from the front, and what reaches the model is a prefix of what you assembled.

Why this is nastier in RAG than in chat

In chat, losing early context degrades gracefully — the conversation just gets a bit dumber. In RAG it degrades misleadingly:

• The chunk that actually answered the question may be the one that got truncated, so the model answers from the remaining chunks and sounds confident.
• Evaluation then attributes the miss to retrieval quality. You go tune the embedding model, the chunker, the reranker — none of which was the problem.
• It's load-bearing on reranking specifically: reranking is only worth the latency if the top results actually survive into the prompt. If you're truncating, a better reranker buys you nothing.

Worse, it's provider-dependent. The same index and the same query can behave differently across two providers serving the same model, which reads as non-determinism rather than as a configuration difference.

What I'd like to know

1. Does LlamaIndex have a way to assert or discover the real served context for the LLM in use, rather than reading context_window from static model metadata? If a node postprocessor or the token counter is already doing something smarter here, I'd like to be pointed at it.

2. Is there a recommended pattern for failing loudly instead of truncating silently? What I want is for an assembled prompt that exceeds the real ceiling to raise, not get quietly cut — even at the cost of a failed query. A silently truncated RAG answer is worse than no answer, because it looks fine.

3. Has anyone benchmarked retrieval quality across providers holding the index constant? If the served-ceiling effect is as large as I think, provider choice should show up as a retrieval-quality difference, which would be a slightly alarming result worth having.

Practical discovery method in the meantime, in case it's useful to anyone: send a deliberately oversized prompt and read the error text — providers usually leak the true maximum there. Their docs have not been reliable for this.

[ducanhnguyen223]

This matches the boundary I would treat as unsafe for RAG: LlamaIndex can budget against configured metadata, but it cannot discover a provider’s effective KV-cache/concurrency ceiling before the request. PromptHelper/Settings.context_window are planning limits, not an attestation from the serving endpoint.

A practical fail-closed pattern is:

1. Set context_window to a measured/configured effective ceiling per deployment, not the model-card maximum.
2. Use the tokenizer for the actual model (LlamaIndex notes that its global tokenizer affects counting, chunking, and prompting).
3. Count the fully rendered prompt/messages, reserve max_output_tokens, and reject before the LLM call when input + reserved_output > effective_ceiling. Log the ceiling source and tokenizer version with the retrieval run.
4. Keep provider errors as a second guard, not as the normal discovery mechanism. If the provider rejects a request, record the observed limit but do not silently retry with truncated context.

That makes the retrieval test separable from serving limits: hold the index/query constant, vary only the configured ceiling/provider, and report (a) retrieved evidence, (b) rendered-token count, (c) accepted/rejected, and (d) whether the answer used every cited node. TokenCountingHandler is useful for instrumentation, but the preflight check must run after template/message rendering.

References: https://docs.llamaindex.ai/en/stable/api_reference/service_context/prompt_helper/ and https://docs.llamaindex.ai/en/latest/module_guides/models/llms/usage_custom/

[behrnt-slatgng]

Agreed, "planning limits, not an attestation" is the right way to put it. The last metric you list, whether the answer actually used every cited node, is one I hadn't thought to track. It is also the one that would catch the case I care about most: retrieval looks fine, the prompt fits, and the answer still quietly ignores half of the evidence.

Quick question: does LlamaIndex run any preflight check today where input + reserved_output > context_window raises instead of truncating, or is that entirely user code for now? If it's user code, a documented pattern next to PromptHelper would probably save people a lot of the debugging I went through.