Your retrieval budget may be based on a context window you are not actually being served

[behrnt-slatgng]

Disclosure up front: I build Grunz, a hosted chat + coding agent on open-weight models. Not pitching anything — this is a retrieval-sizing problem I keep hitting and I suspect it silently affects more Haystack pipelines than people realise.

Your retrieval budget is probably based on a number you are not actually being served

Splitter length, top_k, and reranking are all tuned against an assumed context budget. That budget almost always comes off the model card.

The model card number is a property of the weights. The number you actually receive is a property of whoever serves them — a KV-cache budget traded against concurrency. Across the hosted endpoints I have tested, most serve around 32K regardless of what the card claims. A few reach 256K. I have not found one actually serving the 1M figures in release posts.

If you tuned top_k against an assumed 128K and you are being served 32K, you are over-retrieving. And the failure is not an error — it is truncation.

Why this is nastier in RAG than in chat

In chat, losing early context degrades gracefully. In RAG it degrades misleadingly:

• The document that actually answered the question may be the one that got cut, so the model answers from what survived and sounds confident doing it.
• Evaluation then attributes the miss to retrieval quality. You go tune the embedding model, the splitter, the reranker — none of which was the problem.
• It is load-bearing on reranking specifically. Reranking is only worth the latency if the top results survive into the prompt. If you are truncating, a better reranker buys you nothing.

It is also provider-dependent, so the same index and the same query can behave differently across two providers serving the same model. That reads as non-determinism rather than as a configuration difference.

Questions

1. Does Haystack expose the effective served context for the generator in use, as opposed to a static value from model metadata? If a component or the token counter is already doing something smarter here I would like to be pointed at it.

2. Is there a pattern for failing loudly rather than truncating silently? What I want is for an assembled prompt exceeding the real ceiling to raise, not get quietly cut, even at the cost of a failed query. A silently truncated RAG answer is worse than no answer because it looks fine.

3. Has anyone benchmarked retrieval quality across providers holding the index constant? If the served-ceiling effect is as large as I think, provider choice should show up as a retrieval-quality difference, which is a slightly alarming result worth having.

Discovery method in the meantime, in case it helps anyone: send a deliberately oversized prompt and read the error text — providers usually leak the true maximum there even when the docs do not.

[ducanhnguyen223]

Haystack currently gives you the pieces for a fail-closed guard, but not provider-side context attestation:

• PromptBuilder / ChatPromptBuilder render the final prompt/messages; they do not discover the serving endpoint's effective KV-cache ceiling or enforce it.
• The token-counter APIs can measure the rendered input (including tool schemas where supported). Use the same model/provider counter as the generator; an approximate counter is useful as a fallback.
• Generator meta token usage is post-call telemetry, so it is too late to prevent silent truncation.

A practical pipeline boundary is:

1. Render the complete prompt/messages.
2. Count that rendered request with a model-specific counter.
3. Compare input_tokens + reserved_output_tokens with a configured effective ceiling per deployment (not the model-card maximum).
4. Raise/route to a visible context_budget_exceeded outcome before calling the generator.
5. Record the ceiling source, tokenizer/counter version, rendered count, reserved output, and selected document IDs.

Keep provider rejection as a second guard and record the observed limit, but do not retry by silently dropping context. This also makes cross-provider retrieval tests separable: hold the index/query fixed and report retrieved evidence, rendered tokens, accepted/rejected, and answer citations.

Relevant docs:

• https://docs.haystack.deepset.ai/docs/promptbuilder
• https://docs.haystack.deepset.ai/docs/chatpromptbuilder
• https://docs.haystack.deepset.ai/docs/3.1/token-counters

[behrnt-slatgng]

Thanks, that's a clear split: the builders render the prompt, the counters measure it, and meta usage only arrives after the call, when it's too late. The five-step boundary is basically the guard I wish I'd had from the start.

One question about where it should live. Is there an idiomatic place in Haystack to declare the effective ceiling per deployment (a generator init parameter, or a separate component in front of the generator)? And would a small component that counts the rendered messages and raises a context_budget_exceeded outcome be welcome upstream, or is that something you'd rather leave to user code?