Truncation eats your sources, not your question: a well-cited wrong answer

[behrnt-slatgng]

Disclosure first: I build Grunz, a hosted chat + coding agent on open-weight models. Nothing to sell — Perplexica supports local and hosted models, and there is a silent failure at that boundary that I lost a lot of time to.

Search results are context, and context gets truncated without telling you

The advertised context window on a model card is a property of the weights. The window you actually get is whatever the operator configured — a KV-cache budget traded against concurrency.

Most hosted endpoints serve around 32K regardless of what the card claims. A few reach 256K. I have not found one actually serving the 1M figures in release posts. It is rarely documented, and there is no error when you exceed it. The request is served against whichever ceiling is lowest and the front of the prompt is gone.

Why this hurts a search assistant specifically

Scraped pages are enormous and mostly boilerplate. A handful of results blows past 32K easily. So the truncation case is not an edge case here, it is the normal path for any query that pulls several substantial sources.

Truncation usually eats the sources, not the question. The question is near the end of the assembled prompt; the retrieved content sits in front of it. So what gets cut is precisely the evidence, and the model answers from whatever survived — confidently, and with citations to the sources that happened to fit.

That produces the worst failure mode a search tool has: a well-cited wrong answer. It looks more trustworthy than a normal hallucination because the citation list is real.

It is provider-dependent. The same query with the same sources can answer differently across two providers serving the same model, which reads as non-determinism rather than as a config difference.

The local-model point worth making

If you run the model locally you set the context length and you know it. That is a genuine advantage over any hosted endpoint and it is underrated.

Questions

1. Does Perplexica know the effective served context for the configured model, or does it work from static metadata? If there is already something smarter here I would like to be pointed at it.

2. Is there a sensible place to fail loudly rather than truncate — a check that the assembled prompt fits before sending, and either drops whole sources deliberately or errors? Deliberately dropping the 5th source is much better than silently losing the first half of source 1, and the user could even be told.

3. Has anyone compared answer quality for the same query across providers holding the sources constant? If the ceiling effect is as large as I suspect, provider choice should be showing up as an answer-quality difference that is currently attributed to model quality.

Discovery method meanwhile: send a deliberately oversized prompt and read the error text — providers usually leak the true maximum there even when the docs do not.