Over your own documents, silent truncation is indistinguishable from an absence in your data

[behrnt-slatgng]

Disclosure up front: I build Grunz, a hosted chat + coding agent on open-weight models. Nothing to sell — a "second brain" over your own documents has the most expensive version of a failure I lost months to, and it seemed worth flagging.

Silent truncation is worse over a personal knowledge base than anywhere else

The context window on a model card is a property of the weights. The window you actually get is whatever the provider configured — most hosted endpoints serve around 32K regardless of what the card claims. It is rarely documented and exceeding it does not error. The request is served against whichever ceiling is lowest and the front of the prompt is silently gone.

Everywhere this happens it is bad. Over your own documents it is worse, for one specific reason: you cannot tell.

If you ask a general model something and the answer is thin, you have priors — you know roughly what a good answer looks like. If you ask your own knowledge base something and the answer is thin, the natural conclusion is "I guess I never wrote that down." The failure is indistinguishable from an absence in your data.

So the tool quietly teaches you that your second brain does not contain things it does contain.

Three consequences

It corrupts trust in the index, not the model. People conclude their ingestion failed, or their notes are worse than they thought. They go re-upload documents that were already there.

It hits the largest, most valuable documents hardest. The long contract, the full research paper, the meeting transcript — exactly the things you most wanted a second brain for are the ones most likely to be cut.

Retrieval tuning becomes untrustworthy. If you tuned top-k against an assumed window, you are over-retrieving, the chunk that actually answered gets cut, and you go tune the embedding model. Reranking in particular buys nothing if the top results do not survive into the prompt.

What would help

A visible signal when the assembled prompt did not fit. Even "answered using 6 of 11 retrieved chunks" would convert a silent wrong answer into an honest partial one. For a product whose value is trust in your own data, that seems worth more than the tidier UI.

Deliberate dropping rather than provider-side cutting. Dropping chunk 11 on purpose is much better than losing the first half of chunk 1 by accident, and it is the difference between a ranked decision and a byte-count.

Questions

1. Does Quivr know the effective served context for the configured model, or work from static metadata?

2. Is there a check that the assembled prompt fits before sending, or does it rely on the provider?

3. Has anyone compared answer quality for the same brain across model providers? If the ceiling effect is as big as I think, that shows up as a quality difference currently attributed to the model.

Discovery method meanwhile: send a deliberately oversized prompt and read the error — providers usually leak the true maximum there even when the docs do not.