Local vs hosted for the same model can be two different systems: the served context gap

[behrnt-slatgng]

Disclosure up front: I build Grunz, a hosted chat + coding agent on open-weight models. Nothing to sell — Khoj supports local and hosted models side by side, which makes it one of the few places this particular gap is visible, and it seemed worth writing down.

The served context window is not the one on the model card

The advertised window is a property of the weights. The window you actually receive is a property of whoever serves them — a KV-cache budget traded against concurrency.

Across the hosted endpoints I have tested, most serve around 32K regardless of what the card claims. A few reach 256K. I have not found one actually serving the 1M figures in release posts.

It is almost never documented, and it fails silently. Nothing errors. The request is served against whichever ceiling is lowest and the front of the context is simply gone.

Why this is sharper in Khoj than in a single-provider app

You can switch between local and hosted for the same nominal model. A user running a 128K model locally and the "same" model through a hosted provider is quietly getting two different systems. Anything that behaves well locally and worse hosted will read as a Khoj bug, or as the provider being worse at inference, when the difference is a config number nobody exposed.

It breaks retrieval silently, not loudly. For a personal knowledge base this is the expensive version: the note that actually answered the question may be the one that got truncated, so the answer comes back confident and incomplete. You then assume the search is bad and go tune embeddings or chunking, which was never the problem. Reranking especially buys nothing if the top results do not survive into the prompt.

Long conversations degrade without a signal. The assistant starts contradicting things it established earlier, and the natural read is "the model is not very good," not "we are being served a quarter of the advertised window."

The local-wins point, which is worth saying plainly

Running locally, you set the context length and you know what it is. That certainty is a real advantage over any hosted endpoint and it is underrated — it is one of the few places where self-hosting gives you a hard guarantee rather than a preference.

Questions

1. Does Khoj surface the effective context for the configured model, as opposed to a value from static model metadata? If something already does this I would like to be pointed at it.

2. Is there a way to fail loudly instead of truncating silently — an assembled prompt exceeding the real ceiling raising rather than getting quietly cut? A silently truncated answer over a personal knowledge base is worse than no answer, because it looks right.

3. Has anyone compared answer quality for the same index across local and hosted serving of the same model? If the ceiling effect is as big as I think, that comparison should show a difference people are currently attributing to something else.

Discovery method in the meantime: send a deliberately oversized prompt and read the error — providers usually leak the true maximum there even when the docs do not.