Repository navigation
Served context ceilings are per-endpoint and undocumented, which makes them a per-role problem in model packs #370
behrnt-slatgng
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Disclosure: I build Grunz, a chat and coding agent on open weights. Adjacent tool, so read this with that in mind — but the finding below is a property of providers rather than of any product, and Plandex is one of the few projects where acting on it is straightforward.
The number on the model card is not the number you get
The context window advertised on a model card is a property of the weights. The window you are actually served is whatever your provider configured for that endpoint. In practice most hosted endpoints serve somewhere around 32K regardless of what the card claims. A few reach 256K. I have not yet found one genuinely serving the 1M figures that get quoted.
Crucially it fails silently. No 413, no warning, no truncation notice — the front of the context is evicted and the model answers confidently from what is left.
This matters more for Plandex than for most tools, for two reasons.
It undercuts the thing you are actually selling. Plandex's pitch is handling large context across many files properly. If a user points a role at a provider quietly serving 32K while the model card says 200K, your chunking maths is correct and the endpoint still throws away the front of it. The user experiences that as Plandex losing files, and there is no error anywhere to tell them otherwise.
Model packs make it per-role, not per-project. Because roles can point at different providers, the effective ceiling varies within a single plan. A planner on a genuinely long endpoint feeding a builder on a silently-32K one will produce exactly the class of bug that is miserable to diagnose: the plan is coherent, the build drops context, and nothing in either log says why.
Things that would help users
plandex modelsnext to the declared window. "declared 200K / observed 32K" would save people a lot of confused debugging.The related one
When the window does fill and history is compacted, an unpinned goal gets summarised into vagueness — and the model then reads back its own compacted note, takes it as the plan, and quietly restarts or drops a constraint. Pinning the goal statement verbatim through every compaction, never paraphrased, is a small change that prevents a whole category of "it forgot what I asked" reports. Logging compaction as a real event with what was dropped and kept is the other half, because otherwise there is no record of the moment a long run actually went wrong.
And for anyone here running open weights: ablation costs instruction-following and output-format adherence before it costs knowledge, so tool-call syntax drifts while prose still looks fine. Budget for it if you are parsing output.
Happy to go into detail on any of it.
All reactions