Please expose the effective max_model_len in /v1/models — consumers currently discover it by breaking it

[behrnt-slatgng]

Disclosure: I build Grunz, a hosted chat and coding agent on open weights. I am a consumer of inference endpoints rather than an operator of one, so this is a request from the other side of the API — and OpenLLM is one of the few projects positioned to just fix it.

Consumers cannot discover the context window you are actually serving

From outside an endpoint there is currently no way to learn its real context limit except by breaking it.

The window on a model card is a property of the weights. The window an endpoint serves is whatever the operator configured — --max-model-len, the KV cache budget, whatever the GPU allowed. Those routinely disagree, and the disagreement is not exposed anywhere machine-readable.

In practice, across a lot of hosted endpoints, most serve around 32K regardless of what the model card advertises. A few reach 256K. I have not yet found one genuinely serving the 1M numbers that get quoted for models whose cards claim it.

The failure mode is what makes this expensive. Overrun the real limit and you do not reliably get a 413. Depending on the serving config you get silent eviction from the front of the context — and the front is where the system prompt and instructions live. So the application does not see an error, it sees the model quietly stop following its own instructions, several turns later, with no signal connecting cause to effect.

Right now the only reliable discovery method is to deliberately send an oversized request and parse the error text. That is a bad interface, and it is what everyone is doing.

What would fix it

1. Report the effective max_model_len in the OpenAI-compatible /v1/models response. The spec does not mandate a field, but nothing stops an additional one, and a de-facto convention would spread quickly if a serving framework shipped it. This is the single highest-value item: it turns a guess into a lookup.

2. Surface it in openllm models / the deployment output, so an operator can see what they are actually serving versus what the card claims. Operators are frequently as surprised as consumers are — the number often comes from memory fitting rather than from a decision.

3. Prefer an explicit error to silent eviction. If a request exceeds the served window, a clear 400 naming the real limit is far more useful than a truncated context and a confident answer. If silent rolling is the default somewhere in the stack, making that a documented, switchable behaviour rather than an implicit one would help a lot.

4. Document the interaction between quantization, KV cache and effective context. People choose a quant for quality reasons and then discover their usable window moved, which is not obvious to anyone who has not operated a server.

Why this matters beyond ergonomics

Benchmark numbers for anything agentic are per-provider, not per-model. The same weights served at 32K and at 200K are not the same agent — long runs die on context long before they die on cost. Every published comparison that does not pin the served ceiling is comparing two different things and reporting the difference as a property of the model.

A serving framework that reports its real ceiling makes that class of comparison possible. Right now nobody can do it honestly.

Happy to help test or contribute if any of this is wanted.