Question about Global Model Context Length

[CutWire22]

Hey I got a quick question regarding anything LLM regarding the context length maximum If I have a model that's usually a lot bigger context window for one workflow but the global model is capped at a much smaller context window would that affect how it works on the actual main big workflow? Example. Main Global context model is set to 45k. The workflow B using a different model have context window of 128k. Would the 128k model become botched by the 45k global limit?
I ask this because I was trying to import some files and document and got hit by "This will exceed the 45k limit" warning.

[Dv2Corn]

It depends on the provider of workspace B. When workspace B uses the same provider as the global model (Ollama, LM Studio, LocalAI, Generic OpenAI and other local or self-hosted providers), the answer is yes: the 45k cap applies to its 128k model too.

AnythingLLM gets a workspace's context window from that workspace's provider and model (Workspace._getContextWindow). For these providers, though, the "Model context window" / token limit field on the LLM Preference page is stored once per provider (for Ollama it is OLLAMA_MODEL_TOKEN_LIMIT), not once per model. The window a workspace ends up with is the smaller of that value and the model's own window (Ollama promptWindowLimit):

effective window = min(global token limit, the model's own context length)

So with 45k set globally, the 128k model in workspace B runs with 45k. The cap is not only a UI warning. For Ollama the same number is sent as num_ctx, and the history, system prompt and context limits are worked out from it too. The upload warning you saw uses that number: it fires once attached files would go over 80% of the window (maxContextWindowLimit: 0.8).

To fix it:

• Raise the value (to 128000, for example) in Settings → AI Providers → LLM → Show advanced settings → Model context window, or clear the field. Models with a smaller window are still capped at their own size, because the effective window is the minimum of the two. For Ollama, an empty field means auto-detect, but auto-detect caps at 16,384, so enter the number yourself if you want more than that.
• Exception: Generic OpenAI uses the value as-is and does not compare it with the model's own size. Do not set it higher than your smallest model can take.
• If workspace B uses a different provider (a cloud one like OpenAI or Anthropic, for example), the global 45k field does not apply. That workspace uses the known window of the model it picked.

Keep in mind that a larger context window takes more RAM or VRAM on a local runner.

[timothycarambat]

Answer above is correct, do you need it more dynamic like per workspace?

[CutWire22]

Honestly, it would be cooler/better if each workspace allowed for both context and also what tool is allowed. (along with how many tool calls).
Btw, hey tim. it's me the guy who is always active on the server.
We wish for more workspace to workspace dependent settings.

[timothycarambat]

I agree as well - we are working on #6638 for the workspace specific skills, but yes also need to extend this functionality for providers as well for their specific configs!

[CutWire22]

slightly off topic but I assume this github section you view more often? I always try to get to you on discord by direct ping. I was the one who mentioned the idea of installing a simple honeypot to auto ban the account that spams. (aka "don't post in this channel"). Because these spam bots does it indiscriminately (and the longer you keep them around the more people get infected). the same way these bots spam is what makes them easy to counter.

[timothycarambat]

Yeah, since i also respond to every GH issue and also the other dev is also on GH 24/7 this makes it easier for us both to manage in a place we already are working + allows higher visibility for discussions/issues to be done via contributors and stay OSS!

[abernicht]

context window per workspace is one of the things i miss most.