Repository navigation
|
Ive recently setup Windmill and am greatly enjoying it (setup via docker compose). I have llama.cpp and openrouter set and have been using both successfully. However, I seem unable to modify the context token size. It defaults to 128k (assumed) which is quite low for some work, which then forces an auto compaction when getting close to that. This may be user error, but is it possible to modify this 128k value to something higher? I have tried opening chats with different models with different context sizes and the default stays the same. I also removed the default model in case that was impacting things. This seems simple, so presume I am missing something. I believe I have exhausted the public documentation as well. Somebody please show me the way. |
Replies: 1 comment 1 reply
|
I think the 128k number you're seeing is Windmill's conservative fallback for models whose context window is not recognized by the frontend, not a Docker Compose setting. In the current code this is in export function getKnownModelContextWindow(model: string): number | undefined {
return matchModel(MODEL_CONTEXT_WINDOW_MATCHERS, model)
}
export function getModelContextWindow(model: string) {
// Trim/compaction logic needs a number; assume a conservative window when unknown.
return getKnownModelContextWindow(model) ?? 128000
}So changing providers/models only helps if the model id matches one of the known entries in Practical options:
So short answer: I don't think there is currently a Compose env var to just set the AI chat context window globally; 128k is the unknown-model fallback. |
Ah ok, I think I get it now. All of my openrouter models I have setup in Windmill use the
{provider}/{model}-latestpattern. Those all seem to be unknowable by Windmill all defaulting to 128k. When I set a model on an actual version, not using-latest, it seems Windmill can actually know the context window. I assumed it was something simple. Thank you. Im not sure id suggest this should or could become an issue, or if it is even possible for windmill to fetch the actual model being used under the hood in thelatestflag. Perhaps a brief note somewhere in the AI section of the docs may help a small number of people like me, as im sure I wont be the last to wonder this. But regardless, thanks