Feature request: Configurable context size

[jeberger]

The pool cli assumes that the model has a 125k context window, but that may not be true (especially when using a locally hosted Ollama model). It would be nice if:

• we could specify a context size, and have pool tell Ollama to use that size
• failing that, it would be nice if pool could auto-detect the context size used by the Ollama model
• failing that, it would be nice if we could specify the context size that pool should use and have it use that context size when reporting usage and auto-compacting the context when it overflows.

[agentcooper]

@jeberger We support POOLSIDE_STANDALONE_CONTEXT_LENGTH variable:

POOLSIDE_STANDALONE_BASE_URL="http://127.0.0.1:11434" \
POOLSIDE_API_KEY="EMPTY" \
POOLSIDE_STANDALONE_MODEL="laguna-xs.2:latest" \
POOLSIDE_STANDALONE_CONTEXT_LENGTH=500000 \
pool

I've also added it to the docs.

Does this work for you?

[jeberger]

That works for me, thanks