Context size / KV cache overhead in fit calculations

[g-rock84]

First off, thank you so much for llmfit! It’s an incredibly handy tool!
​
I had a feature idea while using it. ​Would it be feasible to add an adjustable context size filter when determining the models that will fit on a system? I know this is included in the model details, but instead having to go through the details for every model, they could be filtered down even further to the largest models that will fit your system with a given context size.

[AlexsJones]

Thanks, glad it's useful, and this is a fair ask.

Half of it exists today on the CLI. llmfit --max-context 8192 fit re-scores every model's memory against that context instead of its advertised maximum, so the table you get back already is "what fits at 8K". The same flag works on recommend, and the REST API takes max_context per request (plus sort=ctx if you want the list ordered by context rather than score).

The half that doesn't exist is what you're actually describing, which is doing that from inside the TUI. Right now the context cap is read once at startup and there's no key to change it live, so you can't sweep 4K / 8K / 32K and watch the list move. That's the feature, and it's a good one.

Worth opening as an issue so it doesn't get lost in Ideas. I'd frame it as "adjustable context cap in the TUI, re-scoring the list live", since the underlying calculation is already there and it's the control that's missing.

🤝 Handled by Alex's Repo Steward — replies reviewed by @AlexsJones.

Learn more or run your own: https://github.com/AlexsJones/repo-steward

[AlexsJones]

Good question. Context length belongs in the fit calculation because KV cache grows with it, but it should be an explicit assumption rather than a hidden filter. Which command are you using today, and do you need a one-off --context-size override or a saved default for a workload?

[g-rock84]

I currently use llmfit --max-context and it works. I was referring to the TUI though.

Both would be awesome if I'm being a choosey beggar diva. However, the one-off --context-size option would be more than sufficient. Thanks so much for your time!