Make an --autotune flag for available model, hardware *and* user priorities?

[dagbdagb]

Using --fit on with no --ctx-size length on a particular setup gave me a context length of 4.1k.

Fair. Works as intended, I guess. Performs great. For a short while. But it got me thinking:

How hard would it be to make llama-server suggest a (starting point for) a suitable set of parameters taking into account the:

• user's priorities (ordered list)
• user's minimum viable values for the non-primary priorities
• hardware visible to llama-server
• the model file in question

Examples:

• user A wants the highest possible output quality given the model in question, but tg-speed must be above some threshold T, and context length must be minimum C. (no KV-cache quantization etc.)
• user B wants the longest possible context C, but tg-speed T has no importance (offload to NVME if needed etc.)
• user C wants the highest possible tg-speed, then quality and context length, in that particular order. (try to keep everything on GPU etc.)

Some priorites rule out using mmap, others suggest KV-cache quantization is OK. And so on.

A limited list of intended use-cases may be required to make even more useful suggestions:
single-user-chat
single-user-agentic
multi-user-chat
multi-user-agentic