llama tune - finding the best config for your setup

[am17an]

llama.cpp has a bunch of features which are disabled by default. Some of them are

1. -sm tensor
2. spec decoding
3. backend sampling
4. MoE cache
5. ..?

These easily can provide 2-3x performance uplift. I would like to suggest a tool (which at this point I don't how it works), but would automagically "tune" your setup given a particular GGUF file. As a simple example if the GGUF file has an MTP head, it can suggest running with spec-type draft-mtp, another more complex example is setting the correct values for --moe-cache-mib on a hybrid VRAM + RAM setup. The idea would be to provide approximately the best performance out-of-the-box for a llama.cpp installation.

@angt what do you think? Are you guys already working on something like this?

[angt]

I think this is a very good idea. Recently I’ve been thinking about integrating the performance tests from test-backend-ops into llama-app. People could simply install llama and run something like llama bench (without -hf) to produce a detailed performance report, without having to build the tests or download models.

We could cover operations and small graphs with shapes, quantizations, batch sizes and context lengths representative of real models. This would already make it much easier to investigate performance issues, compare backends and collect useful measurements across different hardware. It’s still far from automatic tuning, but it would provide a useful foundation.

Also, maybe we could add, as a first iteration, llama moe-cache-params ? Similar to llama fit-params.

[am17an]

Profiling via GGUF is better IMO as it directly optimizes the use-case. For e.g. moe-cache-params only makes sense for a MoE model, as well as adding MTP heads only makes sense if the hf repo has a MTP head (similar to how mmproj gets installed).

[angt]

Yes, definitely. I was just mentioning the integration of test-backend-ops (or something similar) that could be used to directly benchmark the machine once we know the shape of the tensors (so the GGUF the user want to use).

[am17an]

I'm wondering what a v1 we can ship looks like. Probably something in tools/tune which can run a couple of pre-defined configurations?

[taronaeo]

I think it would be useful if we can use this to tune a model deployment configuration, and generate a model.ini file for them to reuse. For example, auto-tuning -b and -ub according to their setup.

[arch-btw]

Excellent idea! Maybe it would also be possible to have adjunct documentation for some of these features, because sometimes features get added and the only way to really understand it is by locating the pull request, and even that can be difficult to understand.