How does it choose model settings - project goal changed?

[black-snow]

So, I haven't looked at the code yet and I wonder:

The README has been updated and no longer claims to configure optimal settings, which is sad and one reason I started using it in the first place.

I originally wanted to ask how it chooses optimal settings and that this doesn't seem to work well (doesn't configure draft models or cache quants or even num layers on GPU). But now I wonder if the goal of this project has changed @erusev

Unsloth studio does an okay job automatically configuring the settings.

I think it's quite difficult yet a very important feature. If I get 200% the token/sec with another software then I won't use the barn. The only reason now to use llama would be to have an okay setup to research and configure a better raw llama.cpp setup, tbh.

[erusev]

The readme edit was just copy cleanup -- optimal configuration is still a key goal of the app.

Not everything is covered yet -- cache quantization, for one. If you're seeing a real tokens/sec gap on the same model, I'd be curious which settings make the difference.

[black-snow]

Glad to hear that!

Well, I'm too lazy to fully benchmark my setup - which is why I'd want a software to set things up near-optimally for me.

Currently, on my M3 36GB, I run unsloth 4-bit quants for Qwen 27B and the MoE with 8-bit KV quants and 128k context, I think, flash attention on, all layers in (unified) mem, MTP on with 3 tokens predicted - there are some more settings I don't have in my head right now. Similar setup for gemma.
Probably not optimal but better than the llama loadout, even though some of nice things don't really do much on my MBP.

[erusev]

Follow-up: 0.41.0 lets you set these yourself in ~/.config/llama/models.user.ini -- cache-type-k, spec-*, anything llama serve takes, per model (readme).

Not the automatic configuration you're after, but if you find settings that beat our defaults on your Mac, that's exactly what I'd like to hear.

[black-snow]

Thanks a lot for taking the time to write the update @erusev (and for the change itself too, ofc). Now I can go back from bare llama-server to the app.