FiTuna: pick a quant and -ngl by measuring on your own machine, not by guessing

[leeyunseokarchive]

I kept defaulting to Q8_0 and then nudging -ngl by hand until the speed felt acceptable, with no idea whether a lighter config would have cleared the same bar. So I wrote a CLI that measures it instead of guessing.

You give it a model, a target tok/s, and a quality budget in percent. It quantizes each candidate, measures perplexity loss against the F16 baseline, benchmarks the survivors in measured-quality order, then binary-searches the minimum -ngl that still hits the target. Output is the lightest config that actually passed on that machine, plus the llama-server command to run it.

Measured on an Apple M3 Pro, llama.cpp b9960:

Model	Target	Q8_0	What passed
Qwen3-4B-Instruct	30 tok/s, 5% loss	24.22 tok/s, failed	Q4_K_M @ ngl=33, 30.81 tok/s, 1.73%
SmolLM2-135M	240 tok/s, 5% loss	205.91 tok/s, failed	Q6_K, 249.50 tok/s, 0.53%
Midm-2.0-Mini (KR)	40 tok/s, 5% loss	34.26 tok/s, failed	Q4_K_M @ ngl=48, 44.62 tok/s, 2.58%

Q8_0 was my default before this and it missed the target on all three.

Two results I can't explain: on SmolLM2 the smaller Q4_K_M measured slower than Q6_K, and on Qwen3-4B Q8_0 came out slightly worse than Q6_K on perplexity. Both held on re-runs.

Limitations, since they matter more than the table:

• Results only apply to the machine that produced them. No extrapolation, by design.
• Single GPU only. No --tensor-split.
• Quality is perplexity on a corpus you pick, which is a proxy. Switching that corpus from English to Korean flipped one verdict.
• Benchmarks drift with temperature. I measured about ±1.7 tok/s run to run at a 30 tok/s baseline, so verdicts that close are marginal.
• Real hardware testing is macOS and Linux only. Windows has CI and unit tests, nothing more.

MIT, no runtime dependencies, pip install fituna. Runs in about two minutes on a 258MB model.

https://github.com/leeyunseokarchive/fituna

I'm an undergrad and this is the first thing I've built in this space, so if the approach itself is wrong I'd rather hear it now.