Quantization vs Best Quant?

[hugging-cat]

In particular, why is the Quantization field Q4_K_M for everything?

(FYI I know what quantization is 😄 )

[AlexsJones]

Good question — the two fields answer different questions:

• Quantization is the catalog default for that model entry (fit.model.quantization). The curated database records each model at a reference quant, and that reference is Q4_K_M for nearly everything because it's the community-standard sweet spot GGUF releases are published around — which is why you see the same value all the way down the column. It's a property of the catalog entry, not of your machine.
• Best Quant is computed for your hardware: the highest-fidelity quantization whose memory footprint still fits your VRAM/RAM budget for that model. That's the number llmfit actually recommends — the detail pane shows both together as Quant: <best> (default <catalog>).

So "Q4_K_M everywhere" isn't saying those models only come in Q4_K_M — it's the shared baseline the fit math starts from, and Best Quant is where it lands for you.

Fair point that the bare label "Quantization" reads ambiguously, though — something like "Catalog default" would make the distinction self-explanatory. If that would have saved you the question, an issue suggesting the relabel would be a welcome (and easy) contribution.

🤝 Handled by Alex's Repo Steward — replies reviewed by @AlexsJones.

Learn more or run your own: https://github.com/AlexsJones/repo-steward

[AlexsJones]

Short answer: because it's a default the scraper writes, not a measured property of the model — and my earlier reply on this thread overstated it. Correcting that.

I said the catalog records each model "at a reference quant... the quant GGUF releases are published around." That's not what the code does. _detect_format_from_name in scripts/scrape_hf_models.py ends at line 930 with:

return ("gguf", "Q4_K_M")

Detection (scrape_hf_models.py:856-910) only produces a different label when the repo explicitly advertises a pre-quantized format — AWQ, GPTQ, AutoRound, compressed-tensors — via config.json's quantization_config.quant_method, or a -AWQ/-GPTQ/-AUTOROUND name suffix. Every ordinary unquantized safetensors repo falls through to that last line. In the shipped catalog that's 5165 of 5744 entries (~90%) labelled Q4_K_M. Nobody checked what those models are actually published as.

So the column isn't telling you the model is Q4_K_M. It's telling you llmfit assumed Q4_K_M to do its RAM/VRAM math. MODELS.md:3 says so directly: "All memory estimates assume Q4_K_M quantization (0.5 bytes per parameter) unless noted otherwise." The scraper derives min_ram_gb/recommended_ram_gb/min_vram_gb from that assumption (scrape_hf_models.py:975-976).

Best Quant is the field that's real. fit.rs:535-566 walks a fixed hierarchy (QUANT_HIERARCHY, models.rs:5 — Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, Q2_K) best-quality-first and returns the first whose estimated footprint fits your available memory (models.rs:913-937), retrying once at half context before giving up. Pre-quantized models skip selection entirely and reuse the catalog value (fit.rs:536) — that's why those rows show the same thing twice.

You've found a real problem, though, and it's worse than the label. Your instinct that "Q4_K_M for everything" is suspicious is correct, and there are three things here worth fixing separately:

1. The label is wrong, not just ambiguous. "Quantization" implies a fact about the model. It's an assumption. "Assumed quant" or "Sizing basis" would be honest; renaming it to "Catalog default" (what I suggested last time) still implies the catalog knows something it doesn't.

2. Best Quant can recommend quants that don't exist. GgufSource (models.rs:682-687) stores only repo and provider — no quant list, no file sizes. Footprints are computed arithmetically as params_b() * quant_bpp(quant) (models.rs:762-763), never looked up. So llmfit will happily suggest Q6_K for a model with no GGUF release at all, and display.rs:280 formats a download hint from gguf_sources[0].repo + best_quant — a repo/quant combination nothing verified. The ONNX path gets this right: select_onnx_quantization (models.rs:1272-1284) picks among onnx_files, a real map of filename to byte size. GGUF needs the same.

3. Two tables disagree on what Q4_K_M costs. quant_bpp says 0.58 bytes/param (models.rs:14-38); quant_bytes_per_param says 0.5 (models.rs:66-87). The first sizes the model, the second estimates tok/s. Same quant, different assumption, ~16% apart.

(Also scrape_hf_models.py:928 labels MLX models Q4_K_M with a comment conceding MLX doesn't use GGUF quants.)

Issues for (2) and (3) especially would be welcome — those change recommendations, not just wording. If you'd rather just file the relabel and leave the rest to me, that's fine too.

🤝 Handled by Alex's Repo Steward — replies reviewed by @AlexsJones.

Learn more or run your own: https://github.com/AlexsJones/repo-steward