Whats the best quantization type for my setup?

[johnlovesgoats]

trying out ikllama for the first time, what is the best quant type to use for someone that has:

intel i5 14600kf
rtx 3060 12gb
64gb ram

probably would be offloading models to gpu and cpu

[magikRUKKOLA]

Its not about the quant type. Its about the number of the parameters the model has.

IQ2_KL 43.319 GiB (3.047 BPW)
https://huggingface.co/ubergarm/Qwen3.5-122B-A10B-GGUF#iq2_kl-43319-gib-3047-bpw

smol-IQ3_KS 30.728 GiB (3.313 BPW)
https://huggingface.co/ubergarm/Qwen3-Coder-Next-GGUF#smol-iq3_ks-30728-gib-3313-bpw

probably would be offloading models to gpu and cpu

Yeah, there is no other option.

[Khots0]

Its not about the quant type. Its about the number of the parameters the model has.

I used to be laser‑focused on parameter count too, but the more I test different models, the more obvious it becomes that architecture matters far more than raw numbers.

A good example is Minimax vs. StepFun. On paper, their parameter counts suggest they should have similar inference speeds. In practice, they behave completely differently. Minimax tends to choke my dual‑Xeon, no‑GPU setup pretty quickly—starts strong at ~8 t/s, then drops off hard. StepFun, on the other hand, just flies. It stays fast and stable even with large context windows.

So yeah, parameter count is only one piece of the puzzle. Architecture and implementation details can completely change real‑world performance.

[magikRUKKOLA]

I used to be laser‑focused on parameter count too, but the more I test different models, the more obvious it becomes that architecture matters far more than raw numbers.

Yeah, that's why I recommended Qwen3.5. Its about the half of the size of DeepSeek-R1 (both in total parameters and in active parameters). But it delivers pretty good quality. Moreover, it has the graph parallel support and a nice KV-cache in general. So it runs 20 tps with DDR4 [(8 channel 2933 MT/s)] and some VRAM (about two times higher than the deepseek). Moreover, its possible to fully offload the Qwen3.5 (397B) to the 8x3090. The decode speed is about 50 tps. Its very comfortable speed for the local inference.

So yeah, parameter count is only one piece of the puzzle. Architecture and implementation details can completely change real‑world performance.

Of course. But the topic starter came about with two channel DDR4 and 16 GB GDDR6X. This setup is not for the LLM inference, its for the gaming.

[magikRUKKOLA]

@Khots0

Check this out -- this is nuts!

(the latest (coloured black) is the current main branch)

[FNsi]

For speed, unsloth's quants are usually faster in cpu. Accuracy can be compromised in somehow, I just cannot enough model slower than 12 tg/s in my cpu only inference, especially some models like overthinking, have to adjust base on prompts. Personally I like Qwen-Coder-Next, it has been a long time I didn't play local llm, (last immersive run is like llama2 era, discovered NTK in Reddit, amazing 😂) this 80b model is definitely my choice, better than free models in cline.