Repository navigation
Replies: 2 comments
|
The published Two checks that settle it: pull the UD-IQ4_XS revision from before that commit and compare against your build. If you match the old revision, the imatrix belongs to the old quants. Also look at which of the ~494 mismatched tensors differ. If they're all I-quant types, it's the imatrix. If plain Q8_0/Q4_K tensors differ too, it's a llama.cpp version difference, quant output isn't bit stable across commits. — |
|
Yes, I checked single tensors from the repo files with stock ggml (llama.cpp
That split matches your counts exactly. The 494 are the attention projections (120), routed experts (144), So on your second question: yes, something changes the weights of those four groups before they are quantized. It isn't an upstream update, since the Qwen/Qwen3.8-Flash-Next safetensors haven't changed since Aug 26. What that step is, only the Unsloth team can say. |
Uh oh!
There was an error while loading. Please reload this page.
Not a feature request -- a factual question about one published file, with the measurements that led to it.
Short version: quantizing the
BF16/shards ofunsloth/Qwen3.8-Flash-Next-GGUFwith stockllama-quantize, the exact per-tensor type table of UD-IQ4_XS, andimatrix_unsloth.gguffrom that same repo, I get a file whose tensors are 730/1224 byte-identical to yours. The other 494 differ, and they are exactly the ones whose quantization consults the imatrix.So: is
imatrix_unsloth.ggufthe file that was used for UD-IQ4_XS?Detail
The type table was read out of the quantized file itself (1224 entries, zero differences).
Identical:
token_embd,output, the 28.8 GBper_layer_token_embd, all norms -- none of which consult the imatrix.Different:
ffn_gate_exps,ffn_up_exps,ffn_down_exps,attn_qkv,hc_*,ssm_out-- all of which do.Two controls:
The difference is not quantization quality
Measuring reconstruction error against the BF16 source, on the 13 layers where both files use IQ4_NL:
At the same type the two files are equally close to the source.
What I am asking
Is
imatrix_unsloth.ggufin that repo the file used for UD-IQ4_XS? The metadata names it, but if the calibration set is regenerated over time a same-named file could have been replaced after this quant was built. If the one actually used can be shared, I would like to reproduce the result properly.Independently: does the release pipeline include any step beyond
llama-quantizewith a type map plus an imatrix -- a per-group sensitivity pass, for instance? I am not asking for the recipe, only whether such a step exists, since that would tell me to stop guessing at it.Context, same corpus,
-c 512 --chunks 50:Thanks -- these quants are what I run daily on an RX 7900 XTX.
All reactions