Eval bug: Diffusion Model (Dream 7B) Logits Not Matching to HF implementation

[wp4032]

Name and Version

❯ llama.cpp/build/bin/llama-cli --version
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 0.010 sec
ggml_metal_device_init: GPU name: Apple M3 Ultra
ggml_metal_device_init: GPU family: MTLGPUFamilyApple9 (1009)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal4 (5002)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 498216.21 MB
version: 6955 (6105e855)
built with Apple clang version 17.0.0 (clang-1700.3.19.1) for arm64-apple-darwin25.0.0

Operating systems

Mac

GGML backends

Metal

Hardware

Mac Studio M3 Ultra 512GB memory

Models

Dream 7B Base

Problem description & steps to reproduce

I am comparing the logits of the Dream 7B model implementation on HF v.s. llama.cpp on BF16, and there seems to be logit mismatches. For example, I pass in the exact same random tokens matching to both the HF and llama.cpp models. Here are examples of some input tokens I pass in with the mismatch errors:

I have checked Qwen3Moe-30B-A3B BF16 for logits mismatches and they are nearly identical.

Things I have checked

• GGUF files seem to be right
• Divergence across layers seems to be linear (will look more quantitatively)
• RMS Norm eps values are the same

I am doing this because I am trying to integrate another diffusion model, but I am getting logit mismatches there as well.

I am not sure if this is due to a backend issue with build_attn_inp_no_cache();

Happy to share over my code to check logits too

@am17an

First Bad Commit

No response

Relevant log output

--------------------------------
Tokens: 4
	Random tokens: tensor([124518,  75548,  87446, 101107])
	Mean Error: 0.12463831156492233
	Max Error: 0.5572605133056641
	Cosine Similarity: 0.9974273443222046
	Percentage Error: 85.55838012695312%
--------------------------------
Tokens: 5
	Random tokens: tensor([ 45758,  20906, 147022,  15555, 109180])
	Mean Error: 0.27511870861053467
	Max Error: 1.4357795715332031
	Cosine Similarity: 0.9923816919326782
	Percentage Error: 172.04852294921875%
--------------------------------
Tokens: 6
	Random tokens: tensor([ 75036,  32437,  87977,  94129, 116228,  30788])
	Mean Error: 0.15428200364112854
	Max Error: 0.6780109405517578
	Cosine Similarity: 0.9978612661361694
	Percentage Error: 6.23129415512085%
--------------------------------
Tokens: 7
	Random tokens: tensor([ 21585, 129721, 115606,  43139, 122473,  22167,  57813])
	Mean Error: 0.15736085176467896
	Max Error: 0.8004646301269531
	Cosine Similarity: 0.9974124431610107
	Percentage Error: 9.917763710021973%
--------------------------------

[am17an]

Can you share the exact command you are trying to run?

[am17an]

Also the implementation doesn't try to match bit by bit with the HF implementation. You should check the mean and std dev of the activations per layer

[pwilkin]

@wp4032 there are scripts in examples/model-conversion to run comparisons between original and converted models as well as dump tensors for intermediate states, try doing that and see where the divergences happen.

[wp4032]

@am17an here is the command:
llama.cpp/build/bin/llama-eval-callback --hf-repo Dream-v0-Base-7B --model Dream-v0-Base-7B.gguf --prompt "What is a GPU" --seed 1234 -ngl 999 -fa on --verbose

I modified the eval-callback-dump.cpp file to dump out the values in binary and then do a comparison with the HF torch tensors.

For the above prompt, I am getting small divergence, however, would this be significant?

--------------------------------
attn_norm-0
--------------------------------
torch.Size([1, 4, 3584])
attn_norm-0-MUL-norm-0-blk.0.attn_norm.weight{3584, 1, 1, 1}-
Initial shape: 14336
Mean Error: 2.928e-04
Max Error: 1.353e-02
Cosine Similarity: 1.00125
Percentage Error: 0.190%

--------------------------------
Qcur-0
--------------------------------
torch.Size([1, 4, 3584])
Qcur-0-ADD-node_3-blk.0.attn_q.bias{3584, 1, 1, 1}-
Initial shape: 14336
Mean Error: 1.637e-03
Max Error: 1.264e-01
Cosine Similarity: 1.00193
Percentage Error: 0.155%

--------------------------------
Kcur-0
--------------------------------
torch.Size([1, 4, 512])
Kcur-0-ADD-node_7-blk.0.attn_k.bias{512, 1, 1, 1}-
Initial shape: 2048
Mean Error: 9.952e-03
Max Error: 4.046e-01
Cosine Similarity: 0.99849
Percentage Error: 0.145%

--------------------------------
Vcur-0
--------------------------------
torch.Size([1, 4, 512])
Vcur-0-ADD-node_11-blk.0.attn_v.bias{512, 1, 1, 1}-
Initial shape: 2048
Mean Error: 2.910e-04
Max Error: 7.510e-03
Cosine Similarity: 1.00240
Percentage Error: 0.231%

--------------------------------
Qcur_rot-0
--------------------------------
torch.Size([1, 4, 28, 128])
Qcur-0-ROPE-Qcur-0 (reshaped)-Metal#leaf_5#0{4, 1, 1, 1}-
Initial shape: 14336
Mean Error: 1.998e-03
Max Error: 1.412e-01
Cosine Similarity: 1.00198
Percentage Error: 0.189%

--------------------------------
Kcur_rot-0
--------------------------------
torch.Size([1, 4, 4, 128])
Kcur-0-ROPE-Kcur-0 (reshaped)-Metal#leaf_5#0{4, 1, 1, 1}-
Initial shape: 2048
Mean Error: 1.031e-02
Max Error: 4.045e-01
Cosine Similarity: 0.99849
Percentage Error: 0.150%

--------------------------------
attn_out-0
--------------------------------
torch.Size([1, 4, 28, 128])
__fattn__-0-FLASH_ATTN_EXT-Qcur-0 (view) (permuted)-Kcur-0 (permuted) (copy){128, 4, 4, 1}-
Initial shape: 14336
Mean Error: 1.746e-03
Max Error: 4.047e-02
Cosine Similarity: 1.00234
Percentage Error: 1.896%

--------------------------------
ffn_inp-0
--------------------------------
torch.Size([1, 4, 3584])
ffn_inp-0-ADD-node_23-Metal#inp_embd#0{3584, 4, 1, 1}-
Initial shape: 14336
Mean Error: 1.786e-03
Max Error: 2.159e-02
Cosine Similarity: 1.00125
Percentage Error: 2.238%

--------------------------------
ffn_norm-0
--------------------------------
torch.Size([1, 4, 3584])
ffn_norm-0-MUL-norm-0-blk.0.ffn_norm.weight{3584, 1, 1, 1}-
Initial shape: 14336
Mean Error: 3.339e-03
Max Error: 9.015e-02
Cosine Similarity: 1.00044
Percentage Error: 2.337%

--------------------------------
ffn_mlp_out-0
--------------------------------
torch.Size([1, 4, 3584])
ffn_out-0-MUL_MAT-blk.0.ffn_down.weight-ffn_swiglu-0{18944, 4, 1, 1}-
Initial shape: 14336
Mean Error: 2.237e-03
Max Error: 2.485e-02
Cosine Similarity: 0.99849
Percentage Error: 2.843%

--------------------------------
l_out-0
--------------------------------
torch.Size([1, 4, 3584])
l_out-0-ADD-ffn_out-0-ffn_inp-0{3584, 4, 1, 1}-
Initial shape: 14336
Mean Error: 2.847e-03
Max Error: 2.552e-02
Cosine Similarity: 0.99782
Percentage Error: 2.450%

...

--------------------------------
l_out-27
--------------------------------
torch.Size([1, 4, 3584])
l_out-27-ADD-ffn_out-27-ffn_inp-27{3584, 4, 1, 1}-
Initial shape: 14336
Mean Error: 1.766e-01
Max Error: 1.925e+01
Cosine Similarity: 1.00119
Percentage Error: 5.312%

--------------------------------
logits
--------------------------------
torch.Size([1, 4, 152064])
result_output-MUL_MAT-output.weight-result_norm{3584, 4, 1, 1}-
Initial shape: 608256
Mean Error: 1.574e-01
Max Error: 9.945e-01
Cosine Similarity: 1.00114
Percentage Error: 1.633%

[wp4032]

The reason why I am bringing this up is because I am trying to add support to https://github.com/RadicalNumerics/RND1
The model is just Qwen3-30B-A3B AR grafted to do diffusion, however, the logit divergences are reaching up to ~16% albeit the model has 48 layers vs. 28 layers.

[am17an]

For the above prompt, I am getting small divergence, however, would this be significant?

No these look fine to me.

[pwilkin]

@wp4032 Can you dump a few tensors in the eval-callback.cpp style / PyTorch print() style (first few and last few rows/columns)? Sometimes especially for the first token the divergences won't be large, but there'll be something that will obviously show a mismatch (like transposed values).

The last layer output does seem suspiciously divergent to me.

[am17an]

I don't think this can be a problem because this model is just the qwen2 architecture without the kv-cache. The diffusion stuff builds on top of the llama_decode path

[wp4032]

@pwilkin Here is a dump of the few tensors

--------------------------------
attn_norm-0
--------------------------------
**HF Dream**
tensor([[[-0.6406,  0.0466, -0.1289,  ..., -0.3848, -0.1719, -0.0022],
         [ 0.2988, -0.0679,  0.1030,  ..., -0.1318, -0.0179,  0.0767],
         [-0.0466, -0.2832, -0.0447,  ..., -0.2012, -0.0767,  0.0879],
         [ 0.0972, -0.3809,  0.0320,  ..., -0.3223,  0.0776,  0.3379]]],
       dtype=torch.bfloat16)

**llama.cpp**
ggml_debug:              attn_norm-0 = (f32)        MUL(norm-0{3584, 4, 1, 1}, blk.0.attn_norm.weight{3584, 1, 1, 1}}) = {3584, 4, 1, 1}
Saving tensor to attn_norm-0
                                     [
                                      [
                                       [     -0.6388,       0.0465,      -0.1292, ...,      -0.3842,      -0.1716,      -0.0022],
                                       [      0.2985,      -0.0679,       0.1027, ...,      -0.1314,      -0.0179,       0.0767],
                                       [     -0.0467,      -0.2824,      -0.0448, ...,      -0.2012,      -0.0766,       0.0877],
                                       [      0.0968,      -0.3803,       0.0321, ...,      -0.3208,       0.0779,       0.3379],
                                      ],
                                     ]
                                     sum = 59.582508

--------------------------------
Qcur-0
--------------------------------
**HF Dream**
torch.Size([1, 4, 3584])
tensor([[[ 0.3418,  0.9023, -0.4102,  ...,  0.7266, -0.8086,  0.4707],
         [ 0.0571,  2.8125,  2.9219,  ..., -0.2412, -0.2520,  0.5273],
         [ 0.1406,  1.3047,  0.2422,  ...,  0.3535, -0.6094,  0.4121],
         [-0.1104,  1.5625, -0.3750,  ...,  0.6016, -0.5547,  0.2373]]],
       dtype=torch.bfloat16)

**llama.cpp**
Saving tensor to Qcur-0
                                     [
                                      [
                                       [      0.3416,       0.9044,      -0.4094, ...,       0.7278,      -0.8078,       0.4711],
                                       [      0.0572,       2.8174,       2.9135, ...,      -0.2398,      -0.2526,       0.5278],
                                       [      0.1408,       1.3037,       0.2429, ...,       0.3527,      -0.6101,       0.4130],
                                       [     -0.1114,       1.5676,      -0.3758, ...,       0.6011,      -0.5541,       0.2372],
                                      ],
                                     ]
                                     sum = 1114.875977

--------------------------------
Qcur_rot-0
--------------------------------
**HF Dream**
tensor([[[[ 3.4180e-01,  9.0234e-01, -4.1016e-01,  ...,  1.0234e+00,
            6.4844e-01,  5.4688e-01],
          [-9.7656e-01, -4.5312e-01,  9.8438e-01,  ..., -8.4766e-01,
           -3.4375e-01, -3.9062e-01],
          [ 8.6328e-01,  1.0547e+00, -2.1875e-01,  ...,  6.9336e-02,
            6.6895e-02, -7.6562e-01],
...
          [-2.1484e-01,  7.5391e-01, -5.7812e-01,  ...,  5.5664e-02,
           -3.9453e-01,  2.6562e-01],
          [-4.2500e+00, -2.4219e+00, -3.9688e+00,  ...,  6.9336e-02,
           -6.6797e-01,  8.5156e-01],
          [-1.0742e-01,  5.8594e-01, -9.6484e-01,  ...,  6.0156e-01,
           -5.5469e-01,  2.3730e-01]]]], dtype=torch.bfloat16)

**llama.cpp**
Saving tensor to Qcur-0
                                     [
                                      [
                                       [      0.3416,       0.9044,      -0.4094, ...,       1.0241,       0.6464,       0.5432],
                                       [     -0.9756,      -0.4527,       0.9832, ...,      -0.8475,      -0.3430,      -0.3900],
                                       [      0.8641,       1.0566,      -0.2196, ...,       0.0698,       0.0675,      -0.7675],
...
                                       [     -0.2164,       0.7490,      -0.5791, ...,       0.0555,      -0.3946,       0.2659],
                                       [     -4.2435,      -2.4172,      -3.9575, ...,       0.0694,      -0.6689,       0.8533],
                                       [     -0.1081,       0.5809,      -0.9615, ...,       0.6011,      -0.5541,       0.2371],
                                      ],
                                     ]

--------------------------------
attn_out-0
--------------------------------
torch.Size([1, 4, 28, 128])
**HF Dream**
tensor([[[[ 0.1108, -0.1514, -0.2227,  ..., -0.0435, -0.0140, -0.1226],
          [ 0.2617,  0.0170, -0.1436,  ..., -0.0518, -0.0016, -0.0488],
          [ 0.0850, -0.1982, -0.2539,  ..., -0.0425, -0.0233, -0.1436],
...
          [-0.0986, -0.1147, -0.1416,  ..., -0.0359,  0.0239,  0.0153],
          [-0.0918, -0.0864, -0.0972,  ..., -0.0238,  0.0286, -0.0086],
          [-0.0967, -0.1143, -0.1348,  ..., -0.0347,  0.0247,  0.0140]]]],
       dtype=torch.bfloat16)

**llama.cpp**
ggml_debug:              __fattn__-0 = (f32) FLASH_ATTN_EXT(Qcur-0 (view) (permuted){128, 4, 28, 1}, Kcur-0 (permuted) (copy){128, 4, 4, 1}}) = {128, 28, 4, 1}
                                     [
                                      [
                                       [      0.1113,      -0.1508,      -0.2229, ...,      -0.0439,      -0.0144,      -0.1226],
                                       [      0.2595,       0.0167,      -0.1408, ...,      -0.0528,      -0.0014,      -0.0476],
                                       [      0.0852,      -0.1970,      -0.2540, ...,      -0.0422,      -0.0229,      -0.1439],
...
                                       [     -0.0982,      -0.1134,      -0.1388, ...,      -0.0353,       0.0245,       0.0146],
                                       [     -0.0902,      -0.0825,      -0.0901, ...,      -0.0219,       0.0303,      -0.0110],
                                       [     -0.0960,      -0.1124,      -0.1318, ...,      -0.0338,       0.0255,       0.0132],
                                      ],
                                     ]

--------------------------------
l_out-0
--------------------------------
**HF Dream**
torch.Size([1, 4, 3584])
tensor([[[-0.2168,  0.2188, -0.1201,  ...,  0.0181, -0.0464, -0.0403],
         [ 0.0552,  0.0752, -0.0013,  ...,  0.2070, -0.0835, -0.0444],
         [ 0.0752, -0.0496, -0.0957,  ...,  0.0457,  0.0703, -0.0532],
         [ 0.0400, -0.1699, -0.0752,  ..., -0.1562,  0.1309,  0.1885]]],
       dtype=torch.bfloat16)

**llama.cpp**
ggml_debug:                  l_out-0 = (f32)        ADD(ffn_out-0{3584, 4, 1, 1}, ffn_inp-0{3584, 4, 1, 1}}) = {3584, 4, 1, 1}
Saving tensor to l_out-0
                                     [
                                      [
                                       [     -0.2214,       0.2146,      -0.1197, ...,       0.0187,      -0.0467,      -0.0429],
                                       [      0.0500,       0.0704,       0.0061, ...,       0.2069,      -0.0856,      -0.0482],
                                       [      0.0713,      -0.0575,      -0.0903, ...,       0.0413,       0.0700,      -0.0581],
                                       [      0.0358,      -0.1724,      -0.0735, ...,      -0.1533,       0.1269,       0.1819],
                                      ],
                                     ]

--------------------------------
l_out-27
--------------------------------
**HF Dream**
torch.Size([1, 4, 3584])
tensor([[[-4.8125, -0.8750,  4.0625,  ...,  2.1875,  8.8750,  0.9844],
         [ 1.3125,  1.0781,  0.2969,  ..., -1.6328, -0.4531, -0.2734],
         [ 0.4922, -8.8750,  0.6367,  ..., -1.5000,  1.0469,  0.5469],
         [-0.3320, -1.7188,  0.7266,  ..., -4.5625,  3.1250,  1.5703]]],
       dtype=torch.bfloat16)

**llama.cpp**
ggml_debug:              result_norm = (f32)        MUL(norm{3584, 4, 1, 1}, output_norm.weight{3584, 1, 1, 1}}) = {3584, 4, 1, 1}
                                     [
                                      [
                                       [     -3.6425,      -0.6247,       2.8719, ...,       1.6180,       6.4944,       0.5932],
                                       [      1.7229,       1.1334,       0.3168, ...,      -2.0954,      -0.7161,      -0.3103],
                                       [      0.9578,      -8.7932,       0.0768, ...,      -1.8527,       0.7408,       0.3098],
                                       [     -0.2851,      -1.7179,       0.5808, ...,      -4.4494,       2.7729,       1.7212],
                                      ],
                                     ]

--------------------------------
logits
--------------------------------
**HF Dream**
torch.Size([1, 4, 152064])
tensor([[[ -4.0625,  -5.5625,  -1.3438,  ..., -11.9375, -11.9375, -11.9375],
         [ -0.7227,  -1.4062,  -2.6719,  ..., -10.3750, -10.3750, -10.3750],
         [ -3.1250,  -4.2188,  -5.7500,  ..., -10.3750, -10.3750, -10.3750],
         [ -1.8672,  -1.3906,  -3.4688,  ..., -11.7500, -11.7500, -11.7500]]],
       dtype=torch.bfloat16)

**llama.cpp**
ggml_debug:            result_output = (f32)    MUL_MAT(output.weight{3584, 152064, 1, 1}, result_norm{3584, 4, 1, 1}}) = {152064, 4, 1, 1}
Saving tensor to result_output
                                     [
                                      [
                                       [     -4.0605,      -5.6204,      -1.3390, ...,     -11.6773,     -11.6740,     -11.6738],
                                       [     -0.8945,      -1.6306,      -2.9489, ...,     -10.5174,     -10.5149,     -10.5146],
                                       [     -3.6256,      -4.4577,      -6.0833, ...,     -10.4264,     -10.4249,     -10.4246],
                                       [     -1.9183,      -1.5090,      -3.6252, ...,     -11.8188,     -11.8180,     -11.8182],
                                      ],
                                     ]

[pwilkin]

Yeah, those discrepancies on the last output are bad. Don't seem within acceptable range. Can you run make causal-verify-logits from examples/model-conversion and paste the output?

[pwilkin]

Also, can you link to which GGUF you're using?

[pwilkin]

@am17an yeah, but maybe there's some regression.

[wp4032]

I ran make causal-verify-logits, (note I had to modify the script a bit to load in Dream since it is geared towards AutoModelForCausalLM and here is the output:

Input prompt: "Hello, my name is"
Tokenized prompt (5 tokens): Hello, my name is
decode: cannot decode batches with this context (calling encode() instead)
Vocab size: 152064
Saving data to data/llamacpp-Dream-v0-Base-7B.bin
First 10 logits: 1.505608 -0.180042 -1.147197 -0.137608 -2.017541 -1.917705 -1.069074 -0.725663 0.951213 -0.995028 
Last 10 logits: -6.742494 -6.742306 -6.743474 -6.743182 -6.743601 -6.742061 -6.743907 -6.742569 -6.742828 -6.743344 

Data saved to data/llamacpp-Dream-v0-Base-7B.bin
Data saved to data/llamacpp-Dream-v0-Base-7B.txt
ggml_metal_free: deallocating
Checked all required files were found. Proceeding...

🔍 GGML Model Validation for model  Dream-v0-Base-7B
========================================
PyTorch logits  : data/pytorch-Dream-v0-Base-7B.bin
llama.cpp logits: data/llamacpp-Dream-v0-Base-7B.bin

Top 10 PyTorch logits: [6.239219  5.279565  5.2103357 4.996866  4.8161645 4.782921  4.671973
 4.592584  4.5648975 4.252734 ]
Top 10 llama.cpp logits: [6.261218  5.2432237 5.167961  4.9536886 4.7861686 4.762657  4.6705236
 4.5699744 4.5632434 4.2285147]
Max absolute difference: 0.0759
✅ OK: Lightweight model check successful!
       Ok to proceed with NMSE check...
Model name: Dream-v0-Base-7B
PyTorch logits file: data/pytorch-Dream-v0-Base-7B.bin
llama.cpp logits file: data/llamacpp-Dream-v0-Base-7B.bin
📊 NMSE Check for Model Comparison
==================================================
Reference (ground truth): data/pytorch-Dream-v0-Base-7B.bin
Test (to evaluate):       data/llamacpp-Dream-v0-Base-7B.bin

Loading reference logits...
  Shape: (152064,), Type: float32
Loading test logits...
  Shape: (152064,), Type: float32

✅ Shapes match: (152064,)

📈 METRICS
==============================
MSE (Mean Squared Error):     9.564759e-04
Reference Variance:           2.926356e+00
NMSE:                         3.268487e-04
Max Absolute Error:           0.075890
Mean Absolute Error:          0.029820
NMSE (dB):                    -34.86 dB

🎯 INTERPRETATION
==============================
👍 Very good match

📋 GUIDANCE
==============================
✅ EXCELLENT: Your GGML conversion is working very well!
   The differences are negligible for practical use.

📚 NMSE BENCHMARKS
==============================
< 1e-6:  Essentially identical
< 1e-4:  Excellent (typical for good conversions)
< 1e-3:  Very good
< 1e-2:  Good (acceptable for most use cases)
< 0.1:   Acceptable (may need verification)
> 1.0:   Poor (worse than random)

✅ RESULT: PASS (NMSE = 3.27e-04)

Here is the GGUF file
https://huggingface.co/wp4032/Dream7BBase_GGUF/tree/main

[pwilkin]

Yeah, looks a bit suspicious. You might want to run the run-single-model script from there to produce dumps with sums and then compare sums pairwise to the eval-callback dump looking for the first divergence.