Eval bug: Granite-4.0-h-1B fails long context (>16k); same model in Transformers works, Granite-4.0-h-MicroGGUF works)`

[mramendi]

Name and Version

$ llama-server --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
version: 7122 (21d31e081)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu

$ llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
version: 7122 (21d31e081)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
root@C.28352397:~/models$ 

Operating systems

Linux

GGML backends

CUDA
(does NOT happen on CPU)

Hardware

RTX 5090

Models

IBM Granite 4.0-h-1b, several quants (including BF16 GGUF)

Problem description & steps to reproduce

When I infer at a 32k context (simple script generating a NIAH text), the model returns "???????" or "G".

The same script at 16k context works wine with the model.

The same script with the same llama.cpp and with Granite-4-h-micro (3.5B) works fine at 32k context. The same script with a CPU build of llama.cpp and Granite-4-h-1b works fine at 32k context too.

Similar content, inferring on Transformers with Granite-4-h-1b (original safetensors version) works.

First Bad Commit

No response

Relevant log output

(not sure what to paste - no error message is displayed)

[pwilkin]

@gabe-l-hart any ideas?

[gabe-l-hart]

Hmm, when I've seen ??????? output from other models before, it's been due to floating point precision overflows, but this post says it's happening with BF16 which should have the same range as F32 and shouldn't show those issues. Is it possible to share the exact script being used, including the specific GGUF model (and how it was converted if done manually)?

[mramendi]

I have just built llama.cpp for CUDA anew, from the master branch, on a 5090 machine, and reproduced the issue. I am attaching the script,

niah.py

I use the BF16 GGUF from here: https://huggingface.co/ibm-granite/granite-4.0-h-1b-GGUF?show_file_info=granite-4.0-h-1b-bf16.gguf

I run the following command:

./llama-server -m models/granite-4.0-h-1b-bf16.gguf -c 32768

Then on the same machine (in another tmux window) I run the script, with the following result:

⚡ Testing Context Length: ~32000 tokens                                                                 
   ⏱  Time: 2.77s                                                                                       
   📍 Needle Depth: 65%                             
   🤖 Model Answer: '??????????????????????????????????????????????????'                                                                                                                                          
   ❌ FAIL: Drift Detected (Hallucination or Forgetting)                                                 

With the same model but on a CPU build of llama.cpp (on my local laptop) the script completes successfully, the model returns the correct answer. The llama.cpp on the CPU machine is:

./llama-server -m ~/models/granite-4.0-h-1b-bf16.gguf -c 32768 -t 11 -tb 22

Moreover I found another issue too. On CUDA. the ifeval results are significantly lower than the very same model in HF.

On the 5090 machine I run the command:

./llama-server -m models/granite-4.0-h-1b-bf16.gguf -c 32768 -np 64

Then in another tmux window:

lm_eval --model local-chat-completions --tasks ifeval --model_args model=ibm-granite/granite-4.0-1b,base_url=http://localhost:8080/v1/chat/completions,num_concurrent=64 --gen_kwargs temperature=0 --apply_chat_template

Result:

|Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|------|------:|------|-----:|-----------------------|---|-----:|---|------|
|ifeval|      4|none  |     0|inst_level_loose_acc   |↑  |0.7590|±  |   N/A|
|      |       |none  |     0|inst_level_strict_acc  |↑  |0.7434|±  |   N/A|
|      |       |none  |     0|prompt_level_loose_acc |↑  |0.6728|±  |0.0202|
|      |       |none  |     0|prompt_level_strict_acc|↑  |0.6525|±  |0.0205|

And then on the same machine, after stopping llama-server of course:

lm_eval --model hf --tasks ifeval --model_args pretrained=ibm-granite/granite-4.0-1b,dtype=bfloat16 --batch_size 64 --gen_kwargs temperature=0 --apply_chat_template

Result:

|Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|------|------:|------|-----:|-----------------------|---|-----:|---|------|
|ifeval|      4|none  |     0|inst_level_loose_acc   |↑  |0.8285|±  |   N/A|
|      |       |none  |     0|inst_level_strict_acc  |↑  |0.8129|±  |   N/A|
|      |       |none  |     0|prompt_level_loose_acc |↑  |0.7634|±  |0.0183|
|      |       |none  |     0|prompt_level_strict_acc|↑  |0.7486|±  |0.0187|

Now, running ifeval on CPU with an unquantized model is a lengthy task, so I am setting up a Llama.cpp CUDA/CPU ifeval A/B on Q4_K_M and will report when I have the results. (I mean I have the Llama.cpp CUDA results but the Llama.cpp CPU results will take some time)

[gabe-l-hart]

Ok, I'm able to repro this on my GB10 box and I also see the pattern that this starts emerging only around 32k. One important note: This is 32k of input not when run with -c 32768 (ie, it's the size of the input, not the size of the cache tensors).

When I run this against a debug build, I see that it is hitting this assertion, so I'll keep digging from there. I suspect we're getting NaNs somehow.

[gabe-l-hart]

Inspecting the values of cur_p->data does show that 0 is nan and the rest appear to be -inf, so definitely some bad logits here.

[gabe-l-hart]

Ok, I think the issue is around the data type used for the cache. If you use a full-width data type (f32/bf16) for the cache type, I no longer hit the error:

./bin/llama-server -m $HOME/models/ibm-granite/granite-4.0-h-1b-GGUF/granite-4.0-h-1b-bf16.gguf \
  -c 32768 -ctk bf16 -ctv bf16

$ python niah.py 
=== MAMBA QUANTIZATION DRIFT TEST (FIXED) ===

⚡ Testing Context Length: ~32000 tokens
   ⏱️ Time: 8.47s
   📍 Needle Depth: 64%
   🤖 Model Answer: 'The special passcode to unlock the Vault of Glass is 74921.'
   ✅ PASS: Needle Found

=== FINAL SUMMARY ===
{32000: 'PASS'}

[gabe-l-hart]

This finding actually makes a lot of sense w.r.t. what I was seeing with these nano models for the non-hybrid versions hitting float overflows when using f16. By default, llama-server uses f16 for the K/V caches, so it's likely dispatching to versions of the CUDA kernels for f16 and therefore hitting overflows.

[pwilkin]

Just out of curiosity, what happens if you use q8_0 caches? Does it fix the problem? (since it should then force it to f32 for quant/dequant)?