Eval bug: llama-quantize crashes on Granite 4-h 1b #17677
Description
Name and Version
$ ./llama-cli --version
version: 7224 (7b6d745)
built with cc (GCC) 15.2.1 20251111 (Red Hat 15.2.1-4) for x86_64-redhat-linux
Operating systems
Linux
GGML backends
CPU
Hardware
Intel Core i7 Ultra
Models
IBM Granite 4-h 1B
Problem description & steps to reproduce
When I try to quantize IBM Granite 4-h 1B (or a fine-tune of that), llama-quantize crashes.
This reproduces on IBM's official BF16 GGUF that is available here: https://huggingface.co/ibm-granite/granite-4.0-h-1b-GGUF?show_file_info=granite-4.0-h-1b-bf16.gguf
(my use case was my fine-tune but I tested on the official model of course, to make sure this is not an issue with my pipeline)
First Bad Commit
Not sure. I didn't have the problem a couple of weeks ago but that build may have not been recent
Relevant log output
$ ./llama-quantize ~/models/granite-4.0-h-1b-bf16.gguf Q8_0
main: build = 7224 (7b6d74536)
main: built with cc (GCC) 15.2.1 20251111 (Red Hat 15.2.1-4) for x86_64-redhat-linux
main: quantizing '/home/mramendi/models/granite-4.0-h-1b-bf16.gguf' to '/home/mramendi/models/ggml-model-Q8_0.gguf' as Q8_0
llama_model_loader: loaded meta data with 47 key-value pairs and 506 tensors from /home/mramendi/models/granite-4.0-h-1b-bf16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = granitehybrid
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Granite 4.0 H 1b
llama_model_loader: - kv 3: general.basename str = granite-4.0-h
llama_model_loader: - kv 4: general.size_label str = 1B
llama_model_loader: - kv 5: general.license str = apache-2.0
llama_model_loader: - kv 6: general.base_model.count u32 = 1
llama_model_loader: - kv 7: general.base_model.0.name str = Granite 4.0 H 1b Base
llama_model_loader: - kv 8: general.base_model.0.organization str = Ibm Granite
llama_model_loader: - kv 9: general.base_model.0.repo_url str = https://huggingface.co/ibm-granite/gr...
llama_model_loader: - kv 10: general.tags arr[str,2] = ["language", "granite-4.0"]
llama_model_loader: - kv 11: granitehybrid.block_count u32 = 40
llama_model_loader: - kv 12: granitehybrid.context_length u32 = 1048576
llama_model_loader: - kv 13: granitehybrid.embedding_length u32 = 1536
llama_model_loader: - kv 14: granitehybrid.feed_forward_length u32 = 4096
llama_model_loader: - kv 15: granitehybrid.attention.head_count u32 = 12
llama_model_loader: - kv 16: granitehybrid.attention.head_count_kv arr[i32,40] = [0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, ...
llama_model_loader: - kv 17: granitehybrid.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 18: granitehybrid.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 19: granitehybrid.expert_count u32 = 0
llama_model_loader: - kv 20: granitehybrid.expert_used_count u32 = 0
llama_model_loader: - kv 21: general.file_type u32 = 32
llama_model_loader: - kv 22: granitehybrid.vocab_size u32 = 100352
llama_model_loader: - kv 23: granitehybrid.rope.dimension_count u32 = 128
llama_model_loader: - kv 24: granitehybrid.attention.scale f32 = 0.007812
llama_model_loader: - kv 25: granitehybrid.embedding_scale f32 = 12.000000
llama_model_loader: - kv 26: granitehybrid.residual_scale f32 = 0.220000
llama_model_loader: - kv 27: granitehybrid.logit_scale f32 = 6.000000
llama_model_loader: - kv 28: granitehybrid.expert_shared_feed_forward_length u32 = 4096
llama_model_loader: - kv 29: granitehybrid.ssm.conv_kernel u32 = 4
llama_model_loader: - kv 30: granitehybrid.ssm.state_size u32 = 128
llama_model_loader: - kv 31: granitehybrid.ssm.group_count u32 = 1
llama_model_loader: - kv 32: granitehybrid.ssm.inner_size u32 = 3072
llama_model_loader: - kv 33: granitehybrid.ssm.time_step_rank u32 = 48
llama_model_loader: - kv 34: granitehybrid.rope.scaling.finetuned bool = false
llama_model_loader: - kv 35: general.quantization_version u32 = 2
llama_model_loader: - kv 36: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 37: tokenizer.ggml.pre str = dbrx
llama_model_loader: - kv 38: tokenizer.ggml.tokens arr[str,100352] = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 39: tokenizer.ggml.token_type arr[i32,100352] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 40: tokenizer.ggml.merges arr[str,100000] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv 41: tokenizer.ggml.bos_token_id u32 = 100257
llama_model_loader: - kv 42: tokenizer.ggml.eos_token_id u32 = 100257
llama_model_loader: - kv 43: tokenizer.ggml.unknown_token_id u32 = 100269
llama_model_loader: - kv 44: tokenizer.ggml.padding_token_id u32 = 100256
llama_model_loader: - kv 45: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 46: tokenizer.chat_template str = {%- set tools_system_message_prefix =...
llama_model_loader: - type f32: 297 tensors
llama_model_loader: - type bf16: 209 tensors
llama_model_quantize_impl: n_layer_attn = 4, n_layer_recr = 36, pruned_attention_w = 0
/home/mramendi/repos/llama.cpp/src/llama-quant.cpp:743: GGML_ASSERT((qs.n_attention_wv == n_layer_attn - pruned_attention_w - n_layer_recr) && "n_attention_wv is unexpected") failed
This GDB supports auto-downloading debuginfo from the following URLs:
<https://debuginfod.fedoraproject.org/>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f278187c77e in __internal_syscall_cancel () from /lib64/libc.so.6
#0 0x00007f278187c77e in __internal_syscall_cancel () from /lib64/libc.so.6
#1 0x00007f278187c7a4 in __syscall_cancel () from /lib64/libc.so.6
#2 0x00007f27818ecc0f in wait4 () from /lib64/libc.so.6
#3 0x00007f2782278963 in ggml_print_backtrace () from /home/mramendi/repos/llama.cpp/bin/libggml-base.so.0
#4 0x00007f2782278aaf in ggml_abort () from /home/mramendi/repos/llama.cpp/bin/libggml-base.so.0
#5 0x00007f27820f9417 in llama_model_quantize_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, llama_model_quantize_params const*) () from /home/mramendi/repos/llama.cpp/bin/libllama.so.0
#6 0x00007f27820f9758 in llama_model_quantize () from /home/mramendi/repos/llama.cpp/bin/libllama.so.0
#7 0x00000000004064a8 in main ()
[Inferior 1 (process 53238) detached]
Aborted (core dumped)