Bug: imatrix crash - nan detected in blk.1.attn_output.weight

[robbiemu]

What happened?

I've already quantized a 2b variant of this model, and one of its instruct fine tune, on a subset of the same data (the first 1000 samples are the same in the same order -- the earlier was 34x1000 samples and now they are 34x5000 samples). now on the 7b model, I get an error. There are differences, my command now has the --save-frequency and --batch-size and --ubatch-size set to bring the hours down. (I have to admit I misread the -ngl flag as --ngl and left that off, trying to add that now)

Name and Version

llama-cli --version
version: 3912 (edc2656)
built with Apple clang version 16.0.0 (clang-1600.0.26.3) for arm64-apple-darwin24.0.0

What operating system are you seeing the problem on?

Mac

Relevant log output

/Users/macdev/Downloads/build/bin/llama-imatrix \
 -m ./salamandra-7b_bf16.gguf \
 -f ./imatrix/oscar/imatrix-dataset.txt \
 -o ./imatrix/oscar/imatrix.dat \
 --save-frequency 356 --ubatch-size 4096 --batch-size 9192 --threads 15 \
 --ctx-size 8192 \
 --rope-freq-base 10000.0 \
 --top-p 0.95 \
 --temp 0 \
 --repeat-penalty 1.2
build: 3906 (7eee341b) with Apple clang version 15.0.0 (clang-1500.3.9.4) for arm64-apple-darwin23.6.0
llama_model_loader: loaded meta data with 29 key-value pairs and 291 tensors from ./salamandra-7b_bf16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                         general.size_label str              = 7.8B
llama_model_loader: - kv   3:                            general.license str              = apache-2.0
llama_model_loader: - kv   4:                               general.tags arr[str,1]       = ["text-generation"]
llama_model_loader: - kv   5:                          general.languages arr[str,36]      = ["bg", "ca", "code", "cs", "cy", "da"...
llama_model_loader: - kv   6:                          llama.block_count u32              = 32
llama_model_loader: - kv   7:                       llama.context_length u32              = 8192
llama_model_loader: - kv   8:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   9:                  llama.feed_forward_length u32              = 11008
llama_model_loader: - kv  10:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  11:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  12:                       llama.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  13:     llama.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  14:                          general.file_type u32              = 32
llama_model_loader: - kv  15:                           llama.vocab_size u32              = 256000
llama_model_loader: - kv  16:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  17:            tokenizer.ggml.add_space_prefix bool             = true
llama_model_loader: - kv  18:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  19:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  20:                      tokenizer.ggml.tokens arr[str,256000]  = ["<unk>", "<s>", "</s>", "<pad>", "<|...
llama_model_loader: - kv  21:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  22:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  23:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  24:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  25:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  26:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  27:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  28:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type bf16:  226 tensors
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 104
llm_load_vocab: token to piece cache size = 1.8842 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 256000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 11008
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 8B
llm_load_print_meta: model ftype      = BF16
llm_load_print_meta: model params     = 7.77 B
llm_load_print_meta: model size       = 14.47 GiB (16.00 BPW)
llm_load_print_meta: general.name     = n/a
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 145 '<0x0A>'
llm_load_print_meta: EOT token        = 5 '<|im_end|>'
llm_load_print_meta: EOG token        = 2 '</s>'
llm_load_print_meta: EOG token        = 5 '<|im_end|>'
llm_load_print_meta: max token length = 72
llm_load_tensors: ggml ctx size =    0.27 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:      Metal buffer size = 12817.03 MiB
llm_load_tensors:        CPU buffer size =  2000.00 MiB
...........................................................................
llama_new_context_with_model: n_ctx      = 8192
llama_new_context_with_model: n_batch    = 8192
llama_new_context_with_model: n_ubatch   = 4096
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3 Max
ggml_metal_init: picking default device: Apple M3 Max
ggml_metal_init: using embedded metal library
ggml_metal_init: GPU name:   Apple M3 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 42949.67 MB
llama_kv_cache_init:      Metal KV buffer size =  1024.00 MiB
llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.98 MiB
llama_new_context_with_model:      Metal compute buffer size =  4384.00 MiB
llama_new_context_with_model:        CPU compute buffer size =  4000.00 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 451

system_info: n_threads = 15 (n_threads_batch = 15) / 16 | AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 1 | LLAMAFILE = 1 |
compute_imatrix: tokenizing the input ..
compute_imatrix: tokenization took 255156 ms
compute_imatrix: computing over 12457 chunks with batch_size 8192
compute_imatrix: 57.44 seconds per pass - ETA 198 hours 45.67 minutes
[1]6.6357,[2]6.2563,[3]7.9948,[4]8.4627,[5]9.6051,[6]9.3009,[7]8.6191,[8]8.4246,[9]8.4066,[10]8.3625,[11]8.6955,[12]8.7766,[13]8.8980,[14]9.1834,[15]8.9132,[16]9.2713,[17]9.5073,[18]9.7867,[19]9.6462,[20]9.0441,[21]8.9722,[22]8.8748,[23]9.0193,[24]9.0332,[25]8.6903,[26]8.7325,[27]8.7363,[28]8.7711,[29]8.9056,[30]9.0395,[31]8.9306,[32]9.0220,[33]8.7711,[34]8.7847,[35]8.7584,[36]8.6798,[37]8.5357,[38]8.5432,[39]8.5379,[40]8.5529,[41]8.4827,[42]8.5009,[43]8.3460,[44]8.2342,[45]8.2379,[46]8.2105,[47]8.0725,[48]8.1300,[49]8.1965,[50]8.2583,[51]8.3348,[52]8.3276,[53]8.3039,[54]8.3552,[55]8.3921,[56]8.4113,[57]8.4797,[58]8.2914,[59]8.1574,[60]7.9299,[61]7.6995,[62]7.6783,[63]7.6994,[64]7.7063,[65]7.7271,[66]7.7604,[67]7.8190,[68]7.8291,[69]7.7810,[70]7.6424,[71]7.7119,[72]7.7153,[73]7.7591,[74]7.8025,[75]7.8430,[76]7.7903,[77]7.8158,[78]7.8813,[79]7.8878,[80]7.9519,[81]8.0622,[82]8.0805,[83]8.0981,[84]7.9537,[85]7.8371,[86]7.7102,[87]7.5913,[88]7.4637,[89]7.3735,[90]7.3090,[91]7.1828,[92]7.1577,[93]7.1689,[94]7.2415,[95]7.1740,[96]7.1992,[97]7.1817,[98]7.2124,[99]7.2384,[100]7.2707,[101]7.2781,[102]7.2921,[103]7.3233,[104]7.3537,[105]7.4109,[106]7.4193,[107]7.4181,[108]7.4080,[109]7.4947,[110]7.4652,[111]7.4980,[112]7.5510,[113]7.6089,[114]7.6270,[115]7.6204,[116]7.6481,[117]7.6620,[118]7.6804,[119]7.7198,[120]7.7411,[121]7.7758,[122]7.7900,[123]7.8268,[124]7.8747,[125]7.9093,nan detected in blk.1.attn_output.weight

[robbiemu]

actually I'm unclear what these chunks even are.

there are 34 * 5000 samples = 170000 samples. many (most) are multiline, so:

wc -l imatrix/oscar/imatrix-dataset.txt
 1288254 imatrix/oscar/imatrix-dataset.txt
wc -w !$
wc -w imatrix/oscar/imatrix-dataset.txt
 59917799 imatrix/oscar/imatrix-dataset.txt
ls -la !$
ls -la imatrix/oscar/imatrix-dataset.txt
-rw-r--r--@ 1 macdev  wheel   443M Oct 12 11:37 imatrix/oscar/imatrix-dataset.txt

even it it is chunked by word 12457 chunks goes into 59.n million ~4800 times, not 4096 or 9192 times.

edit: of course, they are tokens, not lines or words :)

[robbiemu]

I've been playing with these flags for a little bit now and I find that I'm able to get crashes at different chunks, so I guess this is just what OOM looks like. I think the biggest factor here is mu batch size.

Going with -ngl -1 --batch-size 8192 --ubatch-size 1024 and dropping the save-freq since I see a PR to fix loading from previous work. (I'm not convinced that -ngl -1 does anything on metal but then it doesn't matter)

I will mark this closed when I get over double my previous completed batches.

[slaren]

I guess this is just what OOM looks like

That's not likely to be the issue, you would get a memory allocation error in that case. The most common cause of nans during computation is an overflow during the attention, since results are stored in F16 and have a limited range. The fact that this happens in the attn_output tensor suggests that this is the case.
Also keep in mind that BF16 is not supported in the Metal backend, which will cause most of these operations to be run on the CPU instead.

[robbiemu]

Feel free to reopen if you think its a bug and not just my system memory or whatever. (I’m approaching 3 times the number of chunks now without issue and everything is working, with 1k ubatch and 8k batch, just as described earlier).

I noticed the cpu use (gpu is definitely used too though), that’s why when I found -ngl I had to use it.. but i do still see largely the same behavior, and, thanks, now I know why.

[robbiemu]

I reported the error to the team that made the model, but then thinking about it some more .. I think I've found a real issue.

First I need to make clear an issue with the OP issue. I noted I have v3912, and I first experienced this issue with v3912, but when writing this ticket, I realized that I didn't have logs and quickly reran it, using the old path for v3906, so the logs there are not matching for v3912, as below. So, ultimately the overflow issue I don't expect to solve here. But I am reporting something else:

The model providers use the same tokenizer for the base model at 2b as at 7b, and the same one at 2b-instruct as at 7b-instruct. I've already made complete GGUFs from imatrices on 1/5 of this data for the 2b base model, and this problem is on the 7b base model. Let's demonstrate they are the same:

md5 tokenizer.json
MD5 (tokenizer.json) = dcb7b9a40e4967cadb8eeadb4ed3f5d2
md5 ../salamandra-2b/tokenizer.json
MD5 (../salamandra-2b/tokenizer.json) = dcb7b9a40e4967cadb8eeadb4ed3f5d2
md5 tokenizer.model
MD5 (tokenizer.model) = 2378d392cfb6f4b30aa9d38f54975b0a
md5 ../salamandra-2b/tokenizer.model
MD5 (../salamandra-2b/tokenizer.model) = 2378d392cfb6f4b30aa9d38f54975b0a

I also have a log of when I made the imatrix for the 2b model successfully, before:

/Users/macdev/Downloads/build/bin/llama-imatrix \
    -m ./salamandra-2b_bf16.gguf \
    -f ./imatrix/oscar/imatrix-dataset.txt \
    -o ./imatrix/oscar/imatrix.dat \
    --ctx-size 8192 \
    --rope-freq-base 10000.0 \
    --top-p 0.95 \
    --temp 0 \
    --repeat-penalty 1.2  2>&1 | tee imatrix_log.txt

build: 3906 (7eee341b) with Apple clang version 15.0.0 (clang-1500.3.9.4) for arm64-apple-darwin23.6.0
llama_model_loader: loaded meta data with 29 key-value pairs and 219 tensors from ./salamandra-2b_bf16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                         general.size_label str              = 2.3B
llama_model_loader: - kv   3:                            general.license str              = apache-2.0
llama_model_loader: - kv   4:                               general.tags arr[str,1]       = ["text-generation"]
llama_model_loader: - kv   5:                          general.languages arr[str,36]      = ["bg", "ca", "code", "cs", "cy", "da"...
llama_model_loader: - kv   6:                          llama.block_count u32              = 24
llama_model_loader: - kv   7:                       llama.context_length u32              = 8192
llama_model_loader: - kv   8:                     llama.embedding_length u32              = 2048
llama_model_loader: - kv   9:                  llama.feed_forward_length u32              = 5440
llama_model_loader: - kv  10:                 llama.attention.head_count u32              = 16
llama_model_loader: - kv  11:              llama.attention.head_count_kv u32              = 16
llama_model_loader: - kv  12:                       llama.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  13:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  14:                          general.file_type u32              = 32
llama_model_loader: - kv  15:                           llama.vocab_size u32              = 256000
llama_model_loader: - kv  16:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  17:            tokenizer.ggml.add_space_prefix bool             = true
llama_model_loader: - kv  18:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  19:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  20:                      tokenizer.ggml.tokens arr[str,256000]  = ["<unk>", "<s>", "</s>", "<pad>", "<|...
llama_model_loader: - kv  21:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  22:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  23:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  24:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  25:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  26:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  27:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  28:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   49 tensors
llama_model_loader: - type bf16:  170 tensors
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 104
llm_load_vocab: token to piece cache size = 1.8842 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 256000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 2048
llm_load_print_meta: n_layer          = 24
llm_load_print_meta: n_head           = 16
llm_load_print_meta: n_head_kv        = 16
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 2048
llm_load_print_meta: n_embd_v_gqa     = 2048
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 5440
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = BF16
llm_load_print_meta: model params     = 2.25 B
llm_load_print_meta: model size       = 4.20 GiB (16.00 BPW)
llm_load_print_meta: general.name     = n/a
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 145 '<0x0A>'
llm_load_print_meta: EOT token        = 5 '<|im_end|>'
llm_load_print_meta: EOG token        = 2 '</s>'
llm_load_print_meta: EOG token        = 5 '<|im_end|>'
llm_load_print_meta: max token length = 72
llm_load_tensors: ggml ctx size =    0.20 MiB
llm_load_tensors: offloading 24 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 25/25 layers to GPU
llm_load_tensors:      Metal buffer size =  4298.39 MiB
llm_load_tensors:        CPU buffer size =  1000.00 MiB
.......................................................
llama_new_context_with_model: n_ctx      = 8192
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3 Max
ggml_metal_init: picking default device: Apple M3 Max
ggml_metal_init: using embedded metal library
ggml_metal_init: GPU name:   Apple M3 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 42949.67 MB
llama_kv_cache_init:      Metal KV buffer size =  1536.00 MiB
llama_new_context_with_model: KV self size  = 1536.00 MiB, K (f16):  768.00 MiB, V (f16):  768.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.98 MiB
llama_new_context_with_model:      Metal compute buffer size =   288.00 MiB
llama_new_context_with_model:        CPU compute buffer size =   500.00 MiB
llama_new_context_with_model: graph nodes  = 774
llama_new_context_with_model: graph splits = 339

system_info: n_threads = 12 (n_threads_batch = 12) / 16 | AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 1 | LLAMAFILE = 1 |
compute_imatrix: tokenizing the input ..
compute_imatrix: tokenization took 51343.6 ms
compute_imatrix: computing over 2761 chunks with batch_size 2048
compute_imatrix: 20.49 seconds per pass - ETA 15 hours 42.95 minutes
[1]8.4459,[2]8.1889,[3]10.8453,[4]11.5995,[5]13.3446,[6]12.8847,[7]11.9359,[8]11.7226,[9]11.

It just goes on. There was no error.

So, just to be crystal clear, now that I am on v3912, in this 2b model's directory, do I get a different log just rerunning the known good llama-imatrix command?

/Users/Shared/Public/Github/llama.cpp/llama-imatrix \
 -m ./salamandra-2b_bf16.gguf \
 -f ./imatrix/oscar/imatrix-dataset.txt \
 -o ./tmp.dat \
 -ngl 99 --n-gpu-layers -1 --threads 15 --batch-size 65536 --ubatch-size 16384 \
 --ctx-size 8192 \
 --rope-freq-base 10000.0 \
 --top-p 0.95 \
 --temp 0 \
 --repeat-penalty 1.2
build: 3927 (10433e8b) with Apple clang version 16.0.0 (clang-1600.0.26.3) for arm64-apple-darwin24.0.0
llama_model_loader: loaded meta data with 29 key-value pairs and 219 tensors from ./salamandra-2b_bf16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                         general.size_label str              = 2.3B
llama_model_loader: - kv   3:                            general.license str              = apache-2.0
llama_model_loader: - kv   4:                               general.tags arr[str,1]       = ["text-generation"]
llama_model_loader: - kv   5:                          general.languages arr[str,36]      = ["bg", "ca", "code", "cs", "cy", "da"...
llama_model_loader: - kv   6:                          llama.block_count u32              = 24
llama_model_loader: - kv   7:                       llama.context_length u32              = 8192
llama_model_loader: - kv   8:                     llama.embedding_length u32              = 2048
llama_model_loader: - kv   9:                  llama.feed_forward_length u32              = 5440
llama_model_loader: - kv  10:                 llama.attention.head_count u32              = 16
llama_model_loader: - kv  11:              llama.attention.head_count_kv u32              = 16
llama_model_loader: - kv  12:                       llama.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  13:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  14:                          general.file_type u32              = 32
llama_model_loader: - kv  15:                           llama.vocab_size u32              = 256000
llama_model_loader: - kv  16:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  17:            tokenizer.ggml.add_space_prefix bool             = true
llama_model_loader: - kv  18:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  19:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  20:                      tokenizer.ggml.tokens arr[str,256000]  = ["<unk>", "<s>", "</s>", "<pad>", "<|...
llama_model_loader: - kv  21:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  22:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  23:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  24:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  25:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  26:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  27:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  28:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   49 tensors
llama_model_loader: - type bf16:  170 tensors
llm_load_vocab: control token:    103 '<|reserved_token_98|>' is not marked as EOG
llm_load_vocab: control token:     60 '<|reserved_token_55|>' is not marked as EOG
llm_load_vocab: control token:     93 '<|reserved_token_88|>' is not marked as EOG
llm_load_vocab: control token:     94 '<|reserved_token_89|>' is not marked as EOG
llm_load_vocab: control token:     56 '<|reserved_token_51|>' is not marked as EOG
llm_load_vocab: control token:     27 '<|reserved_token_22|>' is not marked as EOG
llm_load_vocab: control token:     16 '<|reserved_token_11|>' is not marked as EOG
llm_load_vocab: control token:    101 '<|reserved_token_96|>' is not marked as EOG
llm_load_vocab: control token:     71 '<|reserved_token_66|>' is not marked as EOG
llm_load_vocab: control token:     79 '<|reserved_token_74|>' is not marked as EOG
llm_load_vocab: control token:     30 '<|reserved_token_25|>' is not marked as EOG
llm_load_vocab: control token:     89 '<|reserved_token_84|>' is not marked as EOG
llm_load_vocab: control token:     37 '<|reserved_token_32|>' is not marked as EOG
llm_load_vocab: control token:     76 '<|reserved_token_71|>' is not marked as EOG
llm_load_vocab: control token:     96 '<|reserved_token_91|>' is not marked as EOG
llm_load_vocab: control token:     10 '<|reserved_token_5|>' is not marked as EOG
llm_load_vocab: control token:     61 '<|reserved_token_56|>' is not marked as EOG
llm_load_vocab: control token:     26 '<|reserved_token_21|>' is not marked as EOG
llm_load_vocab: control token:     68 '<|reserved_token_63|>' is not marked as EOG
llm_load_vocab: control token:     74 '<|reserved_token_69|>' is not marked as EOG
llm_load_vocab: control token:     20 '<|reserved_token_15|>' is not marked as EOG
llm_load_vocab: control token:     11 '<|reserved_token_6|>' is not marked as EOG
llm_load_vocab: control token:      9 '<|reserved_token_4|>' is not marked as EOG
llm_load_vocab: control token:     32 '<|reserved_token_27|>' is not marked as EOG
llm_load_vocab: control token:     34 '<|reserved_token_29|>' is not marked as EOG
llm_load_vocab: control token:     23 '<|reserved_token_18|>' is not marked as EOG
llm_load_vocab: control token:     54 '<|reserved_token_49|>' is not marked as EOG
llm_load_vocab: control token:     62 '<|reserved_token_57|>' is not marked as EOG
llm_load_vocab: control token:     38 '<|reserved_token_33|>' is not marked as EOG
llm_load_vocab: control token:     24 '<|reserved_token_19|>' is not marked as EOG
llm_load_vocab: control token:      3 '<pad>' is not marked as EOG
llm_load_vocab: control token:     81 '<|reserved_token_76|>' is not marked as EOG
llm_load_vocab: control token:     50 '<|reserved_token_45|>' is not marked as EOG
llm_load_vocab: control token:     51 '<|reserved_token_46|>' is not marked as EOG
llm_load_vocab: control token:     86 '<|reserved_token_81|>' is not marked as EOG
llm_load_vocab: control token:     22 '<|reserved_token_17|>' is not marked as EOG
llm_load_vocab: control token:     95 '<|reserved_token_90|>' is not marked as EOG
llm_load_vocab: control token:     72 '<|reserved_token_67|>' is not marked as EOG
llm_load_vocab: control token:     69 '<|reserved_token_64|>' is not marked as EOG
llm_load_vocab: control token:      7 '<|reserved_token_2|>' is not marked as EOG
llm_load_vocab: control token:     63 '<|reserved_token_58|>' is not marked as EOG
llm_load_vocab: control token:     31 '<|reserved_token_26|>' is not marked as EOG
llm_load_vocab: control token:     88 '<|reserved_token_83|>' is not marked as EOG
llm_load_vocab: control token:     97 '<|reserved_token_92|>' is not marked as EOG
llm_load_vocab: control token:     42 '<|reserved_token_37|>' is not marked as EOG
llm_load_vocab: control token:     98 '<|reserved_token_93|>' is not marked as EOG
llm_load_vocab: control token:    102 '<|reserved_token_97|>' is not marked as EOG
llm_load_vocab: control token:     84 '<|reserved_token_79|>' is not marked as EOG
llm_load_vocab: control token:     25 '<|reserved_token_20|>' is not marked as EOG
llm_load_vocab: control token:     19 '<|reserved_token_14|>' is not marked as EOG
llm_load_vocab: control token:     33 '<|reserved_token_28|>' is not marked as EOG
llm_load_vocab: control token:     67 '<|reserved_token_62|>' is not marked as EOG
llm_load_vocab: control token:     47 '<|reserved_token_42|>' is not marked as EOG
llm_load_vocab: control token:     59 '<|reserved_token_54|>' is not marked as EOG
llm_load_vocab: control token:     41 '<|reserved_token_36|>' is not marked as EOG
llm_load_vocab: control token:     87 '<|reserved_token_82|>' is not marked as EOG
llm_load_vocab: control token:     57 '<|reserved_token_52|>' is not marked as EOG
llm_load_vocab: control token:     64 '<|reserved_token_59|>' is not marked as EOG
llm_load_vocab: control token:     13 '<|reserved_token_8|>' is not marked as EOG
llm_load_vocab: control token:     18 '<|reserved_token_13|>' is not marked as EOG
llm_load_vocab: control token:     15 '<|reserved_token_10|>' is not marked as EOG
llm_load_vocab: control token:     92 '<|reserved_token_87|>' is not marked as EOG
llm_load_vocab: control token:     55 '<|reserved_token_50|>' is not marked as EOG
llm_load_vocab: control token:      8 '<|reserved_token_3|>' is not marked as EOG
llm_load_vocab: control token:     35 '<|reserved_token_30|>' is not marked as EOG
llm_load_vocab: control token:     49 '<|reserved_token_44|>' is not marked as EOG
llm_load_vocab: control token:     85 '<|reserved_token_80|>' is not marked as EOG
llm_load_vocab: control token:     65 '<|reserved_token_60|>' is not marked as EOG
llm_load_vocab: control token:     21 '<|reserved_token_16|>' is not marked as EOG
llm_load_vocab: control token:     36 '<|reserved_token_31|>' is not marked as EOG
llm_load_vocab: control token:     83 '<|reserved_token_78|>' is not marked as EOG
llm_load_vocab: control token:     90 '<|reserved_token_85|>' is not marked as EOG
llm_load_vocab: control token:      6 '<|reserved_token_1|>' is not marked as EOG
llm_load_vocab: control token:     44 '<|reserved_token_39|>' is not marked as EOG
llm_load_vocab: control token:     45 '<|reserved_token_40|>' is not marked as EOG
llm_load_vocab: control token:     14 '<|reserved_token_9|>' is not marked as EOG
llm_load_vocab: control token:    100 '<|reserved_token_95|>' is not marked as EOG
llm_load_vocab: control token:      0 '<unk>' is not marked as EOG
llm_load_vocab: control token:     12 '<|reserved_token_7|>' is not marked as EOG
llm_load_vocab: control token:     82 '<|reserved_token_77|>' is not marked as EOG
llm_load_vocab: control token:     17 '<|reserved_token_12|>' is not marked as EOG
llm_load_vocab: control token:     48 '<|reserved_token_43|>' is not marked as EOG
llm_load_vocab: control token:     80 '<|reserved_token_75|>' is not marked as EOG
llm_load_vocab: control token:     29 '<|reserved_token_24|>' is not marked as EOG
llm_load_vocab: control token:     43 '<|reserved_token_38|>' is not marked as EOG
llm_load_vocab: control token:     66 '<|reserved_token_61|>' is not marked as EOG
llm_load_vocab: control token:     28 '<|reserved_token_23|>' is not marked as EOG
llm_load_vocab: control token:      4 '<|im_start|>' is not marked as EOG
llm_load_vocab: control token:     77 '<|reserved_token_72|>' is not marked as EOG
llm_load_vocab: control token:     75 '<|reserved_token_70|>' is not marked as EOG
llm_load_vocab: control token:     53 '<|reserved_token_48|>' is not marked as EOG
llm_load_vocab: control token:     39 '<|reserved_token_34|>' is not marked as EOG
llm_load_vocab: control token:      1 '<s>' is not marked as EOG
llm_load_vocab: control token:     52 '<|reserved_token_47|>' is not marked as EOG
llm_load_vocab: control token:     40 '<|reserved_token_35|>' is not marked as EOG
llm_load_vocab: control token:     58 '<|reserved_token_53|>' is not marked as EOG
llm_load_vocab: control token:     99 '<|reserved_token_94|>' is not marked as EOG
llm_load_vocab: control token:     78 '<|reserved_token_73|>' is not marked as EOG
llm_load_vocab: control token:      2 '</s>' is not marked as EOG
llm_load_vocab: control token:     70 '<|reserved_token_65|>' is not marked as EOG
llm_load_vocab: control token:     46 '<|reserved_token_41|>' is not marked as EOG
llm_load_vocab: control token:     91 '<|reserved_token_86|>' is not marked as EOG
llm_load_vocab: control token:     73 '<|reserved_token_68|>' is not marked as EOG
llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
llm_load_vocab: special tokens cache size = 104
llm_load_vocab: token to piece cache size = 1.8842 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 256000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 2048
llm_load_print_meta: n_layer          = 24
llm_load_print_meta: n_head           = 16
llm_load_print_meta: n_head_kv        = 16
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 2048
llm_load_print_meta: n_embd_v_gqa     = 2048
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 5440
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = BF16
llm_load_print_meta: model params     = 2.25 B
llm_load_print_meta: model size       = 4.20 GiB (16.00 BPW)
llm_load_print_meta: general.name     = n/a
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: EOT token        = 5 '<|im_end|>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 145 '<0x0A>'
llm_load_print_meta: EOG token        = 2 '</s>'
llm_load_print_meta: EOG token        = 5 '<|im_end|>'
llm_load_print_meta: max token length = 72
llm_load_tensors: ggml ctx size =    0.20 MiB
llm_load_tensors: offloading 24 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 25/25 layers to GPU
llm_load_tensors:      Metal buffer size =  4298.39 MiB
llm_load_tensors:        CPU buffer size =  1000.00 MiB
.......................................................
llama_new_context_with_model: n_ctx      = 8192
llama_new_context_with_model: n_batch    = 8192
llama_new_context_with_model: n_ubatch   = 8192
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3 Max
ggml_metal_init: picking default device: Apple M3 Max
ggml_metal_init: using embedded metal library
ggml_metal_init: GPU name:   Apple M3 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 42949.67 MB
llama_kv_cache_init:      Metal KV buffer size =  1536.00 MiB
llama_new_context_with_model: KV self size  = 1536.00 MiB, K (f16):  768.00 MiB, V (f16):  768.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.98 MiB
llama_new_context_with_model:      Metal compute buffer size =  4608.00 MiB
llama_new_context_with_model:        CPU compute buffer size =  8000.00 MiB
llama_new_context_with_model: graph nodes  = 774
llama_new_context_with_model: graph splits = 339

system_info: n_threads = 15 (n_threads_batch = 15) / 16 | AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 1 | LLAMAFILE = 1 |
compute_imatrix: tokenizing the input ..
^C

Yes, sadly, I get these llm_load_vocab warnings. This was introduced between v3906 and v3912.

I thought this was the reason why the imatrix didn't build, and came back to reopen this ticket, not realizing at first the in the op I had already tried using v3906 where I get no such warnings.

[robbiemu]

Closed again because I realized these warnings are likely just that, newly added warnings. apparently llama.cpp is not yet processing SentencePiece tokenizers quite yet

Salamandra's input tokenizer/config.yml
tokenization strategy selection notes: tokenizer choices
exact 7b model settings config

[arch-btw]

@robbiemu those llm_load_vocab warnings are happening on llama 3.1 as well. 🤔

[ggerganov]

The "is not marked EOG" warnings are not relevant. I just changed the, to debug logs so they don't appear during normal execution: 9f45fc1

[robbiemu]

Just in case it is the tokenizer (which is SentencePiece/BPE but still has tokenizer.model and tokenizer.json) I modified the update script as below and have a GGUF for it now. How do I add it to llama.cpp to test it locally?

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

# This script downloads the tokenizer models of the specified models from Huggingface and
# generates the get_vocab_base_pre() function for convert_hf_to_gguf.py
#
# This is necessary in order to analyze the type of pre-tokenizer used by the model and
# provide the necessary information to llama.cpp via the GGUF header in order to implement
# the same pre-tokenizer.
#
# ref: https://github.com/ggerganov/llama.cpp/pull/6920
#
# Instructions:
#
# - Add a new model to the "models" list
# - Run the script with your huggingface token:
#
#   python3 convert_hf_to_gguf_update.py <huggingface_token>
#
# - Copy-paste the generated get_vocab_base_pre() function into convert_hf_to_gguf.py
# - Update llama.cpp with the new pre-tokenizer if necessary
#
# TODO: generate tokenizer tests for llama.cpp
#

import logging
import os
import pathlib
import re

import requests
import sys
import json
import shutil

from hashlib import sha256
from enum import IntEnum, auto
from transformers import AutoTokenizer

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger("convert_hf_to_gguf_update")
sess = requests.Session()


class TOKENIZER_TYPE(IntEnum):
    SPM = auto()
    BPE = auto()
    WPM = auto()
    UGM = auto()
    SPM_BPE = auto()


# TODO: this string has to exercise as much pre-tokenizer functionality as possible
#       will be updated with time - contributions welcome
CHK_TXT = '\n \n\n \n\n\n \t \t\t \t\n  \n   \n    \n     \n🚀 (normal) 😶‍🌫️ (multiple emojis concatenated) ✅ 🦙🦙 3 33 333 3333 33333 333333 3333333 33333333 3.3 3..3 3...3 កាន់តែពិសេសអាច😁 ?我想在apple工作1314151天～ ------======= нещо на Български \'\'\'\'\'\'```````\"\"\"\"......!!!!!!?????? I\'ve been \'told he\'s there, \'RE you sure? \'M not sure I\'ll make it, \'D you like some tea? We\'Ve a\'lL'

if len(sys.argv) == 2:
    token = sys.argv[1]
    if not token.startswith("hf_"):
        logger.info("Huggingface token seems invalid")
        logger.info("Usage: python convert_hf_to_gguf_update.py <huggingface_token>")
        sys.exit(1)
else:
    logger.info("Usage: python convert_hf_to_gguf_update.py <huggingface_token>")
    sys.exit(1)

# TODO: add models here, base models preferred
models = [
    {"name": "salamandra-spm-bpe", "tokt": TOKENIZER_TYPE.SPM_BPE, "repo": "https://huggingface.co/BSC-LT/salamandra-2b", },
    {"name": "llama-spm",          "tokt": TOKENIZER_TYPE.SPM,     "repo": "https://huggingface.co/meta-llama/Llama-2-7b-hf", },
    {"name": "llama-bpe",          "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/meta-llama/Meta-Llama-3-8B", },
    {"name": "phi-3",              "tokt": TOKENIZER_TYPE.SPM,     "repo": "https://huggingface.co/microsoft/Phi-3-mini-4k-instruct", },
    {"name": "deepseek-llm",       "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/deepseek-ai/deepseek-llm-7b-base", },
    {"name": "deepseek-coder",     "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base", },
    {"name": "falcon",             "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/tiiuae/falcon-7b", },
    {"name": "bert-bge",           "tokt": TOKENIZER_TYPE.WPM,     "repo": "https://huggingface.co/BAAI/bge-small-en-v1.5", },
    {"name": "mpt",                "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/mosaicml/mpt-7b", },
    {"name": "starcoder",          "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/bigcode/starcoder2-3b", },
    {"name": "gpt-2",              "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/openai-community/gpt2", },
    {"name": "stablelm2",          "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/stabilityai/stablelm-2-zephyr-1_6b", },
    {"name": "refact",             "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/smallcloudai/Refact-1_6-base", },
    {"name": "command-r",          "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/CohereForAI/c4ai-command-r-v01", },
    {"name": "qwen2",              "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/Qwen/Qwen1.5-7B", },
    {"name": "olmo",               "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/allenai/OLMo-1.7-7B-hf", },
    {"name": "dbrx",               "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/databricks/dbrx-base", },
    {"name": "jina-v1-en",         "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/jinaai/jina-reranker-v1-tiny-en", },
    {"name": "jina-v2-en",         "tokt": TOKENIZER_TYPE.WPM,     "repo": "https://huggingface.co/jinaai/jina-embeddings-v2-base-en", }, # WPM!
    {"name": "jina-v2-es",         "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/jinaai/jina-embeddings-v2-base-es", },
    {"name": "jina-v2-de",         "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/jinaai/jina-embeddings-v2-base-de", },
    {"name": "smaug-bpe",          "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/abacusai/Smaug-Llama-3-70B-Instruct", },
    {"name": "poro-chat",          "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/LumiOpen/Poro-34B-chat", },
    {"name": "jina-v2-code",       "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/jinaai/jina-embeddings-v2-base-code", },
    {"name": "viking",             "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/LumiOpen/Viking-7B", }, # Also used for Viking 13B and 33B
    {"name": "gemma",              "tokt": TOKENIZER_TYPE.SPM,     "repo": "https://huggingface.co/google/gemma-2b", },
    {"name": "gemma-2",            "tokt": TOKENIZER_TYPE.SPM,     "repo": "https://huggingface.co/google/gemma-2-9b", },
    {"name": "jais",               "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/core42/jais-13b", },
    {"name": "t5",                 "tokt": TOKENIZER_TYPE.UGM,     "repo": "https://huggingface.co/google-t5/t5-small", },
    {"name": "codeshell",          "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/WisdomShell/CodeShell-7B", },
    {"name": "tekken",             "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/mistralai/Mistral-Nemo-Base-2407", },
    {"name": "smollm",             "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/HuggingFaceTB/SmolLM-135M", },
    {'name': "bloom",              "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/bigscience/bloom", },
    {'name': "gpt3-finnish",       "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/TurkuNLP/gpt3-finnish-small", },
    {"name": "exaone",             "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct", },
    {"name": "phi-2",              "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/microsoft/phi-2", },
    {"name": "chameleon",          "tokt": TOKENIZER_TYPE.BPE,     "repo": "https://huggingface.co/facebook/chameleon-7b", },
]


def download_file_with_auth(url, token, save_path):
    headers = {"Authorization": f"Bearer {token}"}
    response = sess.get(url, headers=headers)
    response.raise_for_status()
    os.makedirs(os.path.dirname(save_path), exist_ok=True)
    with open(save_path, 'wb') as downloaded_file:
        downloaded_file.write(response.content)
    logger.info(f"File {save_path} downloaded successfully")


def download_model(model):
    name = model["name"]
    repo = model["repo"]
    tokt = model["tokt"]

    os.makedirs(f"models/tokenizers/{name}", exist_ok=True)

    files = ["config.json", "tokenizer.json", "tokenizer_config.json"]

    if tokt in [TOKENIZER_TYPE.SPM, TOKENIZER_TYPE.SPM_BPE]:
        files.append("tokenizer.model")

    if tokt == TOKENIZER_TYPE.UGM:
        files.append("spiece.model")

    if os.path.isdir(repo):
        # If repo is a path on the file system, copy the directory
        for file in files:
            src_path = os.path.join(repo, file)
            dst_path = f"models/tokenizers/{name}/{file}"
            if os.path.isfile(dst_path):
                logger.info(f"{name}: File {dst_path} already exists - skipping")
                continue
            if os.path.isfile(src_path):
                shutil.copy2(src_path, dst_path)
                logger.info(f"{name}: Copied {src_path} to {dst_path}")
            else:
                logger.warning(f"{name}: Source file {src_path} does not exist")
    else:
        # If repo is a URL, download the files
        for file in files:
            save_path = f"models/tokenizers/{name}/{file}"
            if os.path.isfile(save_path):
                logger.info(f"{name}: File {save_path} already exists - skipping")
                continue
            download_file_with_auth(f"{repo}/resolve/main/{file}", token, save_path)


for model in models:
    try:
        download_model(model)
    except Exception as e:
        logger.error(f"Failed to download model {model['name']}. Error: {e}")


# generate the source code for the convert_hf_to_gguf.py:get_vocab_base_pre() function:

src_ifs = ""
for model in models:
    name = model["name"]
    tokt = model["tokt"]

    if tokt in { TOKENIZER_TYPE.SPM , TOKENIZER_TYPE.UGM }:
        continue

    # Skip if the tokenizer folder does not exist or there are other download issues previously
    if not os.path.exists(f"models/tokenizers/{name}"):
        logger.warning(f"Directory for tokenizer {name} not found. Skipping...")
        continue

    # create the tokenizer
    try:
        if name in { "t5" , "salamandra-spm-bpe" }:
            tokenizer = AutoTokenizer.from_pretrained(f"models/tokenizers/{name}", use_fast=False)
        else:
            tokenizer = AutoTokenizer.from_pretrained(f"models/tokenizers/{name}")
    except OSError as e:
        logger.error(f"Error loading tokenizer for model {name}. The model may not exist or is not accessible with the provided token. Error: {e}")
        continue  # Skip to the next model if the tokenizer can't be loaded

    chktok = tokenizer.encode(CHK_TXT)
    chkhsh = sha256(str(chktok).encode()).hexdigest()

    logger.info(f"model: {name}")
    logger.info(f"tokt: {tokt}")
    logger.info(f"repo: {model['repo']}")
    logger.info(f"chktok: {chktok}")
    logger.info(f"chkhsh: {chkhsh}")

    # print the "pre_tokenizer" content from the tokenizer.json
    with open(f"models/tokenizers/{name}/tokenizer.json", "r", encoding="utf-8") as f:
        cfg = json.load(f)
        normalizer = cfg["normalizer"]
        logger.info("normalizer: " + json.dumps(normalizer, indent=4))
        pre_tokenizer = cfg["pre_tokenizer"]
        logger.info("pre_tokenizer: " + json.dumps(pre_tokenizer, indent=4))
        if "ignore_merges" in cfg["model"]:
            logger.info("ignore_merges: " + json.dumps(cfg["model"]["ignore_merges"], indent=4))

    logger.info("")

    src_ifs += f"        if chkhsh == \"{chkhsh}\":\n"
    src_ifs += f"            # ref: {model['repo']}\n"
    src_ifs += f"            res = \"{name}\"\n"

src_func = f"""
    def get_vocab_base_pre(self, tokenizer) -> str:
        # encoding this string and hashing the resulting tokens would (hopefully) give us a unique identifier that
        # is specific for the BPE pre-tokenizer used by the model
        # we will use this unique identifier to write a "tokenizer.ggml.pre" entry in the GGUF file which we can
        # use in llama.cpp to implement the same pre-tokenizer

        chktxt = {repr(CHK_TXT)}

        chktok = tokenizer.encode(chktxt)
        chkhsh = sha256(str(chktok).encode()).hexdigest()

        logger.debug(f"chktok: {{chktok}}")
        logger.debug(f"chkhsh: {{chkhsh}}")

        res = None

        # NOTE: if you get an error here, you need to update the convert_hf_to_gguf_update.py script
        #       or pull the latest version of the model from Huggingface
        #       don't edit the hashes manually!
{src_ifs}
        if res is None:
            logger.warning("\\n")
            logger.warning("**************************************************************************************")
            logger.warning("** WARNING: The BPE pre-tokenizer was not recognized!")
            logger.warning("**          There are 2 possible reasons for this:")
            logger.warning("**          - the model has not been added to convert_hf_to_gguf_update.py yet")
            logger.warning("**          - the pre-tokenization config has changed upstream")
            logger.warning("**          Check your model files and convert_hf_to_gguf_update.py and update them accordingly.")
            logger.warning("** ref:     https://github.com/ggerganov/llama.cpp/pull/6920")
            logger.warning("**")
            logger.warning(f"** chkhsh:  {{chkhsh}}")
            logger.warning("**************************************************************************************")
            logger.warning("\\n")
            raise NotImplementedError("BPE pre-tokenizer was not recognized - update get_vocab_base_pre()")

        logger.debug(f"tokenizer.ggml.pre: {{repr(res)}}")
        logger.debug(f"chkhsh: {{chkhsh}}")

        return res
"""

convert_py_pth = pathlib.Path("convert_hf_to_gguf.py")
convert_py = convert_py_pth.read_text(encoding="utf-8")
convert_py = re.sub(
    r"(# Marker: Start get_vocab_base_pre)(.+?)( +# Marker: End get_vocab_base_pre)",
    lambda m: m.group(1) + src_func + m.group(3),
    convert_py,
    flags=re.DOTALL | re.MULTILINE,
)

convert_py_pth.write_text(convert_py, encoding="utf-8")

logger.info("+++ convert_hf_to_gguf.py was updated")

# generate tests for each tokenizer model

tests = [
    "ied 4 ½ months",
    "Führer",
    "",
    " ",
    "  ",
    "   ",
    "\t",
    "\n",
    "\n\n",
    "\n\n\n",
    "\t\n",
    "Hello world",
    " Hello world",
    "Hello World",
    " Hello World",
    " Hello World!",
    "Hello, world!",
    " Hello, world!",
    " this is 🦙.cpp",
    "w048 7tuijk dsdfhu",
    "нещо на Български",
    "កាន់តែពិសេសអាចខលចេញ",
    "🚀 (normal) 😶‍🌫️ (multiple emojis concatenated) ✅ (only emoji that has its own token)",
    "Hello",
    " Hello",
    "  Hello",
    "   Hello",
    "    Hello",
    "    Hello\n    Hello",
    " (",
    "\n =",
    "' era",
    "Hello, y'all! How are you 😁 ?我想在apple工作1314151天～",
    "!!!!!!",
    "3",
    "33",
    "333",
    "3333",
    "33333",
    "333333",
    "3333333",
    "33333333",
    "333333333",
    "Cửa Việt", # llama-bpe fails on this
    " discards",
    CHK_TXT,
]

# write the tests to ./models/ggml-vocab-{name}.gguf.inp
# the format is:
#
# test0
# __ggml_vocab_test__
# test1
# __ggml_vocab_test__
# ...
#

# with each model, encode all tests and write the results in ./models/ggml-vocab-{name}.gguf.out
# for each test, write the resulting tokens on a separate line

for model in models:
    name = model["name"]
    tokt = model["tokt"]

    # Skip if the tokenizer folder does not exist or there are other download issues previously
    if not os.path.exists(f"models/tokenizers/{name}"):
        logger.warning(f"Directory for tokenizer {name} not found. Skipping...")
        continue

    # create the tokenizer
    try:
        if name in {"t5", "salamandra-spm-bpe"}:
            tokenizer = AutoTokenizer.from_pretrained(f"models/tokenizers/{name}", use_fast=False)
        else:
            tokenizer = AutoTokenizer.from_pretrained(f"models/tokenizers/{name}")
    except OSError as e:
        logger.error(f"Failed to load tokenizer for model {name}. Error: {e}")
        continue  # Skip this model and continue with the next one in the loop

    with open(f"models/ggml-vocab-{name}.gguf.inp", "w", encoding="utf-8") as f:
        for text in tests:
            f.write(f"{text}")
            f.write("\n__ggml_vocab_test__\n")

    with open(f"models/ggml-vocab-{name}.gguf.out", "w") as f:
        for text in tests:
            res = tokenizer.encode(text, add_special_tokens=False)
            for r in res:
                f.write(f" {r}")
            f.write("\n")

    logger.info(f"Tests for {name} written in ./models/ggml-vocab-{name}.gguf.*")

# generate commands for creating vocab files

logger.info("\nRun the following commands to generate the vocab files for testing:\n")

for model in models:
    name = model["name"]

    print(f"python3 convert_hf_to_gguf.py models/tokenizers/{name}/ --outfile models/ggml-vocab-{name}.gguf --vocab-only") # noqa: NP100

logger.info("\n")