Bug: llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen35'

[toohamster]

Contact Details

449211678@qq.com

What happened?

The qwen3.5 quantization model I downloaded doesn't work. I downloaded it from here: https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF/

Version

llamafile v0.9.3

What operating system are you seeing the problem on?

Mac

Relevant log output

note: if you have an AMD or NVIDIA GPU then you need to pass -ngl 9999 to enable GPU offloading
llama_model_loader: loaded meta data with 46 key-value pairs and 320 tensors from ./Qwen3.5-0.8B-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen35
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen3.5-0.8B
llama_model_loader: - kv   3:                           general.basename str              = Qwen3.5-0.8B
llama_model_loader: - kv   4:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   5:                         general.size_label str              = 0.8B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                       general.license.link str              = https://huggingface.co/Qwen/Qwen3.5-0...
llama_model_loader: - kv   8:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv   9:                   general.base_model.count u32              = 1
llama_model_loader: - kv  10:                  general.base_model.0.name str              = Qwen3.5 0.8B
llama_model_loader: - kv  11:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  12:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen3.5-0.8B
llama_model_loader: - kv  13:                               general.tags arr[str,1]       = ["image-text-to-text"]
llama_model_loader: - kv  14:                         qwen35.block_count u32              = 24
llama_model_loader: - kv  15:                      qwen35.context_length u32              = 262144
llama_model_loader: - kv  16:                    qwen35.embedding_length u32              = 1024
llama_model_loader: - kv  17:                 qwen35.feed_forward_length u32              = 3584
llama_model_loader: - kv  18:                qwen35.attention.head_count u32              = 8
llama_model_loader: - kv  19:             qwen35.attention.head_count_kv u32              = 2
llama_model_loader: - kv  20:             qwen35.rope.dimension_sections arr[i32,4]       = [11, 11, 10, 0]
llama_model_loader: - kv  21:                      qwen35.rope.freq_base f32              = 10000000.000000
llama_model_loader: - kv  22:    qwen35.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  23:                qwen35.attention.key_length u32              = 256
llama_model_loader: - kv  24:              qwen35.attention.value_length u32              = 256
llama_model_loader: - kv  25:                     qwen35.ssm.conv_kernel u32              = 4
llama_model_loader: - kv  26:                      qwen35.ssm.state_size u32              = 128
llama_model_loader: - kv  27:                     qwen35.ssm.group_count u32              = 16
llama_model_loader: - kv  28:                  qwen35.ssm.time_step_rank u32              = 16
llama_model_loader: - kv  29:                      qwen35.ssm.inner_size u32              = 2048
llama_model_loader: - kv  30:             qwen35.full_attention_interval u32              = 4
llama_model_loader: - kv  31:                qwen35.rope.dimension_count u32              = 64
llama_model_loader: - kv  32:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  33:                         tokenizer.ggml.pre str              = qwen35
llama_model_loader: - kv  34:                      tokenizer.ggml.tokens arr[str,248320]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  35:                  tokenizer.ggml.token_type arr[i32,248320]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  36:                      tokenizer.ggml.merges arr[str,247587]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  37:                tokenizer.ggml.eos_token_id u32              = 248046
llama_model_loader: - kv  38:            tokenizer.ggml.padding_token_id u32              = 248055
llama_model_loader: - kv  39:                    tokenizer.chat_template str              = {%- set image_count = namespace(value...
llama_model_loader: - kv  40:               general.quantization_version u32              = 2
llama_model_loader: - kv  41:                          general.file_type u32              = 15
llama_model_loader: - kv  42:                      quantize.imatrix.file str              = Qwen3.5-0.8B-GGUF/imatrix_unsloth.gguf
llama_model_loader: - kv  43:                   quantize.imatrix.dataset str              = unsloth_calibration_Qwen3.5-0.8B.txt
llama_model_loader: - kv  44:             quantize.imatrix.entries_count u32              = 186
llama_model_loader: - kv  45:              quantize.imatrix.chunks_count u32              = 80
llama_model_loader: - type  f32:  133 tensors
llama_model_loader: - type q8_0:   36 tensors
llama_model_loader: - type q4_K:   98 tensors
llama_model_loader: - type q5_K:   36 tensors
llama_model_loader: - type q6_K:   17 tensors
llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen35'
llama_load_model_from_file: failed to load model
./Qwen3.5-0.8B-Q4_K_M.gguf: failed to load model

[aittalam]

Hi @toohamster ! The llamafile version you are running is obsolete and does not support Qwen3.5 models.

This is on us! We have been working on a new version in a branch and, while we are sharing updates about it in the discussion forum and on our discord, we should have shared this information more widely.

The good news is: next week is the official release week 💙 so you will soon be able to test it directly from main!

But if you want to play with it NOW, well, that's understandable 😄 and I have more good news for you: you can build the new llamafile (0.10.0) from the branch, or you can directly download pre-built binaries from here. You can either download just the llamafile binary and run it with whatever gguf file you want, or you can try one of our pre-built llamafiles (the most similar to the one you wanted to test is this one). Just note that if you want to get all the capabilities of the multimodal Qwen3.5 models, you should download both the LLM weights and the mmproj ones and pass them to the executable as follows: llamafile -m model_filename.gguf --mmproj mmproj_filename.gguf. The llamafiles we provide already contain both sets of weights so you should be able to immediately run Qwen3.5 with multimodal capabilities.

Let us know how this works for you! You are among the first people trying our Qwen3.5 models and giving us feedback, so we'd really appreciate knowing if you encounter any issues with that. 🙏

[toohamster]

@aittalam I have downloaded the pre-installed version; however, it is showing a warning that indicates issues.

cuda: dlopen() isn't supported on x86-64 MacOS: failed to load library

sh ./Qwen3.5-0.8B-Q8_0.llamafile -v
cuda: dlopen() isn't supported on x86-64 MacOS: failed to load library
cuda: dlopen() isn't supported on x86-64 MacOS: failed to load library
══════╝╚══════╝╚═╝  ╚═╝╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚══════╝╚══════╝
software: llamafile 0.10.0-dev
model:    Qwen3.5-0.8B-Q8_0.gguf
compute:  Intel Core m3-6Y30 CPU @ 0.90GHz (amd_avx2: AVX2+FMA)
server:   http://127.0.0.1:8080

A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.

[toohamster]

I tried the smaller 300M model, but it didn't support it either. the model from download: https://huggingface.co/unsloth/embeddinggemma-300m-GGUF/tree/main

llama_model_loader: loaded meta data with 35 key-value pairs and 314 tensors from ./embeddinggemma-300M-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma-embedding
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Embeddinggemma 300m Qat Q8_0 Unquantized
llama_model_loader: - kv   3:                           general.finetune str              = qat-unquantized
llama_model_loader: - kv   4:                           general.basename str              = embeddinggemma
llama_model_loader: - kv   5:                         general.size_label str              = 300M
llama_model_loader: - kv   6:                            general.license str              = gemma
llama_model_loader: - kv   7:                               general.tags arr[str,4]       = ["sentence-transformers", "sentence-s...
llama_model_loader: - kv   8:             gemma-embedding.context_length u32              = 2048
llama_model_loader: - kv   9:           gemma-embedding.embedding_length u32              = 768
llama_model_loader: - kv  10:                gemma-embedding.block_count u32              = 24
llama_model_loader: - kv  11:        gemma-embedding.feed_forward_length u32              = 1152
llama_model_loader: - kv  12:       gemma-embedding.attention.head_count u32              = 3
llama_model_loader: - kv  13: gemma-embedding.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  14:       gemma-embedding.attention.key_length u32              = 256
llama_model_loader: - kv  15:     gemma-embedding.attention.value_length u32              = 256
llama_model_loader: - kv  16:                          general.file_type u32              = 7
llama_model_loader: - kv  17:             gemma-embedding.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  18:   gemma-embedding.attention.sliding_window u32              = 512
llama_model_loader: - kv  19:    gemma-embedding.attention.head_count_kv u32              = 1
llama_model_loader: - kv  20:               gemma-embedding.pooling_type u32              = 1
llama_model_loader: - kv  21:               general.quantization_version u32              = 2
llama_model_loader: - kv  22:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  23:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  24:                      tokenizer.ggml.tokens arr[str,262144]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  25:                      tokenizer.ggml.scores arr[f32,262144]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,262144]  = [3, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  27:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  29:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  30:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  32:               tokenizer.ggml.add_eos_token bool             = true
llama_model_loader: - kv  33:               tokenizer.ggml.add_sep_token bool             = false
llama_model_loader: - kv  34:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - type  f32:  145 tensors
llama_model_loader: - type q8_0:  169 tensors
llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'gemma-embedding'
llama_load_model_from_file: failed to load model
./embeddinggemma-300M-Q8_0.gguf: failed to load model

[aittalam]

@aittalam I have downloaded the pre-installed version; however, it is showing a warning that indicates issues.

cuda: dlopen() isn't supported on x86-64 MacOS: failed to load library

sh ./Qwen3.5-0.8B-Q8_0.llamafile -v
cuda: dlopen() isn't supported on x86-64 MacOS: failed to load library
cuda: dlopen() isn't supported on x86-64 MacOS: failed to load library
══════╝╚══════╝╚═╝  ╚═╝╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚══════╝╚══════╝
software: llamafile 0.10.0-dev
model:    Qwen3.5-0.8B-Q8_0.gguf
compute:  Intel Core m3-6Y30 CPU @ 0.90GHz (amd_avx2: AVX2+FMA)
server:   http://127.0.0.1:8080

A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.

Hi @toohamster! llamafile starts anyway, right? It looks like you are running on an Intel, not arm, Mac, thus you cannot rely on Metal GPU. llamafile should work anyway, but relying just on the Mac's CPU instead.

Does it work properly afterwards? My personal Mac is an x86 too, I can test llamafiles on it if there's any need to fix issues there.

[aittalam]

I tried the smaller 300M model, but it didn't support it either. the model from download: https://huggingface.co/unsloth/embeddinggemma-300m-GGUF/tree/main

Can you tell me which version of llamafile you were using to load this? 0.10.0 should be able to load it, I am happy to investigate further if that was the one you tried to run the model with

[toohamster]

@aittalam

我试过较小的 300M 型号，但它也不支持。下载链接：https://huggingface.co/unsloth/embeddinggemma-300m-GGUF/tree/main

请问您用来加载这个模型的 llamafile 版本是多少？0.10.0 版本应该可以加载，如果您尝试运行模型时使用的是这个版本，我很乐意进一步调查。

➜  model-list sh ../Qwen3.5-0.8B-Q8_0.llamafile --model embeddinggemma-300M-Q8_0.gguf --verbose
cuda: dlopen() isn't supported on x86-64 MacOS: failed to load library
cuda: dlopen() isn't supported on x86-64 MacOS: failed to load library
cuda: no pre-built GPU library found
cuda: to enable GPU support, build with:
cuda:   llamafile/cuda.sh   (for NVIDIA)
cuda:   llamafile/rocm.sh   (for AMD)
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (CPU)
server_main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 1773488033 (7f5ee5496) with cosmocc for cosmopolitan
system info: n_threads = 4, n_threads_batch = 4, total_threads = 4

system_info: n_threads = 4 (n_threads_batch = 4) / 4 | CPU : LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

init: using 6 threads for HTTP server
start: binding port with default address family
server_main: loading model
srv    load_model: loading model 'embeddinggemma-300M-Q8_0.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: getting device memory data for initial parameters:
llama_model_loader: loaded meta data with 35 key-value pairs and 314 tensors from embeddinggemma-300M-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma-embedding
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Embeddinggemma 300m Qat Q8_0 Unquantized
llama_model_loader: - kv   3:                           general.finetune str              = qat-unquantized
llama_model_loader: - kv   4:                           general.basename str              = embeddinggemma
llama_model_loader: - kv   5:                         general.size_label str              = 300M
llama_model_loader: - kv   6:                            general.license str              = gemma
llama_model_loader: - kv   7:                               general.tags arr[str,4]       = ["sentence-transformers", "sentence-s...
llama_model_loader: - kv   8:             gemma-embedding.context_length u32              = 2048
llama_model_loader: - kv   9:           gemma-embedding.embedding_length u32              = 768
llama_model_loader: - kv  10:                gemma-embedding.block_count u32              = 24
llama_model_loader: - kv  11:        gemma-embedding.feed_forward_length u32              = 1152
llama_model_loader: - kv  12:       gemma-embedding.attention.head_count u32              = 3
llama_model_loader: - kv  13: gemma-embedding.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  14:       gemma-embedding.attention.key_length u32              = 256
llama_model_loader: - kv  15:     gemma-embedding.attention.value_length u32              = 256
llama_model_loader: - kv  16:                          general.file_type u32              = 7
llama_model_loader: - kv  17:             gemma-embedding.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  18:   gemma-embedding.attention.sliding_window u32              = 512
llama_model_loader: - kv  19:    gemma-embedding.attention.head_count_kv u32              = 1
llama_model_loader: - kv  20:               gemma-embedding.pooling_type u32              = 1
llama_model_loader: - kv  21:               general.quantization_version u32              = 2
llama_model_loader: - kv  22:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  23:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  24:                      tokenizer.ggml.tokens arr[str,262144]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  25:                      tokenizer.ggml.scores arr[f32,262144]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,262144]  = [3, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  27:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  29:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  30:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  32:               tokenizer.ggml.add_eos_token bool             = true
llama_model_loader: - kv  33:               tokenizer.ggml.add_sep_token bool             = false
llama_model_loader: - kv  34:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - type  f32:  145 tensors
llama_model_loader: - type q8_0:  169 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q8_0
print_info: file size   = 307.13 MiB (8.51 BPW)
init_tokenizer: initializing tokenizer for type 1
load: 6242 unused tokens
load: control token: 255999 '<start_of_image>' is not marked as EOG
load: control token:      3 '<unk>' is not marked as EOG
load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
load: control token:      4 '<mask>' is not marked as EOG
load: control token:    105 '<start_of_turn>' is not marked as EOG
load: control token:      2 '<bos>' is not marked as EOG
load: control token: 256000 '<end_of_image>' is not marked as EOG
load: control token:      1 '<eos>' is not marked as EOG
load: control token:      0 '<pad>' is not marked as EOG
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 1 ('<eos>')
load:   - 106 ('<end_of_turn>')
load:   - 212 ('</s>')
load: special tokens cache size = 6414
load: token to piece cache size = 1.9446 MB
print_info: arch                  = gemma-embedding
print_info: vocab_only            = 0
print_info: no_alloc              = 1
print_info: n_ctx_train           = 2048
print_info: n_embd                = 768
print_info: n_embd_inp            = 768
print_info: n_layer               = 24
print_info: n_head                = 3
print_info: n_head_kv             = 1
print_info: n_rot                 = 256
print_info: n_swa                 = 512
print_info: is_swa_any            = 1
print_info: n_embd_head_k         = 256
print_info: n_embd_head_v         = 256
print_info: n_gqa                 = 3
print_info: n_embd_k_gqa          = 256
print_info: n_embd_v_gqa          = 256
print_info: f_norm_eps            = 0.0e+00
print_info: f_norm_rms_eps        = 1.0e-06
print_info: f_clamp_kqv           = 0.0e+00
print_info: f_max_alibi_bias      = 0.0e+00
print_info: f_logit_scale         = 0.0e+00
print_info: f_attn_scale          = 6.2e-02
print_info: n_ff                  = 1152
print_info: n_expert              = 0
print_info: n_expert_used         = 0
print_info: n_expert_groups       = 0
print_info: n_group_used          = 0
print_info: causal attn           = 0
print_info: pooling type          = 1
print_info: rope type             = 2
print_info: rope scaling          = linear
print_info: freq_base_train       = 1000000.0
print_info: freq_scale_train      = 1
print_info: freq_base_swa         = 10000.0
print_info: freq_scale_swa        = 1
print_info: n_ctx_orig_yarn       = 2048
print_info: rope_yarn_log_mul     = 0.0000
print_info: rope_finetuned        = unknown
print_info: model type            = 0.3B
print_info: model params          = 302.86 M
print_info: general.name          = Embeddinggemma 300m Qat Q8_0 Unquantized
print_info: vocab type            = SPM
print_info: n_vocab               = 262144
print_info: n_merges              = 0
print_info: BOS token             = 2 '<bos>'
print_info: EOS token             = 1 '<eos>'
print_info: EOT token             = 106 '<end_of_turn>'
print_info: UNK token             = 3 '<unk>'
print_info: PAD token             = 0 '<pad>'
print_info: LF token              = 248 '<0x0A>'
print_info: EOG token             = 1 '<eos>'
print_info: EOG token             = 106 '<end_of_turn>'
print_info: EOG token             = 212 '</s>'
print_info: max token length      = 48
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
load_tensors: layer   0 assigned to device CPU, is_swa = 1
load_tensors: layer   1 assigned to device CPU, is_swa = 1
load_tensors: layer   2 assigned to device CPU, is_swa = 1
load_tensors: layer   3 assigned to device CPU, is_swa = 1
load_tensors: layer   4 assigned to device CPU, is_swa = 1
load_tensors: layer   5 assigned to device CPU, is_swa = 0
load_tensors: layer   6 assigned to device CPU, is_swa = 1
load_tensors: layer   7 assigned to device CPU, is_swa = 1
load_tensors: layer   8 assigned to device CPU, is_swa = 1
load_tensors: layer   9 assigned to device CPU, is_swa = 1
load_tensors: layer  10 assigned to device CPU, is_swa = 1
load_tensors: layer  11 assigned to device CPU, is_swa = 0
load_tensors: layer  12 assigned to device CPU, is_swa = 1
load_tensors: layer  13 assigned to device CPU, is_swa = 1
load_tensors: layer  14 assigned to device CPU, is_swa = 1
load_tensors: layer  15 assigned to device CPU, is_swa = 1
load_tensors: layer  16 assigned to device CPU, is_swa = 1
load_tensors: layer  17 assigned to device CPU, is_swa = 0
load_tensors: layer  18 assigned to device CPU, is_swa = 1
load_tensors: layer  19 assigned to device CPU, is_swa = 1
load_tensors: layer  20 assigned to device CPU, is_swa = 1
load_tensors: layer  21 assigned to device CPU, is_swa = 1
load_tensors: layer  22 assigned to device CPU, is_swa = 1
load_tensors: layer  23 assigned to device CPU, is_swa = 0
load_tensors: layer  24 assigned to device CPU, is_swa = 0
create_tensor: loading tensor token_embd.weight
create_tensor: loading tensor output_norm.weight
create_tensor: loading tensor blk.0.attn_norm.weight
create_tensor: loading tensor blk.0.attn_q.weight
create_tensor: loading tensor blk.0.attn_k.weight
create_tensor: loading tensor blk.0.attn_v.weight
create_tensor: loading tensor blk.0.attn_output.weight
create_tensor: loading tensor blk.0.post_attention_norm.weight
create_tensor: loading tensor blk.0.attn_k_norm.weight
create_tensor: loading tensor blk.0.attn_q_norm.weight
create_tensor: loading tensor blk.0.ffn_norm.weight
create_tensor: loading tensor blk.0.ffn_gate.weight
create_tensor: loading tensor blk.0.ffn_up.weight
create_tensor: loading tensor blk.0.ffn_down.weight
create_tensor: loading tensor blk.0.post_ffw_norm.weight
create_tensor: loading tensor blk.1.attn_norm.weight
create_tensor: loading tensor blk.1.attn_q.weight
create_tensor: loading tensor blk.1.attn_k.weight
create_tensor: loading tensor blk.1.attn_v.weight
create_tensor: loading tensor blk.1.attn_output.weight
create_tensor: loading tensor blk.1.post_attention_norm.weight
create_tensor: loading tensor blk.1.attn_k_norm.weight
create_tensor: loading tensor blk.1.attn_q_norm.weight
create_tensor: loading tensor blk.1.ffn_norm.weight
create_tensor: loading tensor blk.1.ffn_gate.weight
create_tensor: loading tensor blk.1.ffn_up.weight
create_tensor: loading tensor blk.1.ffn_down.weight
create_tensor: loading tensor blk.1.post_ffw_norm.weight
create_tensor: loading tensor blk.2.attn_norm.weight
create_tensor: loading tensor blk.2.attn_q.weight
create_tensor: loading tensor blk.2.attn_k.weight
create_tensor: loading tensor blk.2.attn_v.weight
create_tensor: loading tensor blk.2.attn_output.weight
create_tensor: loading tensor blk.2.post_attention_norm.weight
create_tensor: loading tensor blk.2.attn_k_norm.weight
create_tensor: loading tensor blk.2.attn_q_norm.weight
create_tensor: loading tensor blk.2.ffn_norm.weight
create_tensor: loading tensor blk.2.ffn_gate.weight
create_tensor: loading tensor blk.2.ffn_up.weight
create_tensor: loading tensor blk.2.ffn_down.weight
create_tensor: loading tensor blk.2.post_ffw_norm.weight
create_tensor: loading tensor blk.3.attn_norm.weight
create_tensor: loading tensor blk.3.attn_q.weight
create_tensor: loading tensor blk.3.attn_k.weight
create_tensor: loading tensor blk.3.attn_v.weight
create_tensor: loading tensor blk.3.attn_output.weight
create_tensor: loading tensor blk.3.post_attention_norm.weight
create_tensor: loading tensor blk.3.attn_k_norm.weight
create_tensor: loading tensor blk.3.attn_q_norm.weight
create_tensor: loading tensor blk.3.ffn_norm.weight
create_tensor: loading tensor blk.3.ffn_gate.weight
create_tensor: loading tensor blk.3.ffn_up.weight
create_tensor: loading tensor blk.3.ffn_down.weight
create_tensor: loading tensor blk.3.post_ffw_norm.weight
create_tensor: loading tensor blk.4.attn_norm.weight
create_tensor: loading tensor blk.4.attn_q.weight
create_tensor: loading tensor blk.4.attn_k.weight
create_tensor: loading tensor blk.4.attn_v.weight
create_tensor: loading tensor blk.4.attn_output.weight
create_tensor: loading tensor blk.4.post_attention_norm.weight
create_tensor: loading tensor blk.4.attn_k_norm.weight
create_tensor: loading tensor blk.4.attn_q_norm.weight
create_tensor: loading tensor blk.4.ffn_norm.weight
create_tensor: loading tensor blk.4.ffn_gate.weight
create_tensor: loading tensor blk.4.ffn_up.weight
create_tensor: loading tensor blk.4.ffn_down.weight
create_tensor: loading tensor blk.4.post_ffw_norm.weight
create_tensor: loading tensor blk.5.attn_norm.weight
create_tensor: loading tensor blk.5.attn_q.weight
create_tensor: loading tensor blk.5.attn_k.weight
create_tensor: loading tensor blk.5.attn_v.weight
create_tensor: loading tensor blk.5.attn_output.weight
create_tensor: loading tensor blk.5.post_attention_norm.weight
create_tensor: loading tensor blk.5.attn_k_norm.weight
create_tensor: loading tensor blk.5.attn_q_norm.weight
create_tensor: loading tensor blk.5.ffn_norm.weight
create_tensor: loading tensor blk.5.ffn_gate.weight
create_tensor: loading tensor blk.5.ffn_up.weight
create_tensor: loading tensor blk.5.ffn_down.weight
create_tensor: loading tensor blk.5.post_ffw_norm.weight
create_tensor: loading tensor blk.6.attn_norm.weight
create_tensor: loading tensor blk.6.attn_q.weight
create_tensor: loading tensor blk.6.attn_k.weight
create_tensor: loading tensor blk.6.attn_v.weight
create_tensor: loading tensor blk.6.attn_output.weight
create_tensor: loading tensor blk.6.post_attention_norm.weight
create_tensor: loading tensor blk.6.attn_k_norm.weight
create_tensor: loading tensor blk.6.attn_q_norm.weight
create_tensor: loading tensor blk.6.ffn_norm.weight
create_tensor: loading tensor blk.6.ffn_gate.weight
create_tensor: loading tensor blk.6.ffn_up.weight
create_tensor: loading tensor blk.6.ffn_down.weight
create_tensor: loading tensor blk.6.post_ffw_norm.weight
create_tensor: loading tensor blk.7.attn_norm.weight
create_tensor: loading tensor blk.7.attn_q.weight
create_tensor: loading tensor blk.7.attn_k.weight
create_tensor: loading tensor blk.7.attn_v.weight
create_tensor: loading tensor blk.7.attn_output.weight
create_tensor: loading tensor blk.7.post_attention_norm.weight
create_tensor: loading tensor blk.7.attn_k_norm.weight
create_tensor: loading tensor blk.7.attn_q_norm.weight
create_tensor: loading tensor blk.7.ffn_norm.weight
create_tensor: loading tensor blk.7.ffn_gate.weight
create_tensor: loading tensor blk.7.ffn_up.weight
create_tensor: loading tensor blk.7.ffn_down.weight
create_tensor: loading tensor blk.7.post_ffw_norm.weight
create_tensor: loading tensor blk.8.attn_norm.weight
create_tensor: loading tensor blk.8.attn_q.weight
create_tensor: loading tensor blk.8.attn_k.weight
create_tensor: loading tensor blk.8.attn_v.weight
create_tensor: loading tensor blk.8.attn_output.weight
create_tensor: loading tensor blk.8.post_attention_norm.weight
create_tensor: loading tensor blk.8.attn_k_norm.weight
create_tensor: loading tensor blk.8.attn_q_norm.weight
create_tensor: loading tensor blk.8.ffn_norm.weight
create_tensor: loading tensor blk.8.ffn_gate.weight
create_tensor: loading tensor blk.8.ffn_up.weight
create_tensor: loading tensor blk.8.ffn_down.weight
create_tensor: loading tensor blk.8.post_ffw_norm.weight
create_tensor: loading tensor blk.9.attn_norm.weight
create_tensor: loading tensor blk.9.attn_q.weight
create_tensor: loading tensor blk.9.attn_k.weight
create_tensor: loading tensor blk.9.attn_v.weight
create_tensor: loading tensor blk.9.attn_output.weight
create_tensor: loading tensor blk.9.post_attention_norm.weight
create_tensor: loading tensor blk.9.attn_k_norm.weight
create_tensor: loading tensor blk.9.attn_q_norm.weight
create_tensor: loading tensor blk.9.ffn_norm.weight
create_tensor: loading tensor blk.9.ffn_gate.weight
create_tensor: loading tensor blk.9.ffn_up.weight
create_tensor: loading tensor blk.9.ffn_down.weight
create_tensor: loading tensor blk.9.post_ffw_norm.weight
create_tensor: loading tensor blk.10.attn_norm.weight
create_tensor: loading tensor blk.10.attn_q.weight
create_tensor: loading tensor blk.10.attn_k.weight
create_tensor: loading tensor blk.10.attn_v.weight
create_tensor: loading tensor blk.10.attn_output.weight
create_tensor: loading tensor blk.10.post_attention_norm.weight
create_tensor: loading tensor blk.10.attn_k_norm.weight
create_tensor: loading tensor blk.10.attn_q_norm.weight
create_tensor: loading tensor blk.10.ffn_norm.weight
create_tensor: loading tensor blk.10.ffn_gate.weight
create_tensor: loading tensor blk.10.ffn_up.weight
create_tensor: loading tensor blk.10.ffn_down.weight
create_tensor: loading tensor blk.10.post_ffw_norm.weight
create_tensor: loading tensor blk.11.attn_norm.weight
create_tensor: loading tensor blk.11.attn_q.weight
create_tensor: loading tensor blk.11.attn_k.weight
create_tensor: loading tensor blk.11.attn_v.weight
create_tensor: loading tensor blk.11.attn_output.weight
create_tensor: loading tensor blk.11.post_attention_norm.weight
create_tensor: loading tensor blk.11.attn_k_norm.weight
create_tensor: loading tensor blk.11.attn_q_norm.weight
create_tensor: loading tensor blk.11.ffn_norm.weight
create_tensor: loading tensor blk.11.ffn_gate.weight
create_tensor: loading tensor blk.11.ffn_up.weight
create_tensor: loading tensor blk.11.ffn_down.weight
create_tensor: loading tensor blk.11.post_ffw_norm.weight
create_tensor: loading tensor blk.12.attn_norm.weight
create_tensor: loading tensor blk.12.attn_q.weight
create_tensor: loading tensor blk.12.attn_k.weight
create_tensor: loading tensor blk.12.attn_v.weight
create_tensor: loading tensor blk.12.attn_output.weight
create_tensor: loading tensor blk.12.post_attention_norm.weight
create_tensor: loading tensor blk.12.attn_k_norm.weight
create_tensor: loading tensor blk.12.attn_q_norm.weight
create_tensor: loading tensor blk.12.ffn_norm.weight
create_tensor: loading tensor blk.12.ffn_gate.weight
create_tensor: loading tensor blk.12.ffn_up.weight
create_tensor: loading tensor blk.12.ffn_down.weight
create_tensor: loading tensor blk.12.post_ffw_norm.weight
create_tensor: loading tensor blk.13.attn_norm.weight
create_tensor: loading tensor blk.13.attn_q.weight
create_tensor: loading tensor blk.13.attn_k.weight
create_tensor: loading tensor blk.13.attn_v.weight
create_tensor: loading tensor blk.13.attn_output.weight
create_tensor: loading tensor blk.13.post_attention_norm.weight
create_tensor: loading tensor blk.13.attn_k_norm.weight
create_tensor: loading tensor blk.13.attn_q_norm.weight
create_tensor: loading tensor blk.13.ffn_norm.weight
create_tensor: loading tensor blk.13.ffn_gate.weight
create_tensor: loading tensor blk.13.ffn_up.weight
create_tensor: loading tensor blk.13.ffn_down.weight
create_tensor: loading tensor blk.13.post_ffw_norm.weight
create_tensor: loading tensor blk.14.attn_norm.weight
create_tensor: loading tensor blk.14.attn_q.weight
create_tensor: loading tensor blk.14.attn_k.weight
create_tensor: loading tensor blk.14.attn_v.weight
create_tensor: loading tensor blk.14.attn_output.weight
create_tensor: loading tensor blk.14.post_attention_norm.weight
create_tensor: loading tensor blk.14.attn_k_norm.weight
create_tensor: loading tensor blk.14.attn_q_norm.weight
create_tensor: loading tensor blk.14.ffn_norm.weight
create_tensor: loading tensor blk.14.ffn_gate.weight
create_tensor: loading tensor blk.14.ffn_up.weight
create_tensor: loading tensor blk.14.ffn_down.weight
create_tensor: loading tensor blk.14.post_ffw_norm.weight
create_tensor: loading tensor blk.15.attn_norm.weight
create_tensor: loading tensor blk.15.attn_q.weight
create_tensor: loading tensor blk.15.attn_k.weight
create_tensor: loading tensor blk.15.attn_v.weight
create_tensor: loading tensor blk.15.attn_output.weight
create_tensor: loading tensor blk.15.post_attention_norm.weight
create_tensor: loading tensor blk.15.attn_k_norm.weight
create_tensor: loading tensor blk.15.attn_q_norm.weight
create_tensor: loading tensor blk.15.ffn_norm.weight
create_tensor: loading tensor blk.15.ffn_gate.weight
create_tensor: loading tensor blk.15.ffn_up.weight
create_tensor: loading tensor blk.15.ffn_down.weight
create_tensor: loading tensor blk.15.post_ffw_norm.weight
create_tensor: loading tensor blk.16.attn_norm.weight
create_tensor: loading tensor blk.16.attn_q.weight
create_tensor: loading tensor blk.16.attn_k.weight
create_tensor: loading tensor blk.16.attn_v.weight
create_tensor: loading tensor blk.16.attn_output.weight
create_tensor: loading tensor blk.16.post_attention_norm.weight
create_tensor: loading tensor blk.16.attn_k_norm.weight
create_tensor: loading tensor blk.16.attn_q_norm.weight
create_tensor: loading tensor blk.16.ffn_norm.weight
create_tensor: loading tensor blk.16.ffn_gate.weight
create_tensor: loading tensor blk.16.ffn_up.weight
create_tensor: loading tensor blk.16.ffn_down.weight
create_tensor: loading tensor blk.16.post_ffw_norm.weight
create_tensor: loading tensor blk.17.attn_norm.weight
create_tensor: loading tensor blk.17.attn_q.weight
create_tensor: loading tensor blk.17.attn_k.weight
create_tensor: loading tensor blk.17.attn_v.weight
create_tensor: loading tensor blk.17.attn_output.weight
create_tensor: loading tensor blk.17.post_attention_norm.weight
create_tensor: loading tensor blk.17.attn_k_norm.weight
create_tensor: loading tensor blk.17.attn_q_norm.weight
create_tensor: loading tensor blk.17.ffn_norm.weight
create_tensor: loading tensor blk.17.ffn_gate.weight
create_tensor: loading tensor blk.17.ffn_up.weight
create_tensor: loading tensor blk.17.ffn_down.weight
create_tensor: loading tensor blk.17.post_ffw_norm.weight
create_tensor: loading tensor blk.18.attn_norm.weight
create_tensor: loading tensor blk.18.attn_q.weight
create_tensor: loading tensor blk.18.attn_k.weight
create_tensor: loading tensor blk.18.attn_v.weight
create_tensor: loading tensor blk.18.attn_output.weight
create_tensor: loading tensor blk.18.post_attention_norm.weight
create_tensor: loading tensor blk.18.attn_k_norm.weight
create_tensor: loading tensor blk.18.attn_q_norm.weight
create_tensor: loading tensor blk.18.ffn_norm.weight
create_tensor: loading tensor blk.18.ffn_gate.weight
create_tensor: loading tensor blk.18.ffn_up.weight
create_tensor: loading tensor blk.18.ffn_down.weight
create_tensor: loading tensor blk.18.post_ffw_norm.weight
create_tensor: loading tensor blk.19.attn_norm.weight
create_tensor: loading tensor blk.19.attn_q.weight
create_tensor: loading tensor blk.19.attn_k.weight
create_tensor: loading tensor blk.19.attn_v.weight
create_tensor: loading tensor blk.19.attn_output.weight
create_tensor: loading tensor blk.19.post_attention_norm.weight
create_tensor: loading tensor blk.19.attn_k_norm.weight
create_tensor: loading tensor blk.19.attn_q_norm.weight
create_tensor: loading tensor blk.19.ffn_norm.weight
create_tensor: loading tensor blk.19.ffn_gate.weight
create_tensor: loading tensor blk.19.ffn_up.weight
create_tensor: loading tensor blk.19.ffn_down.weight
create_tensor: loading tensor blk.19.post_ffw_norm.weight
create_tensor: loading tensor blk.20.attn_norm.weight
create_tensor: loading tensor blk.20.attn_q.weight
create_tensor: loading tensor blk.20.attn_k.weight
create_tensor: loading tensor blk.20.attn_v.weight
create_tensor: loading tensor blk.20.attn_output.weight
create_tensor: loading tensor blk.20.post_attention_norm.weight
create_tensor: loading tensor blk.20.attn_k_norm.weight
create_tensor: loading tensor blk.20.attn_q_norm.weight
create_tensor: loading tensor blk.20.ffn_norm.weight
create_tensor: loading tensor blk.20.ffn_gate.weight
create_tensor: loading tensor blk.20.ffn_up.weight
create_tensor: loading tensor blk.20.ffn_down.weight
create_tensor: loading tensor blk.20.post_ffw_norm.weight
create_tensor: loading tensor blk.21.attn_norm.weight
create_tensor: loading tensor blk.21.attn_q.weight
create_tensor: loading tensor blk.21.attn_k.weight
create_tensor: loading tensor blk.21.attn_v.weight
create_tensor: loading tensor blk.21.attn_output.weight
create_tensor: loading tensor blk.21.post_attention_norm.weight
create_tensor: loading tensor blk.21.attn_k_norm.weight
create_tensor: loading tensor blk.21.attn_q_norm.weight
create_tensor: loading tensor blk.21.ffn_norm.weight
create_tensor: loading tensor blk.21.ffn_gate.weight
create_tensor: loading tensor blk.21.ffn_up.weight
create_tensor: loading tensor blk.21.ffn_down.weight
create_tensor: loading tensor blk.21.post_ffw_norm.weight
create_tensor: loading tensor blk.22.attn_norm.weight
create_tensor: loading tensor blk.22.attn_q.weight
create_tensor: loading tensor blk.22.attn_k.weight
create_tensor: loading tensor blk.22.attn_v.weight
create_tensor: loading tensor blk.22.attn_output.weight
create_tensor: loading tensor blk.22.post_attention_norm.weight
create_tensor: loading tensor blk.22.attn_k_norm.weight
create_tensor: loading tensor blk.22.attn_q_norm.weight
create_tensor: loading tensor blk.22.ffn_norm.weight
create_tensor: loading tensor blk.22.ffn_gate.weight
create_tensor: loading tensor blk.22.ffn_up.weight
create_tensor: loading tensor blk.22.ffn_down.weight
create_tensor: loading tensor blk.22.post_ffw_norm.weight
create_tensor: loading tensor blk.23.attn_norm.weight
create_tensor: loading tensor blk.23.attn_q.weight
create_tensor: loading tensor blk.23.attn_k.weight
create_tensor: loading tensor blk.23.attn_v.weight
create_tensor: loading tensor blk.23.attn_output.weight
create_tensor: loading tensor blk.23.post_attention_norm.weight
create_tensor: loading tensor blk.23.attn_k_norm.weight
create_tensor: loading tensor blk.23.attn_q_norm.weight
create_tensor: loading tensor blk.23.ffn_norm.weight
create_tensor: loading tensor blk.23.ffn_gate.weight
create_tensor: loading tensor blk.23.ffn_up.weight
create_tensor: loading tensor blk.23.ffn_down.weight
create_tensor: loading tensor blk.23.post_ffw_norm.weight
load_tensors: tensor 'token_embd.weight' (q8_0) (and 314 others) cannot be used with preferred buffer type CPU_REPACK, using CPU instead
load_tensors:          CPU model buffer size =     0.00 MiB
llama_context: constructing llama_context
llama_context: n_seq_max     = 4
llama_context: n_ctx         = 16384
llama_context: n_ctx_seq     = 16384
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 0
llama_context: flash_attn    = auto
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (16384) > n_ctx_train (2048) -- possible training context overflow
set_abort_callback: call
llama_context:        CPU  output buffer size =     4.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 1
sched_reserve: reserving ...
sched_reserve: max_nodes = 2512
sched_reserve: worst-case: n_tokens = 512, n_seqs = 4, n_outputs = 4
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  4, n_outputs =    4
graph_reserve: making n_tokens a multiple of n_seqs - n_tokens = 4, n_seqs = 4, n_outputs = 4
sched_reserve: Flash Attention was auto, set to enabled
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  4, n_outputs =  512
graph_reserve: reserving a graph for ubatch with n_tokens =    4, n_seqs =  4, n_outputs =    4
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  4, n_outputs =  512
sched_reserve:        CPU compute buffer size =    15.76 MiB
sched_reserve: graph nodes  = 872
sched_reserve: graph splits = 1
sched_reserve: reserve took 3.39 ms, sched copies = 1
llama_memory_breakdown_print: | memory breakdown [MiB] | total   free    self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - Host               |                  322 =   307 +       0 +      15                |
llama_params_fit_impl: no devices with dedicated memory found
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.35 seconds
llama_model_loader: loaded meta data with 35 key-value pairs and 314 tensors from embeddinggemma-300M-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma-embedding
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Embeddinggemma 300m Qat Q8_0 Unquantized
llama_model_loader: - kv   3:                           general.finetune str              = qat-unquantized
llama_model_loader: - kv   4:                           general.basename str              = embeddinggemma
llama_model_loader: - kv   5:                         general.size_label str              = 300M
llama_model_loader: - kv   6:                            general.license str              = gemma
llama_model_loader: - kv   7:                               general.tags arr[str,4]       = ["sentence-transformers", "sentence-s...
llama_model_loader: - kv   8:             gemma-embedding.context_length u32              = 2048
llama_model_loader: - kv   9:           gemma-embedding.embedding_length u32              = 768
llama_model_loader: - kv  10:                gemma-embedding.block_count u32              = 24
llama_model_loader: - kv  11:        gemma-embedding.feed_forward_length u32              = 1152
llama_model_loader: - kv  12:       gemma-embedding.attention.head_count u32              = 3
llama_model_loader: - kv  13: gemma-embedding.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  14:       gemma-embedding.attention.key_length u32              = 256
llama_model_loader: - kv  15:     gemma-embedding.attention.value_length u32              = 256
llama_model_loader: - kv  16:                          general.file_type u32              = 7
llama_model_loader: - kv  17:             gemma-embedding.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  18:   gemma-embedding.attention.sliding_window u32              = 512
llama_model_loader: - kv  19:    gemma-embedding.attention.head_count_kv u32              = 1
llama_model_loader: - kv  20:               gemma-embedding.pooling_type u32              = 1
llama_model_loader: - kv  21:               general.quantization_version u32              = 2
llama_model_loader: - kv  22:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  23:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  24:                      tokenizer.ggml.tokens arr[str,262144]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  25:                      tokenizer.ggml.scores arr[f32,262144]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,262144]  = [3, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  27:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  29:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  30:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  32:               tokenizer.ggml.add_eos_token bool             = true
llama_model_loader: - kv  33:               tokenizer.ggml.add_sep_token bool             = false
llama_model_loader: - kv  34:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - type  f32:  145 tensors
llama_model_loader: - type q8_0:  169 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q8_0
print_info: file size   = 307.13 MiB (8.51 BPW)
init_tokenizer: initializing tokenizer for type 1
load: 6242 unused tokens
load: control token: 255999 '<start_of_image>' is not marked as EOG
load: control token:      3 '<unk>' is not marked as EOG
load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
load: control token:      4 '<mask>' is not marked as EOG
load: control token:    105 '<start_of_turn>' is not marked as EOG
load: control token:      2 '<bos>' is not marked as EOG
load: control token: 256000 '<end_of_image>' is not marked as EOG
load: control token:      1 '<eos>' is not marked as EOG
load: control token:      0 '<pad>' is not marked as EOG
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 1 ('<eos>')
load:   - 106 ('<end_of_turn>')
load:   - 212 ('</s>')
load: special tokens cache size = 6414
load: token to piece cache size = 1.9446 MB
print_info: arch                  = gemma-embedding
print_info: vocab_only            = 0
print_info: no_alloc              = 0
print_info: n_ctx_train           = 2048
print_info: n_embd                = 768
print_info: n_embd_inp            = 768
print_info: n_layer               = 24
print_info: n_head                = 3
print_info: n_head_kv             = 1
print_info: n_rot                 = 256
print_info: n_swa                 = 512
print_info: is_swa_any            = 1
print_info: n_embd_head_k         = 256
print_info: n_embd_head_v         = 256
print_info: n_gqa                 = 3
print_info: n_embd_k_gqa          = 256
print_info: n_embd_v_gqa          = 256
print_info: f_norm_eps            = 0.0e+00
print_info: f_norm_rms_eps        = 1.0e-06
print_info: f_clamp_kqv           = 0.0e+00
print_info: f_max_alibi_bias      = 0.0e+00
print_info: f_logit_scale         = 0.0e+00
print_info: f_attn_scale          = 6.2e-02
print_info: n_ff                  = 1152
print_info: n_expert              = 0
print_info: n_expert_used         = 0
print_info: n_expert_groups       = 0
print_info: n_group_used          = 0
print_info: causal attn           = 0
print_info: pooling type          = 1
print_info: rope type             = 2
print_info: rope scaling          = linear
print_info: freq_base_train       = 1000000.0
print_info: freq_scale_train      = 1
print_info: freq_base_swa         = 10000.0
print_info: freq_scale_swa        = 1
print_info: n_ctx_orig_yarn       = 2048
print_info: rope_yarn_log_mul     = 0.0000
print_info: rope_finetuned        = unknown
print_info: model type            = 0.3B
print_info: model params          = 302.86 M
print_info: general.name          = Embeddinggemma 300m Qat Q8_0 Unquantized
print_info: vocab type            = SPM
print_info: n_vocab               = 262144
print_info: n_merges              = 0
print_info: BOS token             = 2 '<bos>'
print_info: EOS token             = 1 '<eos>'
print_info: EOT token             = 106 '<end_of_turn>'
print_info: UNK token             = 3 '<unk>'
print_info: PAD token             = 0 '<pad>'
print_info: LF token              = 248 '<0x0A>'
print_info: EOG token             = 1 '<eos>'
print_info: EOG token             = 106 '<end_of_turn>'
print_info: EOG token             = 212 '</s>'
print_info: max token length      = 48
load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
load_tensors: layer   0 assigned to device CPU, is_swa = 1
load_tensors: layer   1 assigned to device CPU, is_swa = 1
load_tensors: layer   2 assigned to device CPU, is_swa = 1
load_tensors: layer   3 assigned to device CPU, is_swa = 1
load_tensors: layer   4 assigned to device CPU, is_swa = 1
load_tensors: layer   5 assigned to device CPU, is_swa = 0
load_tensors: layer   6 assigned to device CPU, is_swa = 1
load_tensors: layer   7 assigned to device CPU, is_swa = 1
load_tensors: layer   8 assigned to device CPU, is_swa = 1
load_tensors: layer   9 assigned to device CPU, is_swa = 1
load_tensors: layer  10 assigned to device CPU, is_swa = 1
load_tensors: layer  11 assigned to device CPU, is_swa = 0
load_tensors: layer  12 assigned to device CPU, is_swa = 1
load_tensors: layer  13 assigned to device CPU, is_swa = 1
load_tensors: layer  14 assigned to device CPU, is_swa = 1
load_tensors: layer  15 assigned to device CPU, is_swa = 1
load_tensors: layer  16 assigned to device CPU, is_swa = 1
load_tensors: layer  17 assigned to device CPU, is_swa = 0
load_tensors: layer  18 assigned to device CPU, is_swa = 1
load_tensors: layer  19 assigned to device CPU, is_swa = 1
load_tensors: layer  20 assigned to device CPU, is_swa = 1
load_tensors: layer  21 assigned to device CPU, is_swa = 1
load_tensors: layer  22 assigned to device CPU, is_swa = 1
load_tensors: layer  23 assigned to device CPU, is_swa = 0
load_tensors: layer  24 assigned to device CPU, is_swa = 0
create_tensor: loading tensor token_embd.weight
create_tensor: loading tensor output_norm.weight
create_tensor: loading tensor blk.0.attn_norm.weight
create_tensor: loading tensor blk.0.attn_q.weight
create_tensor: loading tensor blk.0.attn_k.weight
create_tensor: loading tensor blk.0.attn_v.weight
create_tensor: loading tensor blk.0.attn_output.weight
create_tensor: loading tensor blk.0.post_attention_norm.weight
create_tensor: loading tensor blk.0.attn_k_norm.weight
create_tensor: loading tensor blk.0.attn_q_norm.weight
create_tensor: loading tensor blk.0.ffn_norm.weight
create_tensor: loading tensor blk.0.ffn_gate.weight
create_tensor: loading tensor blk.0.ffn_up.weight
create_tensor: loading tensor blk.0.ffn_down.weight
create_tensor: loading tensor blk.0.post_ffw_norm.weight
create_tensor: loading tensor blk.1.attn_norm.weight
create_tensor: loading tensor blk.1.attn_q.weight
create_tensor: loading tensor blk.1.attn_k.weight
create_tensor: loading tensor blk.1.attn_v.weight
create_tensor: loading tensor blk.1.attn_output.weight
create_tensor: loading tensor blk.1.post_attention_norm.weight
create_tensor: loading tensor blk.1.attn_k_norm.weight
create_tensor: loading tensor blk.1.attn_q_norm.weight
create_tensor: loading tensor blk.1.ffn_norm.weight
create_tensor: loading tensor blk.1.ffn_gate.weight
create_tensor: loading tensor blk.1.ffn_up.weight
create_tensor: loading tensor blk.1.ffn_down.weight
create_tensor: loading tensor blk.1.post_ffw_norm.weight
create_tensor: loading tensor blk.2.attn_norm.weight
create_tensor: loading tensor blk.2.attn_q.weight
create_tensor: loading tensor blk.2.attn_k.weight
create_tensor: loading tensor blk.2.attn_v.weight
create_tensor: loading tensor blk.2.attn_output.weight
create_tensor: loading tensor blk.2.post_attention_norm.weight
create_tensor: loading tensor blk.2.attn_k_norm.weight
create_tensor: loading tensor blk.2.attn_q_norm.weight
create_tensor: loading tensor blk.2.ffn_norm.weight
create_tensor: loading tensor blk.2.ffn_gate.weight
create_tensor: loading tensor blk.2.ffn_up.weight
create_tensor: loading tensor blk.2.ffn_down.weight
create_tensor: loading tensor blk.2.post_ffw_norm.weight
create_tensor: loading tensor blk.3.attn_norm.weight
create_tensor: loading tensor blk.3.attn_q.weight
create_tensor: loading tensor blk.3.attn_k.weight
create_tensor: loading tensor blk.3.attn_v.weight
create_tensor: loading tensor blk.3.attn_output.weight
create_tensor: loading tensor blk.3.post_attention_norm.weight
create_tensor: loading tensor blk.3.attn_k_norm.weight
create_tensor: loading tensor blk.3.attn_q_norm.weight
create_tensor: loading tensor blk.3.ffn_norm.weight
create_tensor: loading tensor blk.3.ffn_gate.weight
create_tensor: loading tensor blk.3.ffn_up.weight
create_tensor: loading tensor blk.3.ffn_down.weight
create_tensor: loading tensor blk.3.post_ffw_norm.weight
create_tensor: loading tensor blk.4.attn_norm.weight
create_tensor: loading tensor blk.4.attn_q.weight
create_tensor: loading tensor blk.4.attn_k.weight
create_tensor: loading tensor blk.4.attn_v.weight
create_tensor: loading tensor blk.4.attn_output.weight
create_tensor: loading tensor blk.4.post_attention_norm.weight
create_tensor: loading tensor blk.4.attn_k_norm.weight
create_tensor: loading tensor blk.4.attn_q_norm.weight
create_tensor: loading tensor blk.4.ffn_norm.weight
create_tensor: loading tensor blk.4.ffn_gate.weight
create_tensor: loading tensor blk.4.ffn_up.weight
create_tensor: loading tensor blk.4.ffn_down.weight
create_tensor: loading tensor blk.4.post_ffw_norm.weight
create_tensor: loading tensor blk.5.attn_norm.weight
create_tensor: loading tensor blk.5.attn_q.weight
create_tensor: loading tensor blk.5.attn_k.weight
create_tensor: loading tensor blk.5.attn_v.weight
create_tensor: loading tensor blk.5.attn_output.weight
create_tensor: loading tensor blk.5.post_attention_norm.weight
create_tensor: loading tensor blk.5.attn_k_norm.weight
create_tensor: loading tensor blk.5.attn_q_norm.weight
create_tensor: loading tensor blk.5.ffn_norm.weight
create_tensor: loading tensor blk.5.ffn_gate.weight
create_tensor: loading tensor blk.5.ffn_up.weight
create_tensor: loading tensor blk.5.ffn_down.weight
create_tensor: loading tensor blk.5.post_ffw_norm.weight
create_tensor: loading tensor blk.6.attn_norm.weight
create_tensor: loading tensor blk.6.attn_q.weight
create_tensor: loading tensor blk.6.attn_k.weight
create_tensor: loading tensor blk.6.attn_v.weight
create_tensor: loading tensor blk.6.attn_output.weight
create_tensor: loading tensor blk.6.post_attention_norm.weight
create_tensor: loading tensor blk.6.attn_k_norm.weight
create_tensor: loading tensor blk.6.attn_q_norm.weight
create_tensor: loading tensor blk.6.ffn_norm.weight
create_tensor: loading tensor blk.6.ffn_gate.weight
create_tensor: loading tensor blk.6.ffn_up.weight
create_tensor: loading tensor blk.6.ffn_down.weight
create_tensor: loading tensor blk.6.post_ffw_norm.weight
create_tensor: loading tensor blk.7.attn_norm.weight
create_tensor: loading tensor blk.7.attn_q.weight
create_tensor: loading tensor blk.7.attn_k.weight
create_tensor: loading tensor blk.7.attn_v.weight
create_tensor: loading tensor blk.7.attn_output.weight
create_tensor: loading tensor blk.7.post_attention_norm.weight
create_tensor: loading tensor blk.7.attn_k_norm.weight
create_tensor: loading tensor blk.7.attn_q_norm.weight
create_tensor: loading tensor blk.7.ffn_norm.weight
create_tensor: loading tensor blk.7.ffn_gate.weight
create_tensor: loading tensor blk.7.ffn_up.weight
create_tensor: loading tensor blk.7.ffn_down.weight
create_tensor: loading tensor blk.7.post_ffw_norm.weight
create_tensor: loading tensor blk.8.attn_norm.weight
create_tensor: loading tensor blk.8.attn_q.weight
create_tensor: loading tensor blk.8.attn_k.weight
create_tensor: loading tensor blk.8.attn_v.weight
create_tensor: loading tensor blk.8.attn_output.weight
create_tensor: loading tensor blk.8.post_attention_norm.weight
create_tensor: loading tensor blk.8.attn_k_norm.weight
create_tensor: loading tensor blk.8.attn_q_norm.weight
create_tensor: loading tensor blk.8.ffn_norm.weight
create_tensor: loading tensor blk.8.ffn_gate.weight
create_tensor: loading tensor blk.8.ffn_up.weight
create_tensor: loading tensor blk.8.ffn_down.weight
create_tensor: loading tensor blk.8.post_ffw_norm.weight
create_tensor: loading tensor blk.9.attn_norm.weight
create_tensor: loading tensor blk.9.attn_q.weight
create_tensor: loading tensor blk.9.attn_k.weight
create_tensor: loading tensor blk.9.attn_v.weight
create_tensor: loading tensor blk.9.attn_output.weight
create_tensor: loading tensor blk.9.post_attention_norm.weight
create_tensor: loading tensor blk.9.attn_k_norm.weight
create_tensor: loading tensor blk.9.attn_q_norm.weight
create_tensor: loading tensor blk.9.ffn_norm.weight
create_tensor: loading tensor blk.9.ffn_gate.weight
create_tensor: loading tensor blk.9.ffn_up.weight
create_tensor: loading tensor blk.9.ffn_down.weight
create_tensor: loading tensor blk.9.post_ffw_norm.weight
create_tensor: loading tensor blk.10.attn_norm.weight
create_tensor: loading tensor blk.10.attn_q.weight
create_tensor: loading tensor blk.10.attn_k.weight
create_tensor: loading tensor blk.10.attn_v.weight
create_tensor: loading tensor blk.10.attn_output.weight
create_tensor: loading tensor blk.10.post_attention_norm.weight
create_tensor: loading tensor blk.10.attn_k_norm.weight
create_tensor: loading tensor blk.10.attn_q_norm.weight
create_tensor: loading tensor blk.10.ffn_norm.weight
create_tensor: loading tensor blk.10.ffn_gate.weight
create_tensor: loading tensor blk.10.ffn_up.weight
create_tensor: loading tensor blk.10.ffn_down.weight
create_tensor: loading tensor blk.10.post_ffw_norm.weight
create_tensor: loading tensor blk.11.attn_norm.weight
create_tensor: loading tensor blk.11.attn_q.weight
create_tensor: loading tensor blk.11.attn_k.weight
create_tensor: loading tensor blk.11.attn_v.weight
create_tensor: loading tensor blk.11.attn_output.weight
create_tensor: loading tensor blk.11.post_attention_norm.weight
create_tensor: loading tensor blk.11.attn_k_norm.weight
create_tensor: loading tensor blk.11.attn_q_norm.weight
create_tensor: loading tensor blk.11.ffn_norm.weight
create_tensor: loading tensor blk.11.ffn_gate.weight
create_tensor: loading tensor blk.11.ffn_up.weight
create_tensor: loading tensor blk.11.ffn_down.weight
create_tensor: loading tensor blk.11.post_ffw_norm.weight
create_tensor: loading tensor blk.12.attn_norm.weight
create_tensor: loading tensor blk.12.attn_q.weight
create_tensor: loading tensor blk.12.attn_k.weight
create_tensor: loading tensor blk.12.attn_v.weight
create_tensor: loading tensor blk.12.attn_output.weight
create_tensor: loading tensor blk.12.post_attention_norm.weight
create_tensor: loading tensor blk.12.attn_k_norm.weight
create_tensor: loading tensor blk.12.attn_q_norm.weight
create_tensor: loading tensor blk.12.ffn_norm.weight
create_tensor: loading tensor blk.12.ffn_gate.weight
create_tensor: loading tensor blk.12.ffn_up.weight
create_tensor: loading tensor blk.12.ffn_down.weight
create_tensor: loading tensor blk.12.post_ffw_norm.weight
create_tensor: loading tensor blk.13.attn_norm.weight
create_tensor: loading tensor blk.13.attn_q.weight
create_tensor: loading tensor blk.13.attn_k.weight
create_tensor: loading tensor blk.13.attn_v.weight
create_tensor: loading tensor blk.13.attn_output.weight
create_tensor: loading tensor blk.13.post_attention_norm.weight
create_tensor: loading tensor blk.13.attn_k_norm.weight
create_tensor: loading tensor blk.13.attn_q_norm.weight
create_tensor: loading tensor blk.13.ffn_norm.weight
create_tensor: loading tensor blk.13.ffn_gate.weight
create_tensor: loading tensor blk.13.ffn_up.weight
create_tensor: loading tensor blk.13.ffn_down.weight
create_tensor: loading tensor blk.13.post_ffw_norm.weight
create_tensor: loading tensor blk.14.attn_norm.weight
create_tensor: loading tensor blk.14.attn_q.weight
create_tensor: loading tensor blk.14.attn_k.weight
create_tensor: loading tensor blk.14.attn_v.weight
create_tensor: loading tensor blk.14.attn_output.weight
create_tensor: loading tensor blk.14.post_attention_norm.weight
create_tensor: loading tensor blk.14.attn_k_norm.weight
create_tensor: loading tensor blk.14.attn_q_norm.weight
create_tensor: loading tensor blk.14.ffn_norm.weight
create_tensor: loading tensor blk.14.ffn_gate.weight
create_tensor: loading tensor blk.14.ffn_up.weight
create_tensor: loading tensor blk.14.ffn_down.weight
create_tensor: loading tensor blk.14.post_ffw_norm.weight
create_tensor: loading tensor blk.15.attn_norm.weight
create_tensor: loading tensor blk.15.attn_q.weight
create_tensor: loading tensor blk.15.attn_k.weight
create_tensor: loading tensor blk.15.attn_v.weight
create_tensor: loading tensor blk.15.attn_output.weight
create_tensor: loading tensor blk.15.post_attention_norm.weight
create_tensor: loading tensor blk.15.attn_k_norm.weight
create_tensor: loading tensor blk.15.attn_q_norm.weight
create_tensor: loading tensor blk.15.ffn_norm.weight
create_tensor: loading tensor blk.15.ffn_gate.weight
create_tensor: loading tensor blk.15.ffn_up.weight
create_tensor: loading tensor blk.15.ffn_down.weight
create_tensor: loading tensor blk.15.post_ffw_norm.weight
create_tensor: loading tensor blk.16.attn_norm.weight
create_tensor: loading tensor blk.16.attn_q.weight
create_tensor: loading tensor blk.16.attn_k.weight
create_tensor: loading tensor blk.16.attn_v.weight
create_tensor: loading tensor blk.16.attn_output.weight
create_tensor: loading tensor blk.16.post_attention_norm.weight
create_tensor: loading tensor blk.16.attn_k_norm.weight
create_tensor: loading tensor blk.16.attn_q_norm.weight
create_tensor: loading tensor blk.16.ffn_norm.weight
create_tensor: loading tensor blk.16.ffn_gate.weight
create_tensor: loading tensor blk.16.ffn_up.weight
create_tensor: loading tensor blk.16.ffn_down.weight
create_tensor: loading tensor blk.16.post_ffw_norm.weight
create_tensor: loading tensor blk.17.attn_norm.weight
create_tensor: loading tensor blk.17.attn_q.weight
create_tensor: loading tensor blk.17.attn_k.weight
create_tensor: loading tensor blk.17.attn_v.weight
create_tensor: loading tensor blk.17.attn_output.weight
create_tensor: loading tensor blk.17.post_attention_norm.weight
create_tensor: loading tensor blk.17.attn_k_norm.weight
create_tensor: loading tensor blk.17.attn_q_norm.weight
create_tensor: loading tensor blk.17.ffn_norm.weight
create_tensor: loading tensor blk.17.ffn_gate.weight
create_tensor: loading tensor blk.17.ffn_up.weight
create_tensor: loading tensor blk.17.ffn_down.weight
create_tensor: loading tensor blk.17.post_ffw_norm.weight
create_tensor: loading tensor blk.18.attn_norm.weight
create_tensor: loading tensor blk.18.attn_q.weight
create_tensor: loading tensor blk.18.attn_k.weight
create_tensor: loading tensor blk.18.attn_v.weight
create_tensor: loading tensor blk.18.attn_output.weight
create_tensor: loading tensor blk.18.post_attention_norm.weight
create_tensor: loading tensor blk.18.attn_k_norm.weight
create_tensor: loading tensor blk.18.attn_q_norm.weight
create_tensor: loading tensor blk.18.ffn_norm.weight
create_tensor: loading tensor blk.18.ffn_gate.weight
create_tensor: loading tensor blk.18.ffn_up.weight
create_tensor: loading tensor blk.18.ffn_down.weight
create_tensor: loading tensor blk.18.post_ffw_norm.weight
create_tensor: loading tensor blk.19.attn_norm.weight
create_tensor: loading tensor blk.19.attn_q.weight
create_tensor: loading tensor blk.19.attn_k.weight
create_tensor: loading tensor blk.19.attn_v.weight
create_tensor: loading tensor blk.19.attn_output.weight
create_tensor: loading tensor blk.19.post_attention_norm.weight
create_tensor: loading tensor blk.19.attn_k_norm.weight
create_tensor: loading tensor blk.19.attn_q_norm.weight
create_tensor: loading tensor blk.19.ffn_norm.weight
create_tensor: loading tensor blk.19.ffn_gate.weight
create_tensor: loading tensor blk.19.ffn_up.weight
create_tensor: loading tensor blk.19.ffn_down.weight
create_tensor: loading tensor blk.19.post_ffw_norm.weight
create_tensor: loading tensor blk.20.attn_norm.weight
create_tensor: loading tensor blk.20.attn_q.weight
create_tensor: loading tensor blk.20.attn_k.weight
create_tensor: loading tensor blk.20.attn_v.weight
create_tensor: loading tensor blk.20.attn_output.weight
create_tensor: loading tensor blk.20.post_attention_norm.weight
create_tensor: loading tensor blk.20.attn_k_norm.weight
create_tensor: loading tensor blk.20.attn_q_norm.weight
create_tensor: loading tensor blk.20.ffn_norm.weight
create_tensor: loading tensor blk.20.ffn_gate.weight
create_tensor: loading tensor blk.20.ffn_up.weight
create_tensor: loading tensor blk.20.ffn_down.weight
create_tensor: loading tensor blk.20.post_ffw_norm.weight
create_tensor: loading tensor blk.21.attn_norm.weight
create_tensor: loading tensor blk.21.attn_q.weight
create_tensor: loading tensor blk.21.attn_k.weight
create_tensor: loading tensor blk.21.attn_v.weight
create_tensor: loading tensor blk.21.attn_output.weight
create_tensor: loading tensor blk.21.post_attention_norm.weight
create_tensor: loading tensor blk.21.attn_k_norm.weight
create_tensor: loading tensor blk.21.attn_q_norm.weight
create_tensor: loading tensor blk.21.ffn_norm.weight
create_tensor: loading tensor blk.21.ffn_gate.weight
create_tensor: loading tensor blk.21.ffn_up.weight
create_tensor: loading tensor blk.21.ffn_down.weight
create_tensor: loading tensor blk.21.post_ffw_norm.weight
create_tensor: loading tensor blk.22.attn_norm.weight
create_tensor: loading tensor blk.22.attn_q.weight
create_tensor: loading tensor blk.22.attn_k.weight
create_tensor: loading tensor blk.22.attn_v.weight
create_tensor: loading tensor blk.22.attn_output.weight
create_tensor: loading tensor blk.22.post_attention_norm.weight
create_tensor: loading tensor blk.22.attn_k_norm.weight
create_tensor: loading tensor blk.22.attn_q_norm.weight
create_tensor: loading tensor blk.22.ffn_norm.weight
create_tensor: loading tensor blk.22.ffn_gate.weight
create_tensor: loading tensor blk.22.ffn_up.weight
create_tensor: loading tensor blk.22.ffn_down.weight
create_tensor: loading tensor blk.22.post_ffw_norm.weight
create_tensor: loading tensor blk.23.attn_norm.weight
create_tensor: loading tensor blk.23.attn_q.weight
create_tensor: loading tensor blk.23.attn_k.weight
create_tensor: loading tensor blk.23.attn_v.weight
create_tensor: loading tensor blk.23.attn_output.weight
create_tensor: loading tensor blk.23.post_attention_norm.weight
create_tensor: loading tensor blk.23.attn_k_norm.weight
create_tensor: loading tensor blk.23.attn_q_norm.weight
create_tensor: loading tensor blk.23.ffn_norm.weight
create_tensor: loading tensor blk.23.ffn_gate.weight
create_tensor: loading tensor blk.23.ffn_up.weight
create_tensor: loading tensor blk.23.ffn_down.weight
create_tensor: loading tensor blk.23.post_ffw_norm.weight
load_tensors: tensor 'token_embd.weight' (q8_0) (and 314 others) cannot be used with preferred buffer type CPU_REPACK, using CPU instead
load_tensors:   CPU_Mapped model buffer size =   307.13 MiB
...................................
common_init_result: added <eos> logit bias = -inf
common_init_result: added <end_of_turn> logit bias = -inf
common_init_result: added </s> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max     = 4
llama_context: n_ctx         = 16384
llama_context: n_ctx_seq     = 16384
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 0
llama_context: flash_attn    = auto
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (16384) > n_ctx_train (2048) -- possible training context overflow
set_abort_callback: call
llama_context:        CPU  output buffer size =     4.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 1
sched_reserve: reserving ...
sched_reserve: max_nodes = 2512
sched_reserve: worst-case: n_tokens = 512, n_seqs = 4, n_outputs = 4
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  4, n_outputs =    4
graph_reserve: making n_tokens a multiple of n_seqs - n_tokens = 4, n_seqs = 4, n_outputs = 4
sched_reserve: Flash Attention was auto, set to enabled
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  4, n_outputs =  512
graph_reserve: reserving a graph for ubatch with n_tokens =    4, n_seqs =  4, n_outputs =    4
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  4, n_outputs =  512
sched_reserve:        CPU compute buffer size =    15.76 MiB
sched_reserve: graph nodes  = 872
sched_reserve: graph splits = 1
sched_reserve: reserve took 3.53 ms, sched copies = 1
set_adapters_lora: adapters = 0
adapters_lora_are_same: adapters = 0
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
set_warmup: value = 1
decode: cannot decode batches with this context (calling encode() instead)
llama.cpp/ggml/src/ggml.c:1283: assert(ne % ggml_blck_size(type) == 0) failed (cosmoaddr2line /Users/xuyimu/Downloads/llama/model-list/../Qwen3.5-0.8B-Q8_0.llamafile 6cdd6f d36246 d83833 c9da45 c7c550 6d841f 6da6e4 6dab56 e5ccc1 e3a0f9 e426d8 e31a12 6d9da1 706664 6ea1aa 6ed2c1 7f528f 7f4ebb 7f5611 7f6005 7fa19c)
llama.cpp/ggml/src/ggml.c:1283: assert(ne % ggml_blck_size(type) == 0) failed (cosmoaddr2line /Users/xuyimu/Downloads/llama/model-list/../Qwen3.5-0.8B-Q8_0.llamafile 6cdd6f d36246 d83833 c9da45 c7c550 6d841f 6da6e4 6dab56 e5ccc1 e3a0f9 e3c624 e59da3 e8fb45 ea0495 0)
llama.cpp/ggml/src/ggml.c:1283: assert(ne % ggml_blck_size(type) == 0) failed (cosmoaddr2line /Users/xuyimu/Downloads/llama/model-list/../Qwen3.5-0.8B-Q8_0.llamafile 6cdd6f d36246 d83833 c9da45 c7c550 6d841f 6da6e4 6dab56 e5ccc1 e3a0f9 e3c624 e59da3 e8fb45 ea0495 0)
llama.cpp/ggml/src/ggml.c:1283: assert(ne % ggml_blck_size(type) == 0) failed (cosmoaddr2line /Users/xuyimu/Downloads/llama/model-list/../Qwen3.5-0.8B-Q8_0.llamafile 6cdd6f d36246 d83833 c9da45 c7c550 6d841f 6da6e4 6dab56 e5ccc1 e3a0f9 e3c624 e59da3 e8fb45 ea0495 0)
[1]    31895 abort      sh ../Qwen3.5-0.8B-Q8_0.llamafile --model embeddinggemma-300M-Q8_0.gguf

[toohamster]

@aittalam
I tried the smaller model, but it didn't support it either. the model from download:
https://huggingface.co/ggml-org/GLM-OCR-GGUF/tree/main

➜  model-list sh ../Qwen3.5-0.8B-Q8_0.llamafile --model mmproj-GLM-OCR-Q8_0.gguf
cuda: dlopen() isn't supported on x86-64 MacOS: failed to load library
cuda: dlopen() isn't supported on x86-64 MacOS: failed to load library
llama_model_load: error loading model: CLIP cannot be used as main model, use it with --mmproj instead
llama_model_load_from_file_impl: failed to load model
llama_params_fit: encountered an error while trying to fit params to free device memory: failed to load model
llama_model_load: error loading model: CLIP cannot be used as main model, use it with --mmproj instead
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'mmproj-GLM-OCR-Q8_0.gguf'
srv    load_model: failed to load model, 'mmproj-GLM-OCR-Q8_0.gguf'
server_main: exiting due to model loading error
➜  model-list sh ../Qwen3.5-0.8B-Q8_0.llamafile --model mmproj-GLM-OCR-Q8_0.gguf --verbose
cuda: dlopen() isn't supported on x86-64 MacOS: failed to load library
cuda: dlopen() isn't supported on x86-64 MacOS: failed to load library
cuda: no pre-built GPU library found
cuda: to enable GPU support, build with:
cuda:   llamafile/cuda.sh   (for NVIDIA)
cuda:   llamafile/rocm.sh   (for AMD)
register_backend: registered backend CPU (1 devices)
register_device: registered device CPU (CPU)
server_main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 1773488033 (7f5ee5496) with cosmocc for cosmopolitan
system info: n_threads = 4, n_threads_batch = 4, total_threads = 4

system_info: n_threads = 4 (n_threads_batch = 4) / 4 | CPU : LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

init: using 6 threads for HTTP server
start: binding port with default address family
server_main: loading model
srv    load_model: loading model 'mmproj-GLM-OCR-Q8_0.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: getting device memory data for initial parameters:
llama_model_loader: loaded meta data with 21 key-value pairs and 348 tensors from mmproj-GLM-OCR-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = clip
llama_model_loader: - kv   1:                               general.type str              = mmproj
llama_model_loader: - kv   2:                         general.size_label str              = 434M
llama_model_loader: - kv   3:                            general.license str              = mit
llama_model_loader: - kv   4:                               general.tags arr[str,1]       = ["image-to-text"]
llama_model_loader: - kv   5:                          general.languages arr[str,8]       = ["zh", "en", "fr", "es", "ru", "de", ...
llama_model_loader: - kv   6:                          general.file_type u32              = 7
llama_model_loader: - kv   7:                    clip.has_vision_encoder bool             = true
llama_model_loader: - kv   8:                 clip.vision.projection_dim u32              = 1536
llama_model_loader: - kv   9:                     clip.vision.image_size u32              = 336
llama_model_loader: - kv  10:                     clip.vision.patch_size u32              = 14
llama_model_loader: - kv  11:               clip.vision.embedding_length u32              = 1024
llama_model_loader: - kv  12:            clip.vision.feed_forward_length u32              = 4096
llama_model_loader: - kv  13:                    clip.vision.block_count u32              = 24
llama_model_loader: - kv  14:           clip.vision.attention.head_count u32              = 16
llama_model_loader: - kv  15:                     clip.vision.image_mean arr[f32,3]       = [0.481455, 0.457828, 0.408211]
llama_model_loader: - kv  16:                      clip.vision.image_std arr[f32,3]       = [0.268630, 0.261303, 0.275777]
llama_model_loader: - kv  17:                        clip.projector_type str              = glm4v
llama_model_loader: - kv  18:                              clip.use_silu bool             = true
llama_model_loader: - kv  19:   clip.vision.attention.layer_norm_epsilon f32              = 0.000010
llama_model_loader: - kv  20:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  224 tensors
llama_model_loader: - type q8_0:  124 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q8_0
print_info: file size   = 461.94 MiB (8.93 BPW)
llama_model_load: error loading model: CLIP cannot be used as main model, use it with --mmproj instead
llama_model_load_from_file_impl: failed to load model
llama_params_fit: encountered an error while trying to fit params to free device memory: failed to load model
llama_params_fit: fitting params to free memory took 0.01 seconds
llama_model_loader: loaded meta data with 21 key-value pairs and 348 tensors from mmproj-GLM-OCR-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = clip
llama_model_loader: - kv   1:                               general.type str              = mmproj
llama_model_loader: - kv   2:                         general.size_label str              = 434M
llama_model_loader: - kv   3:                            general.license str              = mit
llama_model_loader: - kv   4:                               general.tags arr[str,1]       = ["image-to-text"]
llama_model_loader: - kv   5:                          general.languages arr[str,8]       = ["zh", "en", "fr", "es", "ru", "de", ...
llama_model_loader: - kv   6:                          general.file_type u32              = 7
llama_model_loader: - kv   7:                    clip.has_vision_encoder bool             = true
llama_model_loader: - kv   8:                 clip.vision.projection_dim u32              = 1536
llama_model_loader: - kv   9:                     clip.vision.image_size u32              = 336
llama_model_loader: - kv  10:                     clip.vision.patch_size u32              = 14
llama_model_loader: - kv  11:               clip.vision.embedding_length u32              = 1024
llama_model_loader: - kv  12:            clip.vision.feed_forward_length u32              = 4096
llama_model_loader: - kv  13:                    clip.vision.block_count u32              = 24
llama_model_loader: - kv  14:           clip.vision.attention.head_count u32              = 16
llama_model_loader: - kv  15:                     clip.vision.image_mean arr[f32,3]       = [0.481455, 0.457828, 0.408211]
llama_model_loader: - kv  16:                      clip.vision.image_std arr[f32,3]       = [0.268630, 0.261303, 0.275777]
llama_model_loader: - kv  17:                        clip.projector_type str              = glm4v
llama_model_loader: - kv  18:                              clip.use_silu bool             = true
llama_model_loader: - kv  19:   clip.vision.attention.layer_norm_epsilon f32              = 0.000010
llama_model_loader: - kv  20:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  224 tensors
llama_model_loader: - type q8_0:  124 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q8_0
print_info: file size   = 461.94 MiB (8.93 BPW)
llama_model_load: error loading model: CLIP cannot be used as main model, use it with --mmproj instead
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'mmproj-GLM-OCR-Q8_0.gguf'
srv    load_model: failed to load model, 'mmproj-GLM-OCR-Q8_0.gguf'
srv    operator(): operator(): cleaning up before exit...
server_main: exiting due to model loading error

[aittalam]

Hi @toohamster , and sorry for the late reply!

Related to your previous tests:

sh ./Qwen3.5-0.8B-Q8_0.llamafile -v

I tested this on my Intel MacBook and worked properly after the warnings (which are ok,
they just say CUDA acceleration is not available). This Mac has 16GB RAM so I can load
the model with its default context size (which is 16K tokens). If you have any problem fitting
the model in your RAM, you can try a smaller context size (e.g. adding --ctx-size 4096 to
your command). The model runs quite slowly on my hardware, i.e. less than 3 tok/sec.

sh ../Qwen3.5-0.8B-Q8_0.llamafile --model embeddinggemma-300M-Q8_0.gguf

In this case you are providing an embedding model, which is supported by llamafile 0.10.0
but will not work together with the pre-packaged Qwen3.5-0.8B llamafile. The reason is
that this llamafile is pre-configured to run a multimodal model and will try to mix Qwen's
mmproj weights with Gemma's embedding ones. The correct way to run embeddings
with llamafile is the following:

• download the small llamafile executable (e.g. this one)
• run it as llamafile -m embeddinggemma-300m-q8_0.gguf --embeddings --server

Note that in this case you will not run a chat, but rather an API that allows you to
calculate sentence embeddings. You can test it with e.g.

curl -H "Content-Type: application/json" -X POST  "http://localhost:8080/embedding" --data '{"content":["some text to embed","some other text"]}'

Last but not least:

sh ../Qwen3.5-0.8B-Q8_0.llamafile --model mmproj-GLM-OCR-Q8_0.gguf

In this case, you are passing llamafile the mmproj weights of another model (GLM-OCR) as model weights. If you wanted to run GLM-OCR you should pass the weights as follows:

./llamafile -m GLM-OCR-Q8_0.gguf --mmproj mmproj-GLM-OCR-Q8_0.gguf

# or if you have Qwen3.5 you can also do this:
./Qwen3.5-0.8B-Q8_0.llamafile -m GLM-OCR-Q8_0.gguf --mmproj mmproj-GLM-OCR-Q8_0.gguf

... if I got it right, though, one of your main issues is the model size, is that correct?
One suggestion I could give you here is to try and use a more heavily quantized version of Qwen3.5-0.8B.
You can check the available quants here, choose one (e.g. https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF/blob/main/Qwen3.5-0.8B-Q4_K_M.gguf), then call the llamafile server by passing the model weight only:

./llamafile -m Qwen3.5-0.8B-Q4_K_M.gguf

Without the mmproj weights you won't have multimodal capabilities, but you'll have more RAM available for your context (and the Q4 quants might be a bit faster).

[prdpklyn]

its same with gemma4 as well

llama-server -hf ggml-org/gemma-4-26B-A4B-it-GGUF:Q4_K_M \
        --ctx-size 32768 \                           
    -ngl 99 \
    -fa on\
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 64

llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'gemma4'
llama_model_load_from_file_impl: failed to load model

[aittalam]

its same with gemma4 as well

llamafile should not support gemma4 yet, at least not until the next sync with upstream llama.cpp.

However, the fact that you are running with -hf makes me think you might not be using llamafile: while we build llama-server as an APE too, we do not support https for downloading models on-the-fly so this command should return a different error.

Feel free to open an issue asking for gemma4 support here, and when we sync with a llama.cpp version supporting it we'll close the issue and you will be notified about it :-) I'll close this issue in the meantime, as llamafile properly supports Qwen3.5.