Eval bug: Incorrect VLM image description on Android after commit 4db5641

[forforever73]

Name and Version

zorn:/data/local/tmp/llama.cpp/install_gpu $ LD_LIBRARY_PATH=lib ./bin/llama-mtmd-cli --version
ggml_opencl: selected platform: 'QUALCOMM Snapdragon(TM)'

ggml_opencl: device: 'QUALCOMM Adreno(TM) 750 (OpenCL 3.0 Adreno(TM) 750)'
ggml_opencl: OpenCL driver: OpenCL 3.0 QUALCOMM build: commit unknown Compiler E031.45.02.16
ggml_opencl: vector subgroup broadcast support: false
ggml_opencl: device FP16 support: true
ggml_opencl: mem base addr align: 128
ggml_opencl: max mem alloc size: 1024 MB
ggml_opencl: device max workgroup size: 1024
ggml_opencl: SVM coarse grain buffer support: true
ggml_opencl: SVM fine grain buffer support: true
ggml_opencl: SVM fine grain system support: false
ggml_opencl: SVM atomics support: true
ggml_opencl: flattening quantized weights representation as struct of arrays (GGML_OPENCL_SOA_Q)
ggml_opencl: using kernels optimized for Adreno (GGML_OPENCL_USE_ADRENO_KERNELS)
ggml_opencl: loading OpenCL kernels.........................................................................
ggml_opencl: default device: 'QUALCOMM Adreno(TM) 750 (OpenCL 3.0 Adreno(TM) 750)'
version: 7079 (416e7c7)
built with Android (13989888, +pgo, -bolt, +lto, -mlgo, based on r563880c) clang version 21.0.0 (https://android.googlesource.com/toolchain/llvm-project 5e96669f06077099aa41290cdb4c5e6fa0f59349) for arm64-apple-darwin24.6.0

Operating systems

Other? (Please let us know in description), Linux

GGML backends

OpenCL

Hardware

QUALCOMM Adreno(TM) 750

Models

Qwen2.5-VL-3B-Instruct-Q4_0.gguf

Problem description & steps to reproduce

When running llama-mtmd-cli on an Android device to evaluate the Qwen2.5-VL-3B model, the generated description of an input image becomes completely incorrect. For example, when providing an image of a tree, the model outputs unrelated or nonsensical text instead of a proper description.

After reverting commit 4db5641, the output becomes correct again. This suggests that this commit introduces a regression affecting VLM evaluation. @shaofeiqi

The correct description of tree.jpg like this
The image depicts a close-up view of a tree trunk with several distinct features. The bark of the tree is rough and textured, with a mix of dark and light brown patches indicating variations in the tree's growth patterns or possibly different types of bark. There are several vertical lines running along the length of the trunk, which
but now
The image displays a collection of black and white photographs arranged in a grid format. Each photograph appears to be a close-up or portrait of an individual, though the faces are blurred and indistinguishable. The individuals seem to be wearing formal attire, suggesting that these images might be from a professional or formal event. The background

First Bad Commit

4db5641

Relevant log output

zorn:/data/local/tmp/llama.cpp/install_gpu $ LD_LIBRARY_PATH=lib ./bin/llama-mtmd-cli -m ../Qwen2.5-VL-3B-Instruct-Q4_0.gguf --mmproj ../mmproj-Qwen2.5-VL-3B-Instruct -c 8192 --temp 1 --top-p 0.8 --top-k 1 --repeat-penalty 1.05 -n 64 --image ../tree.jpg -p "Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detailIntroduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in" -ngl 99
llama_perf_context_print:        load time =    4298.45 ms
llama_perf_context_print: prompt eval time =   12418.37 ms /   511 tokens (   24.30 ms per token,    41.15 tokens per second)
llama_perf_context_print:        eval time =    5573.83 ms /    63 runs   (   88.47 ms per token,    11.30 tokens per second)
llama_perf_context_print:       total time =   19217.85 ms /   574 tokens
llama_perf_context_print:    graphs reused =          0
re detailIntroduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in the image in more detail.Introduce the content in" -ngl 99                                                                                                                        <
ggml_opencl: selected platform: 'QUALCOMM Snapdragon(TM)'

ggml_opencl: device: 'QUALCOMM Adreno(TM) 750 (OpenCL 3.0 Adreno(TM) 750)'
ggml_opencl: OpenCL driver: OpenCL 3.0 QUALCOMM build: commit unknown Compiler E031.45.02.16
ggml_opencl: vector subgroup broadcast support: false
ggml_opencl: device FP16 support: true
ggml_opencl: mem base addr align: 128
ggml_opencl: max mem alloc size: 1024 MB
ggml_opencl: device max workgroup size: 1024
ggml_opencl: SVM coarse grain buffer support: true
ggml_opencl: SVM fine grain buffer support: true
ggml_opencl: SVM fine grain system support: false
ggml_opencl: SVM atomics support: true
ggml_opencl: flattening quantized weights representation as struct of arrays (GGML_OPENCL_SOA_Q)
ggml_opencl: using kernels optimized for Adreno (GGML_OPENCL_USE_ADRENO_KERNELS)
ggml_opencl: loading OpenCL kernels.........................................................................
ggml_opencl: default device: 'QUALCOMM Adreno(TM) 750 (OpenCL 3.0 Adreno(TM) 750)'
build: 7079 (416e7c7f4) with Android (13989888, +pgo, -bolt, +lto, -mlgo, based on r563880c) clang version 21.0.0 (https://android.googlesource.com/toolchain/llvm-project 5e96669f06077099aa41290cdb4c5e6fa0f59349) for arm64-apple-darwin24.6.0
llama_model_load_from_file_impl: using device GPUOpenCL (QUALCOMM Adreno(TM) 750) (unknown id) - 0 MiB free
llama_model_loader: loaded meta data with 27 key-value pairs and 434 tensors from ../Qwen2.5-VL-3B-Instruct-Q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2vl
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 VL 3B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Qwen2.5-VL
llama_model_loader: - kv   5:                         general.size_label str              = 3B
llama_model_loader: - kv   6:                        qwen2vl.block_count u32              = 36
llama_model_loader: - kv   7:                     qwen2vl.context_length u32              = 128000
llama_model_loader: - kv   8:                   qwen2vl.embedding_length u32              = 2048
llama_model_loader: - kv   9:                qwen2vl.feed_forward_length u32              = 11008
llama_model_loader: - kv  10:               qwen2vl.attention.head_count u32              = 16
llama_model_loader: - kv  11:            qwen2vl.attention.head_count_kv u32              = 2
llama_model_loader: - kv  12:                     qwen2vl.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  13:   qwen2vl.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  14:            qwen2vl.rope.dimension_sections arr[i32,4]       = [16, 24, 24, 0]
llama_model_loader: - kv  15:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  16:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  17:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  18:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  19:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  22:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  23:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {% set image_count = namespace(value=...
llama_model_loader: - kv  25:               general.quantization_version u32              = 2
llama_model_loader: - kv  26:                          general.file_type u32              = 2
llama_model_loader: - type  f32:  181 tensors
llama_model_loader: - type q4_0:  253 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_0
print_info: file size   = 1.62 GiB (4.50 BPW)
load: printing all EOG tokens:
load:   - 151643 ('<|endoftext|>')
load:   - 151645 ('<|im_end|>')
load:   - 151662 ('<|fim_pad|>')
load:   - 151663 ('<|repo_name|>')
load:   - 151664 ('<|file_sep|>')
load: special tokens cache size = 22
load: token to piece cache size = 0.9310 MB
print_info: arch             = qwen2vl
print_info: vocab_only       = 0
print_info: n_ctx_train      = 128000
print_info: n_embd           = 2048
print_info: n_embd_inp       = 2048
print_info: n_layer          = 36
print_info: n_head           = 16
print_info: n_head_kv        = 2
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 256
print_info: n_embd_v_gqa     = 256
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 11008
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = -1
print_info: rope type        = 8
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 128000
print_info: rope_finetuned   = unknown
print_info: mrope sections   = [16, 24, 24, 0]
print_info: model type       = 3B
print_info: model params     = 3.09 B
print_info: general.name     = Qwen2.5 VL 3B Instruct
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 151643 '<|endoftext|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 36 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 37/37 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   166.92 MiB
load_tensors:       OpenCL model buffer size =  1656.22 MiB
....................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 8192
llama_context: n_ctx_seq     = 8192
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = false
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (8192) < n_ctx_train (128000) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.58 MiB
llama_kv_cache:     OpenCL KV buffer size =   288.00 MiB
llama_kv_cache: size =  288.00 MiB (  8192 cells,  36 layers,  1/1 seqs), K (f16):  144.00 MiB, V (f16):  144.00 MiB
llama_context: Flash Attention was auto, set to enabled
llama_context:     OpenCL compute buffer size =   300.75 MiB
llama_context:        CPU compute buffer size =    20.02 MiB
llama_context: graph nodes  = 1231
llama_context: graph splits = 2
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|im_end|> logit bias = -inf
common_init_from_params: added <|fim_pad|> logit bias = -inf
common_init_from_params: added <|repo_name|> logit bias = -inf
common_init_from_params: added <|file_sep|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 8192
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
mtmd_cli_context: chat template example:
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

clip_model_loader: model name:   Qwen2.5 VL 3B Instruct
clip_model_loader: description:
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    519
clip_model_loader: n_kv:         22

clip_model_loader: has vision encoder
clip_ctx: CLIP using OpenCL backend
load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

load_hparams: projector:          qwen2.5vl_merger
load_hparams: n_embd:             1280
load_hparams: n_head:             16
load_hparams: n_ff:               3420
load_hparams: n_layer:            32
load_hparams: ffn_op:             silu
load_hparams: projection_dim:     2048

--- vision hparams ---
load_hparams: image_size:         560
load_hparams: patch_size:         14
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   0
load_hparams: n_merge:            2
load_hparams: n_wa_pattern:       8
load_hparams: image_min_pixels:   6272
load_hparams: image_max_pixels:   3211264

load_hparams: model size:         1276.39 MiB
load_hparams: metadata size:      0.18 MiB
alloc_compute_meta: warmup with image size = 1288 x 1288
alloc_compute_meta:     OpenCL compute buffer size =   732.56 MiB
alloc_compute_meta:        CPU compute buffer size =   292.41 MiB
alloc_compute_meta: graph splits = 1, nodes = 1092
warmup: flash attention is enabled
main: loading model: ../Qwen2.5-VL-3B-Instruct-Q4_0.gguf
encoding image slice...
image slice encoded in 9483 ms
decoding image batch 1/1, n_tokens_batch = 256
image decoded (batch 1/1) in 46 ms

The image displays a collection of black and white photographs arranged in a grid format. Each photograph appears to be a close-up or portrait of an individual, though the faces are blurred and indistinguishable. The individuals seem to be wearing formal attire, suggesting that these images might be from a professional or formal event. The background

llama_perf_context_print:        load time =    4309.91 ms
llama_perf_context_print: prompt eval time =   13083.05 ms /   511 tokens (   25.60 ms per token,    39.06 tokens per second)
llama_perf_context_print:        eval time =    5580.93 ms /    63 runs   (   88.59 ms per token,    11.29 tokens per second)
llama_perf_context_print:       total time =   19915.24 ms /   574 tokens
llama_perf_context_print:    graphs reused =          0

[lhez]

@forforever73 What are the width and height of tree.jpg? Is it possible for you to share this image file? In addition, what is the quantization of mmproj (Q4_0 or simply f16/f32)?

I grabbed a random image of tree with leaves and it seems to give me sensible description for my setup. My mmproj is in f32 and this could make a difference so want to check with yours.

ggml_opencl: selected platform: 'QUALCOMM Snapdragon(TM)'

ggml_opencl: device: 'QUALCOMM Adreno(TM) 750 (OpenCL 3.0 Adreno(TM) 750)'
ggml_opencl: OpenCL driver: OpenCL 3.0 QUALCOMM build: commit unknown Compiler E031.45.02.16
ggml_opencl: vector subgroup broadcast support: false
ggml_opencl: device FP16 support: true
ggml_opencl: mem base addr align: 128
ggml_opencl: max mem alloc size: 1024 MB
ggml_opencl: device max workgroup size: 1024
ggml_opencl: SVM coarse grain buffer support: true
ggml_opencl: SVM fine grain buffer support: true
ggml_opencl: SVM fine grain system support: false
ggml_opencl: SVM atomics support: true
ggml_opencl: flattening quantized weights representation as struct of arrays (GGML_OPENCL_SOA_Q)
ggml_opencl: using kernels optimized for Adreno (GGML_OPENCL_USE_ADRENO_KERNELS)
ggml_opencl: loading OpenCL kernels.........................................................................
ggml_opencl: default device: 'QUALCOMM Adreno(TM) 750 (OpenCL 3.0 Adreno(TM) 750)'
build: 7083 (2376b7758) with Android (13324770, +pgo, +bolt, +lto, +mlgo, based on r530567d) clang version 19.0.0 (https://android.googlesource.com/toolchain/llvm-project 97a699bf4812a18fb657c2779f5296a4ab2694d2) for x86_64-unknown-linux-gnu
llama_model_load_from_file_impl: using device GPUOpenCL (QUALCOMM Adreno(TM) 750) (unknown id) - 0 MiB free
llama_model_loader: loaded meta data with 27 key-value pairs and 434 tensors from ../llama.cpp-models/qwen2.5-vl-3b-instruct-Q4_0_pure.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2vl
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 VL 3B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Qwen2.5-VL
llama_model_loader: - kv   5:                         general.size_label str              = 3B
llama_model_loader: - kv   6:                        qwen2vl.block_count u32              = 36
llama_model_loader: - kv   7:                     qwen2vl.context_length u32              = 128000
llama_model_loader: - kv   8:                   qwen2vl.embedding_length u32              = 2048
llama_model_loader: - kv   9:                qwen2vl.feed_forward_length u32              = 11008
llama_model_loader: - kv  10:               qwen2vl.attention.head_count u32              = 16
llama_model_loader: - kv  11:            qwen2vl.attention.head_count_kv u32              = 2
llama_model_loader: - kv  12:                     qwen2vl.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  13:   qwen2vl.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  14:            qwen2vl.rope.dimension_sections arr[i32,4]       = [16, 24, 24, 0]
llama_model_loader: - kv  15:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  16:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  17:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  18:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  19:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  22:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  23:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {% set image_count = namespace(value=...
llama_model_loader: - kv  25:               general.quantization_version u32              = 2
llama_model_loader: - kv  26:                          general.file_type u32              = 2
llama_model_loader: - type  f32:  181 tensors
llama_model_loader: - type q4_0:  253 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_0
print_info: file size   = 1.62 GiB (4.50 BPW)
load: printing all EOG tokens:
load:   - 151643 ('<|endoftext|>')
load:   - 151645 ('<|im_end|>')
load:   - 151662 ('<|fim_pad|>')
load:   - 151663 ('<|repo_name|>')
load:   - 151664 ('<|file_sep|>')
load: special tokens cache size = 22
load: token to piece cache size = 0.9310 MB
print_info: arch             = qwen2vl
print_info: vocab_only       = 0
print_info: n_ctx_train      = 128000
print_info: n_embd           = 2048
print_info: n_embd_inp       = 2048
print_info: n_layer          = 36
print_info: n_head           = 16
print_info: n_head_kv        = 2
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 256
print_info: n_embd_v_gqa     = 256
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 11008
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = -1
print_info: rope type        = 8
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 128000
print_info: rope_finetuned   = unknown
print_info: mrope sections   = [16, 24, 24, 0]
print_info: model type       = 3B
print_info: model params     = 3.09 B
print_info: general.name     = Qwen2.5 VL 3B Instruct
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 151643 '<|endoftext|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 36 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 37/37 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   166.92 MiB
load_tensors:       OpenCL model buffer size =  1656.22 MiB
....................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 8192
llama_context: n_ctx_seq     = 8192
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = disabled
llama_context: kv_unified    = false
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (8192) < n_ctx_train (128000) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.58 MiB
llama_kv_cache:     OpenCL KV buffer size =   288.00 MiB
llama_kv_cache: size =  288.00 MiB (  8192 cells,  36 layers,  1/1 seqs), K (f16):  144.00 MiB, V (f16):  144.00 MiB
llama_context:     OpenCL compute buffer size =   300.75 MiB
llama_context:        CPU compute buffer size =    21.02 MiB
llama_context: graph nodes  = 1410
llama_context: graph splits = 2
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|im_end|> logit bias = -inf
common_init_from_params: added <|fim_pad|> logit bias = -inf
common_init_from_params: added <|repo_name|> logit bias = -inf
common_init_from_params: added <|file_sep|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 8192
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
mtmd_cli_context: chat template example:
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

clip_model_loader: model name:   Qwen2.5-VL-3B-Instruct
clip_model_loader: description:  image encoder for Qwen2VL
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    520
clip_model_loader: n_kv:         21

clip_model_loader: has vision encoder
clip_ctx: CLIP using OpenCL backend
load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

load_hparams: projector:          qwen2.5vl_merger
load_hparams: n_embd:             1280
load_hparams: n_head:             16
load_hparams: n_ff:               0
load_hparams: n_layer:            32
load_hparams: ffn_op:             silu
load_hparams: projection_dim:     2048

--- vision hparams ---
load_hparams: image_size:         560
load_hparams: patch_size:         14
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   0
load_hparams: n_merge:            2
load_hparams: n_wa_pattern:       8
load_hparams: image_min_pixels:   6272
load_hparams: image_max_pixels:   3211264

load_hparams: model size:         2550.83 MiB
load_hparams: metadata size:      0.18 MiB
alloc_compute_meta: warmup with image size = 1288 x 1288
alloc_compute_meta:     OpenCL compute buffer size =   732.56 MiB
alloc_compute_meta:        CPU compute buffer size =   292.41 MiB
alloc_compute_meta: graph splits = 1, nodes = 1092
warmup: flash attention is enabled
main: loading model: ../llama.cpp-models/qwen2.5-vl-3b-instruct-Q4_0_pure.gguf
encoding image slice...
image slice encoded in 98045 ms
decoding image batch 1/1, n_tokens_batch = 1457
image decoded (batch 1/1) in 5659 ms

The image depicts a dense cluster of tree leaves, with the majority of the frame filled by green foliage. The leaves are small and appear to be part of a deciduous tree, as indicated by their size and shape. The leaves are tightly packed together, creating a lush canopy that obscures most of the sky behind them

llama_perf_context_print:        load time =    8815.89 ms
llama_perf_context_print: prompt eval time =  106797.44 ms /  1712 tokens (   62.38 ms per token,    16.03 tokens per second)
llama_perf_context_print:        eval time =    4065.12 ms /    63 runs   (   64.53 ms per token,    15.50 tokens per second)
llama_perf_context_print:       total time =  121117.37 ms /  1775 tokens
llama_perf_context_print:    graphs reused =          0

[forforever73]

@lhez Thanks for the follow-up. I'm using an FP16 mmproj. The resolution of my tree.jpg image is 448 × 448, and I’ve attached the image here.

[lhez]

@forforever73 Thank you, I was able to reproduce it. It happens with f16 mmproj. The new kernel gets used for ordinary f16f32 matmat multiply, which it cannot handle. Could you give #17392 a try?