Eval bug: Gemma-4-26B-A4B cannot process images

[ZUIcat]

Name and Version

build_info: b8672-25eec6f32

Operating systems

Windows

GGML backends

Vulkan

Hardware

AMD 8840U (780M)

Models

Gemma-4-26B-A4B-it-UD-Q4_K_XL__unsloth_cb8ed2.gguf
Gemma-4-26B-A4B-it-mmproj-F16__unsloth_90a77a.gguf

Problem description & steps to reproduce

I tried using Gemma-4-26B-A4B for OCR, but it never worked—it would freeze and then crash. My machine is an AMD 8840U (780M). I used the Vulkan backend. Later, I tried the HIP backend (self-compiled using TheRock project). The first time, it output repetitive garbage; the second time, it worked as expected.

.\llama.cpp_vulkan\llama-server.exe --webui-config-file config_webui.json --props --slots --metrics --host 0.0.0.0 --port 8090 --gpu-layers all --cache-type-k q8_0 --cache-type-v q8_0 --cache-reuse 256 --model .\_model\Gemma-4-26B-A4B-it\Gemma-4-26B-A4B-it-UD-Q4_K_XL__unsloth_cb8ed2.gguf --mmproj .\_model\Gemma-4-26B-A4B-it\Gemma-4-26B-A4B-it-mmproj-F16__unsloth_90a77a.gguf --ctx-size 100000 --temp 1.0 --top-k 64 --top-p 0.95 --reasoning off --alias Gemma-4-26B-A4B-it-Instruct --parallel -1 --no-mmap

First Bad Commit

No response

Relevant log output

Logs

.\llama.cpp_vulkan\llama-server.exe --webui-config-file config_webui.json --props --slots --metrics --host 0.0.0.0 --port 8090 --gpu-layers all --cache-type-k q8_0 --cache-type-v q8_0 --cache-reuse 256 --model .\_model\Gemma-4-26B-A4B-it\Gemma-4-26B-A4B-it-UD-Q4_K_XL__unsloth_cb8ed2.gguf --mmproj .\_model\Gemma-4-26B-A4B-it\Gemma-4-26B-A4B-it-mmproj-F16__unsloth_90a77a.gguf --ctx-size 100000 --temp 1.0 --top-k 64 --top-p 0.95 --reasoning off --alias Gemma-4-26B-A4B-it-Instruct --parallel -1 --no-mmap

load_backend: loaded RPC backend from T:\_AI\Llama\llama.cpp_vulkan\ggml-rpc.dll
load_backend: loaded Vulkan backend from T:\_AI\Llama\llama.cpp_vulkan\ggml-vulkan.dll
load_backend: loaded CPU backend from T:\_AI\Llama\llama.cpp_vulkan\ggml-cpu-zen4.dll
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build_info: b8672-25eec6f32
system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
Running without SSL
init: using 15 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model '.\_model\Gemma-4-26B-A4B-it\Gemma-4-26B-A4B-it-UD-Q4_K_XL__unsloth_cb8ed2.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected to use 18374 MiB of device memory vs. 23242 MiB of free device memory
llama_params_fit_impl: will leave 4868 >= 1024 MiB of free device memory, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.75 seconds
llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon 780M Graphics ) (unknown id) - 23246 MiB free
llama_model_loader: loaded meta data with 60 key-value pairs and 658 tensors from .\_model\Gemma-4-26B-A4B-it\Gemma-4-26B-A4B-it-UD-Q4_K_XL__unsloth_cb8ed2.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma4
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                     general.sampling.top_k i32              = 64
llama_model_loader: - kv   3:                     general.sampling.top_p f32              = 0.950000
llama_model_loader: - kv   4:                      general.sampling.temp f32              = 1.000000
llama_model_loader: - kv   5:                               general.name str              = Gemma-4-26B-A4B-It
llama_model_loader: - kv   6:                           general.finetune str              = it
llama_model_loader: - kv   7:                           general.basename str              = Gemma-4-26B-A4B-It
llama_model_loader: - kv   8:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   9:                         general.size_label str              = 26B-A4B
llama_model_loader: - kv  10:                            general.license str              = apache-2.0
llama_model_loader: - kv  11:                       general.license.link str              = https://ai.google.dev/gemma/docs/gemm...
llama_model_loader: - kv  12:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv  13:                   general.base_model.count u32              = 1
llama_model_loader: - kv  14:                  general.base_model.0.name str              = Gemma 4 26B A4B It
llama_model_loader: - kv  15:          general.base_model.0.organization str              = Google
llama_model_loader: - kv  16:              general.base_model.0.repo_url str              = https://huggingface.co/google/gemma-4...
llama_model_loader: - kv  17:                               general.tags arr[str,2]       = ["unsloth", "image-text-to-text"]
llama_model_loader: - kv  18:                         gemma4.block_count u32              = 30
llama_model_loader: - kv  19:                      gemma4.context_length u32              = 262144
llama_model_loader: - kv  20:                    gemma4.embedding_length u32              = 2816
llama_model_loader: - kv  21:                 gemma4.feed_forward_length u32              = 2112
llama_model_loader: - kv  22:                gemma4.attention.head_count u32              = 16
llama_model_loader: - kv  23:             gemma4.attention.head_count_kv arr[i32,30]      = [8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, ...
llama_model_loader: - kv  24:                      gemma4.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  25:                  gemma4.rope.freq_base_swa f32              = 10000.000000
llama_model_loader: - kv  26:    gemma4.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  27:                        gemma4.expert_count u32              = 128
llama_model_loader: - kv  28:                   gemma4.expert_used_count u32              = 8
llama_model_loader: - kv  29:                gemma4.attention.key_length u32              = 512
llama_model_loader: - kv  30:              gemma4.attention.value_length u32              = 512
llama_model_loader: - kv  31:             gemma4.final_logit_softcapping f32              = 30.000000
llama_model_loader: - kv  32:            gemma4.attention.sliding_window u32              = 1024
llama_model_loader: - kv  33:          gemma4.attention.shared_kv_layers u32              = 0
llama_model_loader: - kv  34:    gemma4.embedding_length_per_layer_input u32              = 0
llama_model_loader: - kv  35:    gemma4.attention.sliding_window_pattern arr[bool,30]     = [true, true, true, true, true, false,...
llama_model_loader: - kv  36:            gemma4.attention.key_length_swa u32              = 256
llama_model_loader: - kv  37:          gemma4.attention.value_length_swa u32              = 256
llama_model_loader: - kv  38:          gemma4.expert_feed_forward_length u32              = 704
llama_model_loader: - kv  39:                gemma4.rope.dimension_count u32              = 512
llama_model_loader: - kv  40:            gemma4.rope.dimension_count_swa u32              = 256
llama_model_loader: - kv  41:                       tokenizer.ggml.model str              = gemma4
llama_model_loader: - kv  42:                      tokenizer.ggml.tokens arr[str,262144]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  43:                      tokenizer.ggml.scores arr[f32,262144]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  44:                  tokenizer.ggml.token_type arr[i32,262144]  = [3, 1, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  45:                      tokenizer.ggml.merges arr[str,514906]  = ["\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n", ...
llama_model_loader: - kv  46:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  47:                tokenizer.ggml.eos_token_id u32              = 106
llama_model_loader: - kv  48:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  49:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  50:               tokenizer.ggml.mask_token_id u32              = 4
llama_model_loader: - kv  51:                    tokenizer.chat_template str              = {%- macro format_parameters(propertie...
llama_model_loader: - kv  52:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  53:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  54:               general.quantization_version u32              = 2
llama_model_loader: - kv  55:                          general.file_type u32              = 15
llama_model_loader: - kv  56:                      quantize.imatrix.file str              = gemma-4-26B-A4B-it-GGUF/imatrix_unslo...
llama_model_loader: - kv  57:                   quantize.imatrix.dataset str              = unsloth_calibration_gemma-4-26B-A4B-i...
llama_model_loader: - kv  58:             quantize.imatrix.entries_count u32              = 295
llama_model_loader: - kv  59:              quantize.imatrix.chunks_count u32              = 141
llama_model_loader: - type  f32:  392 tensors
llama_model_loader: - type q5_1:   28 tensors
llama_model_loader: - type q8_0:  200 tensors
llama_model_loader: - type q4_K:   29 tensors
llama_model_loader: - type q5_K:    1 tensors
llama_model_loader: - type bf16:    8 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 15.95 GiB (5.43 BPW)
load: 0 unused tokens
load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
load: printing all EOG tokens:
load:   - 50 ('<|tool_response>')
load:   - 106 ('<turn|>')
load:   - 212 ('</s>')
load: special tokens cache size = 24
load: token to piece cache size = 1.9443 MB
print_info: arch                  = gemma4
print_info: vocab_only            = 0
print_info: no_alloc              = 0
print_info: n_ctx_train           = 262144
print_info: n_embd                = 2816
print_info: n_embd_inp            = 2816
print_info: n_layer               = 30
print_info: n_head                = 16
print_info: n_head_kv             = [8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2]
print_info: n_rot                 = 512
print_info: n_swa                 = 1024
print_info: is_swa_any            = 1
print_info: n_embd_head_k         = 512
print_info: n_embd_head_v         = 512
print_info: n_gqa                 = [2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8]
print_info: n_embd_k_gqa          = [2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024]
print_info: n_embd_v_gqa          = [2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024]
print_info: f_norm_eps            = 0.0e+00
print_info: f_norm_rms_eps        = 1.0e-06
print_info: f_clamp_kqv           = 0.0e+00
print_info: f_max_alibi_bias      = 0.0e+00
print_info: f_logit_scale         = 0.0e+00
print_info: f_attn_scale          = 1.0e+00
print_info: n_ff                  = 2112
print_info: n_expert              = 128
print_info: n_expert_used         = 8
print_info: n_expert_groups       = 0
print_info: n_group_used          = 0
print_info: causal attn           = 1
print_info: pooling type          = -1
print_info: rope type             = 2
print_info: rope scaling          = linear
print_info: freq_base_train       = 1000000.0
print_info: freq_scale_train      = 1
print_info: freq_base_swa         = 10000.0
print_info: freq_scale_swa        = 1
print_info: n_embd_head_k_swa     = 256
print_info: n_embd_head_v_swa     = 256
print_info: n_rot_swa             = 256
print_info: n_ctx_orig_yarn       = 262144
print_info: rope_yarn_log_mul     = 0.0000
print_info: rope_finetuned        = unknown
print_info: model type            = ?B
print_info: model params          = 25.23 B
print_info: general.name          = Gemma-4-26B-A4B-It
print_info: vocab type            = BPE
print_info: n_vocab               = 262144
print_info: n_merges              = 514906
print_info: BOS token             = 2 '<bos>'
print_info: EOS token             = 106 '<turn|>'
print_info: UNK token             = 3 '<unk>'
print_info: PAD token             = 0 '<pad>'
print_info: MASK token            = 4 '<mask>'
print_info: LF token              = 107 '
'
print_info: EOG token             = 50 '<|tool_response>'
print_info: EOG token             = 106 '<turn|>'
print_info: EOG token             = 212 '</s>'
print_info: max token length      = 93
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
str: cannot properly format tensor name output with suffix=weight bid=-1 xid=-1
load_tensors: offloading output layer to GPU
load_tensors: offloading 29 repeating layers to GPU
load_tensors: offloaded 31/31 layers to GPU
load_tensors:      Vulkan0 model buffer size = 16335.67 MiB
load_tensors:  Vulkan_Host model buffer size =   748.00 MiB
..............................................................
common_init_result: added <|tool_response> logit bias = -inf
common_init_result: added <turn|> logit bias = -inf
common_init_result: added </s> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max     = 4
llama_context: n_ctx         = 100096
llama_context: n_ctx_seq     = 100096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (100096) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host  output buffer size =     4.00 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 100096 cells
llama_kv_cache:    Vulkan0 KV buffer size =  1038.59 MiB
llama_kv_cache: size = 1038.59 MiB (100096 cells,   5 layers,  4/1 seqs), K (q8_0):  519.30 MiB, V (q8_0):  519.30 MiB
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0
llama_kv_cache_iswa: creating     SWA KV cache, size = 4608 cells
llama_kv_cache:    Vulkan0 KV buffer size =   478.12 MiB
llama_kv_cache: size =  478.12 MiB (  4608 cells,  25 layers,  4/1 seqs), K (q8_0):  239.06 MiB, V (q8_0):  239.06 MiB
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0
sched_reserve: reserving ...
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve: resolving fused Gated Delta Net support:
sched_reserve: fused Gated Delta Net (autoregressive) enabled
sched_reserve: fused Gated Delta Net (chunked) enabled
sched_reserve:    Vulkan0 compute buffer size =   521.91 MiB
sched_reserve: Vulkan_Host compute buffer size =   215.52 MiB
sched_reserve: graph nodes  = 2647
sched_reserve: graph splits = 2
sched_reserve: reserve took 363.58 ms, sched copies = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
clip_model_loader: model name:   Gemma-4-26B-A4B-It
clip_model_loader: description:
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    356
clip_model_loader: n_kv:         32

clip_model_loader: has vision encoder
clip_ctx: CLIP using Vulkan0 backend
load_hparams: projector:          gemma4v
load_hparams: n_embd:             1152
load_hparams: n_head:             16
load_hparams: n_ff:               4304
load_hparams: n_layer:            27
load_hparams: ffn_op:             gelu_quick
load_hparams: projection_dim:     2816

--- vision hparams ---
load_hparams: image_size:         224
load_hparams: patch_size:         16
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   0
load_hparams: n_merge:            3
load_hparams: n_wa_pattern: 0
load_hparams: image_min_pixels:   580608
load_hparams: image_max_pixels:   645120

load_hparams: model size:         1137.77 MiB
load_hparams: metadata size:      0.12 MiB
warmup: warmup with image size = 768 x 768
alloc_compute_meta:    Vulkan0 compute buffer size =   150.63 MiB
alloc_compute_meta:        CPU compute buffer size =     6.77 MiB
alloc_compute_meta: graph splits = 1, nodes = 1569
warmup: flash attention is enabled
srv    load_model: loaded multimodal model, '.\_model\Gemma-4-26B-A4B-it\Gemma-4-26B-A4B-it-mmproj-F16__unsloth_90a77a.gguf'
srv    load_model: cache_reuse is not supported by multimodal, it will be disabled
srv    load_model: initializing slots, n_slots = 4
no implementations specified for speculative decoding
slot   load_model: id  0 | task -1 | speculative decoding context not initialized
slot   load_model: id  0 | task -1 | new slot, n_ctx = 100096
no implementations specified for speculative decoding
slot   load_model: id  1 | task -1 | speculative decoding context not initialized
slot   load_model: id  1 | task -1 | new slot, n_ctx = 100096
no implementations specified for speculative decoding
slot   load_model: id  2 | task -1 | speculative decoding context not initialized
slot   load_model: id  2 | task -1 | new slot, n_ctx = 100096
no implementations specified for speculative decoding
slot   load_model: id  3 | task -1 | speculative decoding context not initialized
slot   load_model: id  3 | task -1 | new slot, n_ctx = 100096
srv    load_model: prompt cache is enabled, size limit: 8192 MiB
srv    load_model: use `--cache-ram 0` to disable the prompt cache
srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
srv          init: init: idle slots will be saved to prompt cache and cleared upon starting a new task
init: chat template, example_format: '<bos><|turn>system
You are a helpful assistant<turn|>
<|turn>user
Hello<turn|>
<|turn>model
Hi there<turn|>
<|turn>user
How are you?<turn|>
<|turn>model
<|channel>thought
<channel|>'
srv          init: init: chat template, thinking = 0
main: model loaded
main: server is listening on http://0.0.0.0:8090
main: starting the main loop...
srv  update_slots: all slots are idle
srv  log_server_r: done request: GET / 192.168.0.108 200
srv  log_server_r: done request: GET /bundle.css 192.168.0.108 200
srv  log_server_r: done request: GET /bundle.js 192.168.0.108 200
srv  log_server_r: done request: HEAD /cors-proxy 192.168.0.108 404
srv  params_from_: Chat format: peg-gemma4
slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
srv  get_availabl: updating prompt cache
srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 100096 tokens, 8589934592 est)
srv  get_availabl: prompt cache update took 0.02 ms
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> ?temp-ext -> dist
slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 100096, n_keep = 0, task.n_tokens = 312
slot update_slots: id  3 | task 0 | cache reuse is not supported - ignoring n_cache_reuse = 256
slot update_slots: id  3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 22, batch.n_tokens = 22, progress = 0.070513
srv  log_server_r: done request: POST /v1/chat/completions 192.168.0.108 200
slot update_slots: id  3 | task 0 | n_tokens = 22, memory_seq_rm [22, end)
srv  process_chun: processing image...
encoding image slice...
image slice encoded in 3403 ms
decoding image batch 1/1, n_tokens_batch = 280
sched_reserve: reserving ...
sched_reserve:    Vulkan0 compute buffer size =   521.91 MiB
sched_reserve: Vulkan_Host compute buffer size =   215.52 MiB
sched_reserve: graph nodes  = 2647
sched_reserve: graph splits = 2
sched_reserve: reserve took 152.71 ms, sched copies = 1
image decoded (batch 1/1) in 255 ms
srv  process_chun: image processed in 3658 ms
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 308, batch.n_tokens = 6, progress = 0.987179
sched_reserve: reserving ...
sched_reserve:    Vulkan0 compute buffer size =   521.91 MiB
sched_reserve: Vulkan_Host compute buffer size =   215.52 MiB
sched_reserve: graph nodes  = 2647
sched_reserve: graph splits = 2
sched_reserve: reserve took 139.77 ms, sched copies = 1

[olkovi]

Can confirm. Getting the same with AI Max 395+, also with Vulkan, Windows 11. Text processing seems to work fine, but using a single image causes a freeze and crash from the next first generated token.

[chejh-amd]

Yeah this is a known pain point, text inference on AMD Vulkan works fine but the vision encoder path crashes or freezes on images.
Looking at your logs, the vision encoder itself completes ("image decoded in 255 ms") but the server dies right after, so the crash likely happens when the main model tries to consume the image embeddings. @olkovi hitting the same on AI Max 395+ rules out it being 780M-specific.
A few things worth trying:

1. --flash-attn off to see if FA interacts with the vision embedding path
2. Smaller --ctx-size (e.g. 8192) to change the buffer allocation pattern

Fwiw HIP via TheRock seems more reliable for multimodal Gemma4 on AMD right now.

[CatKasha]

works fine for me with rx6600 on vulkan backend, w10, adrenaline 26.3.1
b8830, gemma-4-26B-A4B-it-UD-Q4_K_M.gguf and mmproj-F16.gguf

[ZUIcat]

As of build_info: b8857-a6cc43c28, the issue persists. Is this problem not common? It is consistently reproducible on my end.

[chejh-amd]

Hi @ZUIcat did you get a chance to try the two things from my earlier comment — --flash-attn off and a smaller --ctx-size (like 8192)?

[ZUIcat]

@chejh-amd Sorry for forgetting about this. I just tried it with build_info: b8913-e5f070a1d. The experience was a bit odd — first, turning off flash-attn required me to disable k-v quantization and drastically reduce the context length (200000 → 50000). After making these changes, it did run, but the trade-off felt a bit too high...

Logs

>.\llama.cpp_vulkan\llama-server.exe --webui-config-file config_webui.json --props --slots --metrics --host 0.0.0.0 --port 8090 --gpu-layers all --cache-type-k f16 --cache-type-v f16 --cache-reuse 256 --model .\_model\Gemma-4-26B-A4B-it\Gemma-4-26B-A4B-it-UD-Q4_K_XL__unsloth_871214.gguf --mmproj .\_model\Gemma-4-26B-A4B-it\Gemma-4-26B-A4B-it-mmproj-BF16__unsloth_fc2ebf.gguf --ctx-size 50000 --temp 1.0 --top-k 64 --top-p 0.95 --reasoning off --flash-attn off --alias Gemma-4-26B-A4B-it-Instruct --parallel -1 --no-mmap

load_backend: loaded RPC backend from T:\_AI\Llama\llama.cpp_vulkan\ggml-rpc.dll
load_backend: loaded Vulkan backend from T:\_AI\Llama\llama.cpp_vulkan\ggml-vulkan.dll
load_backend: loaded CPU backend from T:\_AI\Llama\llama.cpp_vulkan\ggml-cpu-zen4.dll
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build_info: b8913-e5f070a1d
system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
init: using 15 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model '.\_model\Gemma-4-26B-A4B-it\Gemma-4-26B-A4B-it-UD-Q4_K_XL__unsloth_871214.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
common_params_fit_impl: getting device memory data for initial parameters:
common_memory_breakdown_print: | memory breakdown [MiB]       | total    free     self   model   context   compute       unaccounted |
common_memory_breakdown_print: |   - Vulkan0 (780M Graphics ) | 24469 = 23242 + (20523 = 16283 +    2370 +    1870) + 17592186025120 |
common_memory_breakdown_print: |   - Host                     |                    882 =   748 +       0 +     134                   |
common_params_fit_impl: projected to use 20523 MiB of device memory vs. 23242 MiB of free device memory
common_params_fit_impl: will leave 2718 >= 1024 MiB of free device memory, no changes needed
common_fit_params: successfully fit params to free device memory
common_fit_params: fitting params to free memory took 0.87 seconds
llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon 780M Graphics ) (unknown id) - 23246 MiB free
llama_model_loader: loaded meta data with 60 key-value pairs and 658 tensors from .\_model\Gemma-4-26B-A4B-it\Gemma-4-26B-A4B-it-UD-Q4_K_XL__unsloth_871214.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma4
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                     general.sampling.top_k i32              = 64
llama_model_loader: - kv   3:                     general.sampling.top_p f32              = 0.950000
llama_model_loader: - kv   4:                      general.sampling.temp f32              = 1.000000
llama_model_loader: - kv   5:                               general.name str              = Gemma-4-26B-A4B-It
llama_model_loader: - kv   6:                           general.finetune str              = it
llama_model_loader: - kv   7:                           general.basename str              = Gemma-4-26B-A4B-It
llama_model_loader: - kv   8:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   9:                         general.size_label str              = 26B-A4B
llama_model_loader: - kv  10:                            general.license str              = apache-2.0
llama_model_loader: - kv  11:                       general.license.link str              = https://ai.google.dev/gemma/docs/gemm...
llama_model_loader: - kv  12:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv  13:                   general.base_model.count u32              = 1
llama_model_loader: - kv  14:                  general.base_model.0.name str              = Gemma 4 26B A4B It
llama_model_loader: - kv  15:          general.base_model.0.organization str              = Google
llama_model_loader: - kv  16:              general.base_model.0.repo_url str              = https://huggingface.co/google/gemma-4...
llama_model_loader: - kv  17:                               general.tags arr[str,2]       = ["unsloth", "image-text-to-text"]
llama_model_loader: - kv  18:                         gemma4.block_count u32              = 30
llama_model_loader: - kv  19:                      gemma4.context_length u32              = 262144
llama_model_loader: - kv  20:                    gemma4.embedding_length u32              = 2816
llama_model_loader: - kv  21:                 gemma4.feed_forward_length u32              = 2112
llama_model_loader: - kv  22:                gemma4.attention.head_count u32              = 16
llama_model_loader: - kv  23:             gemma4.attention.head_count_kv arr[i32,30]      = [8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, ...
llama_model_loader: - kv  24:                      gemma4.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  25:                  gemma4.rope.freq_base_swa f32              = 10000.000000
llama_model_loader: - kv  26:    gemma4.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  27:                        gemma4.expert_count u32              = 128
llama_model_loader: - kv  28:                   gemma4.expert_used_count u32              = 8
llama_model_loader: - kv  29:                gemma4.attention.key_length u32              = 512
llama_model_loader: - kv  30:              gemma4.attention.value_length u32              = 512
llama_model_loader: - kv  31:             gemma4.final_logit_softcapping f32              = 30.000000
llama_model_loader: - kv  32:            gemma4.attention.sliding_window u32              = 1024
llama_model_loader: - kv  33:          gemma4.attention.shared_kv_layers u32              = 0
llama_model_loader: - kv  34:    gemma4.embedding_length_per_layer_input u32              = 0
llama_model_loader: - kv  35:    gemma4.attention.sliding_window_pattern arr[bool,30]     = [true, true, true, true, true, false,...
llama_model_loader: - kv  36:            gemma4.attention.key_length_swa u32              = 256
llama_model_loader: - kv  37:          gemma4.attention.value_length_swa u32              = 256
llama_model_loader: - kv  38:          gemma4.expert_feed_forward_length u32              = 704
llama_model_loader: - kv  39:                gemma4.rope.dimension_count u32              = 512
llama_model_loader: - kv  40:            gemma4.rope.dimension_count_swa u32              = 256
llama_model_loader: - kv  41:                       tokenizer.ggml.model str              = gemma4
llama_model_loader: - kv  42:                      tokenizer.ggml.tokens arr[str,262144]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  43:                      tokenizer.ggml.scores arr[f32,262144]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  44:                  tokenizer.ggml.token_type arr[i32,262144]  = [3, 1, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  45:                      tokenizer.ggml.merges arr[str,514906]  = ["\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n", ...
llama_model_loader: - kv  46:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  47:                tokenizer.ggml.eos_token_id u32              = 106
llama_model_loader: - kv  48:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  49:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  50:               tokenizer.ggml.mask_token_id u32              = 4
llama_model_loader: - kv  51:                    tokenizer.chat_template str              = {%- macro format_parameters(propertie...
llama_model_loader: - kv  52:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  53:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  54:               general.quantization_version u32              = 2
llama_model_loader: - kv  55:                          general.file_type u32              = 15
llama_model_loader: - kv  56:                      quantize.imatrix.file str              = gemma-4-26B-A4B-it-GGUF/imatrix_unslo...
llama_model_loader: - kv  57:                   quantize.imatrix.dataset str              = unsloth_calibration_gemma-4-26B-A4B-i...
llama_model_loader: - kv  58:             quantize.imatrix.entries_count u32              = 295
llama_model_loader: - kv  59:              quantize.imatrix.chunks_count u32              = 141
llama_model_loader: - type  f32:  392 tensors
llama_model_loader: - type q5_1:   28 tensors
llama_model_loader: - type q8_0:  208 tensors
llama_model_loader: - type q4_K:   29 tensors
llama_model_loader: - type q5_K:    1 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 15.90 GiB (5.41 BPW)
load: 0 unused tokens
load: control-looking token:      1 '<eos>' was not control-type; this is probably a bug in the model. its type will be overridden
load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
load: printing all EOG tokens:
load:   - 1 ('<eos>')
load:   - 50 ('<|tool_response>')
load:   - 106 ('<turn|>')
load:   - 212 ('</s>')
load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
load: special tokens cache size = 24
load: token to piece cache size = 1.9445 MB
print_info: arch                  = gemma4
print_info: vocab_only            = 0
print_info: no_alloc              = 0
print_info: n_ctx_train           = 262144
print_info: n_embd                = 2816
print_info: n_embd_inp            = 2816
print_info: n_layer               = 30
print_info: n_head                = 16
print_info: n_head_kv             = [8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2]
print_info: n_rot                 = 512
print_info: n_swa                 = 1024
print_info: is_swa_any            = 1
print_info: n_embd_head_k         = 512
print_info: n_embd_head_v         = 512
print_info: n_gqa                 = [2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8, 2, 2, 2, 2, 2, 8]
print_info: n_embd_k_gqa          = [2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024]
print_info: n_embd_v_gqa          = [2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024, 2048, 2048, 2048, 2048, 2048, 1024]
print_info: f_norm_eps            = 0.0e+00
print_info: f_norm_rms_eps        = 1.0e-06
print_info: f_clamp_kqv           = 0.0e+00
print_info: f_max_alibi_bias      = 0.0e+00
print_info: f_logit_scale         = 0.0e+00
print_info: f_attn_scale          = 1.0e+00
print_info: n_ff                  = 2112
print_info: n_expert              = 128
print_info: n_expert_used         = 8
print_info: n_expert_groups       = 0
print_info: n_group_used          = 0
print_info: causal attn           = 1
print_info: pooling type          = -1
print_info: rope type             = 2
print_info: rope scaling          = linear
print_info: freq_base_train       = 1000000.0
print_info: freq_scale_train      = 1
print_info: freq_base_swa         = 10000.0
print_info: freq_scale_swa        = 1
print_info: n_embd_head_k_swa     = 256
print_info: n_embd_head_v_swa     = 256
print_info: n_rot_swa             = 256
print_info: n_ctx_orig_yarn       = 262144
print_info: rope_yarn_log_mul     = 0.0000
print_info: rope_finetuned        = unknown
print_info: model type            = 26B.A4B
print_info: model params          = 25.23 B
print_info: general.name          = Gemma-4-26B-A4B-It
print_info: vocab type            = BPE
print_info: n_vocab               = 262144
print_info: n_merges              = 514906
print_info: BOS token             = 2 '<bos>'
print_info: EOS token             = 106 '<turn|>'
print_info: UNK token             = 3 '<unk>'
print_info: PAD token             = 0 '<pad>'
print_info: MASK token            = 4 '<mask>'
print_info: LF token              = 107 '
'
print_info: EOG token             = 1 '<eos>'
print_info: EOG token             = 50 '<|tool_response>'
print_info: EOG token             = 106 '<turn|>'
print_info: max token length      = 93
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
load_tensors: offloading output layer to GPU
load_tensors: offloading 29 repeating layers to GPU
load_tensors: offloaded 31/31 layers to GPU
load_tensors:      Vulkan0 model buffer size = 16283.47 MiB
load_tensors:  Vulkan_Host model buffer size =   748.00 MiB
..............................................................
common_init_result: added <eos> logit bias = -inf
common_init_result: added <|tool_response> logit bias = -inf
common_init_result: added <turn|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max     = 4
llama_context: n_ctx         = 50176
llama_context: n_ctx_seq     = 50176
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = disabled
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (50176) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host  output buffer size =     4.00 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 50176 cells
llama_kv_cache: the V embeddings have different sizes across layers and FA is not enabled - padding V cache to 2048
llama_kv_cache:    Vulkan0 KV buffer size =  1470.00 MiB
llama_kv_cache: size = 1470.00 MiB ( 50176 cells,   5 layers,  4/1 seqs), K (f16):  490.00 MiB, V (f16):  980.00 MiB
llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 512
llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 512
llama_kv_cache_iswa: creating     SWA KV cache, size = 4608 cells
llama_kv_cache: the V embeddings have different sizes across layers and FA is not enabled - padding V cache to 2048
llama_kv_cache:    Vulkan0 KV buffer size =   900.00 MiB
llama_kv_cache: size =  900.00 MiB (  4608 cells,  25 layers,  4/1 seqs), K (f16):  450.00 MiB, V (f16):  450.00 MiB
llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 256
llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 256
sched_reserve: reserving ...
sched_reserve: resolving fused Gated Delta Net support:
sched_reserve: fused Gated Delta Net (autoregressive) enabled
sched_reserve: fused Gated Delta Net (chunked) enabled
sched_reserve:    Vulkan0 compute buffer size =  1870.01 MiB
sched_reserve: Vulkan_Host compute buffer size =   134.01 MiB
sched_reserve: graph nodes  = 2800
sched_reserve: graph splits = 2
sched_reserve: reserve took 1153.78 ms, sched copies = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
clip_model_loader: model name:   Gemma-4-26B-A4B-It
clip_model_loader: description:
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    356
clip_model_loader: n_kv:         32

clip_model_loader: has vision encoder
clip_ctx: CLIP using Vulkan0 backend
load_hparams: projector:          gemma4v
load_hparams: n_embd:             1152
load_hparams: n_head:             16
load_hparams: n_ff:               4304
load_hparams: n_layer:            27
load_hparams: ffn_op:             gelu_quick
load_hparams: projection_dim:     2816

--- vision hparams ---
load_hparams: image_size:         224
load_hparams: patch_size:         16
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   0
load_hparams: n_merge:            3
load_hparams: n_wa_pattern: 0
load_hparams: image_min_pixels:   580608
load_hparams: image_max_pixels:   645120

load_hparams: model size:         1139.46 MiB
load_hparams: metadata size:      0.12 MiB
warmup: warmup with image size = 768 x 768
alloc_compute_meta:    Vulkan0 compute buffer size =   381.39 MiB
alloc_compute_meta:        CPU compute buffer size =     6.77 MiB
alloc_compute_meta: graph splits = 1, nodes = 1623
warmup: flash attention is disabled
srv    load_model: loaded multimodal model, '.\_model\Gemma-4-26B-A4B-it\Gemma-4-26B-A4B-it-mmproj-BF16__unsloth_fc2ebf.gguf'
srv    load_model: cache_reuse is not supported by multimodal, it will be disabled
srv    load_model: initializing slots, n_slots = 4
no implementations specified for speculative decoding
slot   load_model: id  0 | task -1 | new slot, n_ctx = 50176
no implementations specified for speculative decoding
slot   load_model: id  1 | task -1 | new slot, n_ctx = 50176
no implementations specified for speculative decoding
slot   load_model: id  2 | task -1 | new slot, n_ctx = 50176
no implementations specified for speculative decoding
slot   load_model: id  3 | task -1 | new slot, n_ctx = 50176
srv    load_model: prompt cache is enabled, size limit: 8192 MiB
srv    load_model: use `--cache-ram 0` to disable the prompt cache
srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
srv          init: init: idle slots will be saved to prompt cache and cleared upon starting a new task
init: chat template, example_format: '<|turn>system
You are a helpful assistant<turn|>
<|turn>user
Hello<turn|>
<|turn>model
Hi there<turn|>
<|turn>user
How are you?<turn|>
<|turn>model
<|channel>thought
<channel|>'
srv          init: init: chat template, thinking = 0
main: model loaded
main: server is listening on http://0.0.0.0:8090
main: starting the main loop...
srv  update_slots: all slots are idle
srv  log_server_r: done request: GET / 10.12.1.24 200
srv  log_server_r: done request: GET /bundle.css 10.12.1.24 200
srv  log_server_r: done request: GET /bundle.js 10.12.1.24 200
srv  log_server_r: done request: HEAD /cors-proxy 10.12.1.24 404
slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
srv  get_availabl: updating prompt cache
srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 50176 tokens, 8589934592 est)
srv  get_availabl: prompt cache update took 0.24 ms
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> ?temp-ext -> dist
slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 50176, n_keep = 0, task.n_tokens = 14
slot update_slots: id  3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 10, batch.n_tokens = 10, progress = 0.714286
srv  log_server_r: done request: POST /v1/chat/completions 10.12.1.24 200
slot update_slots: id  3 | task 0 | n_tokens = 10, memory_seq_rm [10, end)
slot init_sampler: id  3 | task 0 | init sampler, took 0.00 ms, tokens: text = 14, total = 14
slot update_slots: id  3 | task 0 | prompt processing done, n_tokens = 14, batch.n_tokens = 4
slot print_timing: id  3 | task 0 |
prompt eval time =     681.21 ms /    14 tokens (   48.66 ms per token,    20.55 tokens per second)
       eval time =     983.55 ms /    16 tokens (   61.47 ms per token,    16.27 tokens per second)
      total time =    1664.76 ms /    30 tokens
slot      release: id  3 | task 0 | stop processing: n_tokens = 29, truncated = 0
srv  update_slots: all slots are idle
slot get_availabl: id  2 | task -1 | selected slot by LRU, t_last = -1
srv  get_availabl: updating prompt cache
srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 50176 tokens, 8589934592 est)
srv  get_availabl: prompt cache update took 0.01 ms
slot launch_slot_: id  2 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> ?temp-ext -> dist
slot launch_slot_: id  2 | task 18 | processing task, is_child = 0
slot slot_save_an: id  3 | task -1 | saving idle slot to prompt cache
srv   prompt_save:  - saving prompt with length 29, total state size = 6.232 MiB
slot prompt_clear: id  3 | task -1 | clearing prompt with 29 tokens
srv        update:  - cache state: 1 prompts, 6.232 MiB (limits: 8192.000 MiB, 50176 tokens, 50176 est)
srv        update:    - prompt 000001BBA25C3810:      29 tokens, checkpoints:  0,     6.232 MiB
slot update_slots: id  2 | task 18 | new prompt, n_ctx_slot = 50176, n_keep = 0, task.n_tokens = 316
slot update_slots: id  2 | task 18 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  2 | task 18 | prompt processing progress, n_tokens = 33, batch.n_tokens = 33, progress = 0.104430
srv  log_server_r: done request: POST /v1/chat/completions 10.12.1.24 200
slot update_slots: id  2 | task 18 | n_tokens = 33, memory_seq_rm [33, end)
srv  process_chun: processing image...
encoding image slice...
image slice encoded in 5784 ms
decoding image batch 1/1, n_tokens_batch = 273
sched_reserve: reserving ...
sched_reserve:    Vulkan0 compute buffer size =  1870.01 MiB
sched_reserve: Vulkan_Host compute buffer size =   134.01 MiB
sched_reserve: graph nodes  = 2800
sched_reserve: graph splits = 2
sched_reserve: reserve took 381.57 ms, sched copies = 1
image decoded (batch 1/1) in 655 ms
srv  process_chun: image processed in 6440 ms
slot update_slots: id  2 | task 18 | prompt processing progress, n_tokens = 312, batch.n_tokens = 6, progress = 0.987342
sched_reserve: reserving ...
sched_reserve:    Vulkan0 compute buffer size =  1870.01 MiB
sched_reserve: Vulkan_Host compute buffer size =   134.01 MiB
sched_reserve: graph nodes  = 2800
sched_reserve: graph splits = 2
sched_reserve: reserve took 386.31 ms, sched copies = 1
slot update_slots: id  2 | task 18 | n_tokens = 312, memory_seq_rm [312, end)
slot init_sampler: id  2 | task 18 | init sampler, took 0.01 ms, tokens: text = 43, total = 316
slot update_slots: id  2 | task 18 | prompt processing done, n_tokens = 316, batch.n_tokens = 4
slot create_check: id  2 | task 18 | created context checkpoint 1 of 32 (pos_min = 0, pos_max = 311, n_tokens = 312, size = 60.942 MiB)
slot print_timing: id  2 | task 18 |
prompt eval time =   15568.98 ms /   316 tokens (   49.27 ms per token,    20.30 tokens per second)
       eval time =   17825.98 ms /   299 tokens (   59.62 ms per token,    16.77 tokens per second)
      total time =   33394.97 ms /   615 tokens
slot      release: id  2 | task 18 | stop processing: n_tokens = 614, truncated = 0
srv  update_slots: all slots are idle

[chejh-amd]

So FA off + f16 KV + halved context does get images working, but that's a pretty steep trade-off. The memory jump makes sense from the logs: gemma4 has different V embedding sizes across layers (512 for global, 256 for SWA), and without FA the KV cache pads everything to the max. With FA it doesn't need to, so FA is probably the expected path for this model's memory budget.

Worth noting CatKasha's RX 6600 works fine with Vulkan + FA, while 780M and AI Max 395+ both hit this. Only three data points so hard to say if it's architecture-related or something else, but the pattern plus HIP being stable for multimodal might help narrow it down for whoever looks at the Vulkan FA path.

[olkovi]

In my case using f16 KV with FA off at 8192 context did not help. Crash happens no matter what as long as an image is supplied to processing.

[ZUIcat]

I just tested with the latest build (b9093-1e5ad35d5) and the model (Gemma-4-26B-A4B-it-UD-Q4_K_XL__unsloth_453cf0). Not sure what changed in between, but it's working fine now and providing normal outputs.

[jdar-gems]

Same issue with 4-26B-A4B-QAT. Using CUDA 12. All updates, 95tps unless asked to upload a small image. SEBRT crash.