Eval bug: MTP models fail to load when running llama-server, works with llama-cli

[sheldonrobinson]

Name and Version

I have tested using model unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL

llama-cli --version
version: 9592 (ac4cddeb0)
built with Clang 20.1.8 for Windows x86_64

The command for llama-cli works

llama-cli.exe -hf unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL --temp 1.0 --top-p 0.95 --top-k 64 --spec-type draft-mtp --spec-draft-n-max 2

Console Output

0.00.082.282 I common_params_handle_remote_preset: looking for remote preset at https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/resolve/main/preset.ini
0.00.203.418 I common_download_file_single_online: HEAD failed, status: 404
0.00.205.427 I common_params_handle_remote_preset: no remote preset found, skipping

Loading model... |0.00.345.066 I srv    load_model: loading model 'C:\Users\sheld\.cache\huggingface\hub\models--unsloth--gemma-4-12b-it-GGUF\snapshots\3249fa54d5efa384afc552cc6700ad091efd5c39\gemma-4-12b-it-UD-Q4_K_XL.gguf'               \0.00.619.302 I srv    load_model: [mtmd] estimated worst-case memory usage of mmproj is 340.39 MiB                     -0.01.388.368 E llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this is normal during memory fitting)                                                                                    \0.01.464.391 W srv    load_model: [spec] failed to measure draft model memory: failed to create llama_context from model
0.01.464.423 I common_init_result: fitting params to device memory ...
0.01.464.424 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.01.464.426 I common_params_fit_impl: getting device memory data for initial parameters:                              -0.02.237.699 I common_memory_breakdown_print: | memory breakdown [MiB] | total   free    self   model   context   compute    unaccounted |
0.02.237.705 I common_memory_breakdown_print: |   - Host               |                 8172 =  3209 +    4576 +     387                |
0.02.237.706 I common_memory_breakdown_print: |   - CPU_REPACK         |                 4460 =  4460 +       0 +       0                |
0.02.320.185 I common_params_fit_impl: projected to use 8172 MiB of host memory vs. 32502 MiB of total host memory
0.02.320.193 I common_params_fit_impl: will leave 24329 >= 1364 MiB of system memory, no changes needed
0.02.320.243 I common_fit_params: successfully fit params to free device memory
0.02.320.248 I common_fit_params: fitting params to free memory took 0.01 seconds                                      \0.02.443.276 I llama_model_loader: loaded meta data with 58 key-value pairs and 667 tensors from C:\Users\sheld\.cache\huggingface\hub\models--unsloth--gemma-4-12b-it-GGUF\snapshots\3249fa54d5efa384afc552cc6700ad091efd5c39\gemma-4-12b-it-UD-Q4_K_XL.gguf (version GGUF V3 (latest))
0.02.443.303 I llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
0.02.443.305 I llama_model_loader: - kv   0:                       general.architecture str              = gemma4
0.02.443.306 I llama_model_loader: - kv   1:                               general.type str              = model
...
0.11.179.765 I llama_context: constructing llama_context
0.11.179.774 I llama_context: n_seq_max     = 1
0.11.179.774 I llama_context: n_ctx         = 262144
0.11.179.775 I llama_context: n_ctx_seq     = 262144
0.11.179.775 I llama_context: n_batch       = 2048
0.11.179.775 I llama_context: n_ubatch      = 512
0.11.179.776 I llama_context: causal_attn   = 1
0.11.179.776 I llama_context: flash_attn    = auto
0.11.179.776 I llama_context: kv_unified    = false
0.11.179.778 I llama_context: freq_base     = 1000000.0
0.11.179.780 I llama_context: freq_scale    = 1
0.11.179.780 I llama_context: n_rs_seq      = 0
0.11.179.780 I llama_context: n_outputs_max = 3
0.11.179.948 I llama_context:        CPU  output buffer size =     1.00 MiB
0.11.179.950 I llama_kv_cache_iswa: creating non-SWA KV cache, size = 262144 cells
0.11.181.395 W llama_kv_cache: layer   3: sharing with layer 47. k = 00000123CBAFA080, v = 00000123DBAFA080
0.11.181.400 I llama_kv_cache: size =  512.00 MiB (262144 cells,   1 layers,  1/1 seqs), K (f16):  256.00 MiB, V (f16):  256.00 MiB
0.11.181.400 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 512
0.11.181.401 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 512
0.11.181.408 I llama_kv_cache_iswa: creating     SWA KV cache, size = 1536 cells
0.11.181.437 W llama_kv_cache: layer   0: sharing with layer 46. k = 0000012408F01080, v = 0000012409501080
0.11.181.438 W llama_kv_cache: layer   1: sharing with layer 46. k = 0000012408F01080, v = 0000012409501080
0.11.181.440 W llama_kv_cache: layer   2: sharing with layer 46. k = 0000012408F01080, v = 0000012409501080
0.11.181.464 I llama_kv_cache: size =   36.00 MiB (  1536 cells,   3 layers,  1/1 seqs), K (f16):   18.00 MiB, V (f16):   18.00 MiB
0.11.181.465 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 256
0.11.181.466 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 256
0.11.181.476 I sched_reserve: reserving ...
0.11.181.934 I sched_reserve: Flash Attention was auto, set to enabled
0.11.181.935 I sched_reserve: resolving fused Gated Delta Net support:
0.11.182.168 I sched_reserve: fused Gated Delta Net (autoregressive) enabled
0.11.182.370 I sched_reserve: fused Gated Delta Net (chunked) enabled
0.11.183.006 I sched_reserve:        CPU compute buffer size =   313.00 MiB
0.11.183.007 I sched_reserve: graph nodes  = 126
0.11.183.008 I sched_reserve: graph splits = 1
0.11.183.009 I sched_reserve: reserve took 1.53 ms, sched copies = 1
0.11.183.771 I clip_model_loader: model name:   Gemma-4-12B-It
0.11.183.771 I clip_model_loader: description:
0.11.183.772 I clip_model_loader: GGUF version: 3
0.11.183.772 I clip_model_loader: alignment:    32
0.11.183.773 I clip_model_loader: n_tensors:    11
0.11.183.773 I clip_model_loader: n_kv:         41
0.11.183.773 I
0.11.183.775 I clip_model_loader: has vision encoder
0.11.183.775 I clip_model_loader: has audio encoder
0.11.183.809 I clip_ctx: CLIP using CPU backend
0.11.183.950 I load_hparams: projector:          gemma4uv
0.11.183.950 I load_hparams: n_embd:             3840
0.11.183.951 I load_hparams: n_head:             1
0.11.183.951 I load_hparams: n_ff:               0
0.11.183.951 I load_hparams: n_layer:            0
0.11.183.952 I load_hparams: ffn_op:             gelu_quick
0.11.183.952 I load_hparams: projection_dim:     3840
0.11.183.952 I
--- vision hparams ---
0.11.183.953 I load_hparams: image_size:         224
0.11.183.953 I load_hparams: patch_size:         48
0.11.183.953 I load_hparams: has_llava_proj:     0
0.11.183.953 I load_hparams: minicpmv_version:   0
0.11.183.954 I load_hparams: n_merge:            1
0.11.183.954 I load_hparams: n_wa_pattern: 0
0.11.183.955 I load_hparams: image_min_pixels:   92160
0.11.183.955 I load_hparams: image_max_pixels:   645120
0.11.183.955 I
0.11.183.957 I load_hparams: model size:         167.00 MiB
0.11.183.958 I load_hparams: metadata size:      0.00 MiB                                                              |0.11.349.121 I warmup: warmup with image size = 768 x 768
0.11.349.840 I reserve_compute_meta:        CPU compute buffer size =    13.53 MiB
0.11.349.849 I reserve_compute_meta: graph splits = 1, nodes = 21
0.11.349.854 I warmup: flash attention is enabled
0.11.350.672 I clip_ctx: CLIP using CPU backend
0.11.350.813 I load_hparams: projector:          gemma4ua
0.11.350.814 I load_hparams: n_embd:             640
0.11.350.814 I load_hparams: n_head:             1
0.11.350.814 I load_hparams: n_ff:               0
0.11.350.815 I load_hparams: n_layer:            0
0.11.350.815 I load_hparams: ffn_op:             gelu_quick
0.11.350.816 I load_hparams: projection_dim:     3840
0.11.350.816 I
--- audio hparams ---
0.11.350.816 I load_hparams: n_mel_bins:         640
0.11.350.817 I load_hparams: proj_stack_factor:  0
0.11.350.817 I load_hparams: audio_chunk_len:    0
0.11.350.817 I load_hparams: audio_sample_rate:  16000
0.11.350.818 I load_hparams: audio_n_fft:        -1
0.11.350.818 I load_hparams: audio_window_len:   -1
0.11.350.818 I load_hparams: audio_hop_len:      -1
0.11.350.819 I
0.11.350.820 I load_hparams: model size:         167.00 MiB
0.11.350.822 I load_hparams: metadata size:      0.00 MiB                                                              /0.11.464.651 I warmup: warmup with audio size = 3000
0.11.465.966 I reserve_compute_meta:        CPU compute buffer size =    58.59 MiB
0.11.465.974 I reserve_compute_meta: graph splits = 1, nodes = 4
0.11.465.979 I warmup: flash attention is enabled
0.11.467.854 W init_audio: audio input is in experimental stage and may have reduced quality:
    https://github.com/ggml-org/llama.cpp/discussions/13759
0.11.467.860 I srv    load_model: loaded multimodal model, 'C:\Users\sheld\.cache\huggingface\hub\models--unsloth--gemma-4-12b-it-GGUF\snapshots\3249fa54d5efa384afc552cc6700ad091efd5c39\mmproj-BF16.gguf'
0.11.467.897 I srv    load_model: initializing slots, n_slots = 1                                                      |0.12.301.043 I common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
0.12.301.056 I common_speculative_impl_draft_mtp: - n_max=2, n_min=0, p_min=0.00, n_embd=3840, backend_sampling=1
0.12.301.058 I common_speculative_impl_draft_mtp: - gpu_layers=-1, cache_k=f16, cache_v=f16, ctx_tgt=yes, ctx_dft=yes, devices=[default]
0.12.303.370 I sched_reserve: reserving ...
0.12.304.751 I sched_reserve:        CPU compute buffer size =   313.00 MiB
0.12.304.765 I sched_reserve: graph nodes  = 132
0.12.304.766 I sched_reserve: graph splits = 1
0.12.304.767 I sched_reserve: reserve took 1.39 ms, sched copies = 1                                                   /0.12.401.097 I srv    load_model: speculative decoding context initialized
0.12.401.105 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 262144
0.12.401.196 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB
0.12.401.197 I srv    load_model: use `--cache-ram 0` to disable the prompt cache
0.12.401.198 I srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
0.12.401.199 I srv    load_model: context checkpoints enabled, max = 32, min spacing = 256
0.12.401.252 I srv          init: idle slots will be saved to prompt cache upon starting a new task
0.12.420.530 I init: chat template, example_format: '<|turn>system
<|think|>
You are a helpful assistant<turn|>
<|turn>user
Hello<turn|>
<|turn>model
Hi there<turn|>
<|turn>user
How are you?<turn|>
<|turn>model
'
0.12.422.063 I srv          init: init: chat template, thinking = 1


▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

0.12.422.666 I srv  update_slots: all slots are idle
build      : b9592-ac4cddeb0
model      : unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL
modalities : text, vision, audio

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern
  /image <file>       add an image file
  /audio <file>       add an audio file
  /video <file>       add a video file


>

The command for llama.exe serve and llama-server.exe fails

llama-server.exe -hf unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL --temp 1.0 --top-p 0.95 --top-k 64 --spec-type draft-mtp --spec-draft-n-max 2

Console Output

0.00.066.815 I common_params_handle_remote_preset: looking for remote preset at https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/resolve/main/preset.ini
0.00.178.033 I common_download_file_single_online: HEAD failed, status: 404
0.00.179.453 I common_params_handle_remote_preset: no remote preset found, skipping
0.00.314.265 I common_params_print_info: build 9592 (ac4cddeb0) with Clang 20.1.8 for Windows x86_64
0.00.314.270 I log_info: verbosity = 4 (adjust with the `-lv N` CLI arg)
0.00.314.270 I device_info:
0.00.314.279 I   - CPU     : Intel(R) Xeon(R) E-2276M  CPU @ 2.80GHz (32502 MiB, 20610 MiB free)
0.00.314.331 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.314.337 W srv  llama_server: embeddings enabled with n_batch (2048) > n_ubatch (512)
0.00.314.338 W srv  llama_server: setting n_batch = n_ubatch = 512 to avoid assertion failure
0.00.314.340 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.314.427 I srv          init: running without SSL
0.00.314.455 I srv          init: using 11 threads for HTTP server
0.00.314.720 W srv  llama_server: -----------------
0.00.314.723 W srv  llama_server: Built-in tools are enabled, do not expose server to untrusted environments
0.00.314.723 W srv  llama_server: This feature is EXPERIMENTAL and may be changed in the future
0.00.314.723 W srv  llama_server: -----------------
0.00.314.727 I srv         start: binding port with default address family
0.00.322.499 I srv  llama_server: loading model
0.00.322.578 I srv    load_model: loading model 'C:\Users\sheld\.cache\huggingface\hub\models--unsloth--gemma-4-12b-it-GGUF\snapshots\3249fa54d5efa384afc552cc6700ad091efd5c39\gemma-4-12b-it-UD-Q4_K_XL.gguf'
0.00.610.332 I srv    load_model: [mtmd] estimated worst-case memory usage of mmproj is 340.39 MiB
0.01.341.492 E llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this is normal during memory fitting)
0.01.413.598 W srv    load_model: [spec] failed to measure draft model memory: failed to create llama_context from model
0.01.413.661 I common_init_result: fitting params to device memory ...
0.01.413.661 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.01.413.665 I common_params_fit_impl: getting device memory data for initial parameters:
...
0.11.958.128 W llama_init_from_model: model default pooling_type is [-1], but [4] was specified
0.11.958.220 I llama_context: constructing llama_context
0.11.958.228 I llama_context: n_seq_max     = 4
0.11.958.229 I llama_context: n_ctx         = 262144
0.11.958.230 I llama_context: n_ctx_seq     = 262144
0.11.958.230 I llama_context: n_batch       = 512
0.11.958.231 I llama_context: n_ubatch      = 512
0.11.958.232 I llama_context: causal_attn   = 1
0.11.958.233 I llama_context: flash_attn    = auto
0.11.958.233 I llama_context: kv_unified    = true
0.11.958.237 I llama_context: freq_base     = 1000000.0
0.11.958.241 I llama_context: freq_scale    = 1
0.11.958.241 I llama_context: n_rs_seq      = 0
0.11.958.242 I llama_context: n_outputs_max = 512
0.11.959.178 I llama_context:        CPU  output buffer size =     4.06 MiB
0.11.959.182 I llama_kv_cache_iswa: creating non-SWA KV cache, size = 262144 cells
0.11.961.705 W llama_kv_cache: layer   3: sharing with layer 47. k = 0000023AD97DA080, v = 0000023AE97DA080
0.11.961.729 I llama_kv_cache: size =  512.00 MiB (262144 cells,   1 layers,  4/1 seqs), K (f16):  256.00 MiB, V (f16):  256.00 MiB
0.11.961.730 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 512
0.11.961.730 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 512
0.11.961.749 I D:/a/llama.cpp/llama.cpp/src/llama-graph.cpp:2952: GGML_ASSERT(inp != nullptr && "missing result_norm/result_embd tensor") failed

Operating systems

Windows

GGML backends

CUDA, Vulkan

Hardware

nvidia-smi

Wed Jun 10 19:55:17 2026

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 596.59 Driver Version: 596.59 CUDA Version: 13.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Driver-Model | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Quadro T2000 WDDM | 00000000:01:00.0 Off | N/A |
| N/A 50C P8 4W / 35W | 0MiB / 4096MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 Quadro RTX 5000 WDDM | 00000000:0A:00.0 On | Off |
| 34% 31C P8 12W / 230W | 724MiB / 16384MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

Models

Model unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL

Problem description & steps to reproduce

Run command

llama-server.exe -hf unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL --temp 1.0 --top-p 0.95 --top-k 64 --spec-type draft-mtp --spec-draft-n-max 2

First Bad Commit

No response

Relevant log output

Logs

0.16.657.611 I llama_kv_cache_iswa: creating non-SWA KV cache, size = 262144 cells
0.16.659.441 W llama_kv_cache: layer   3: sharing with layer 47. k = 000001306302A080, v = 000001307302A080
0.16.659.455 I llama_kv_cache: size =  512.00 MiB (262144 cells,   1 layers,  4/1 seqs), K (f16):  256.00 MiB, V (f16):  256.00 MiB
0.16.659.456 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 512
0.16.659.457 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 512
0.16.659.469 I llama_kv_cache_iswa: creating     SWA KV cache, size = 4608 cells
0.16.659.518 W llama_kv_cache: layer   0: sharing with layer 46. k = 00000130DAC35080, v = 00000130DBE35080
0.16.659.520 W llama_kv_cache: layer   1: sharing with layer 46. k = 00000130DAC35080, v = 00000130DBE35080
0.16.659.521 W llama_kv_cache: layer   2: sharing with layer 46. k = 00000130DAC35080, v = 00000130DBE35080
0.16.659.525 I llama_kv_cache: size =  108.00 MiB (  4608 cells,   3 layers,  4/1 seqs), K (f16):   54.00 MiB, V (f16):   54.00 MiB
0.16.659.526 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 256
0.16.659.527 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 256
0.16.659.536 I sched_reserve: reserving ...
D:/a/llama.cpp/llama.cpp/src/llama-graph.cpp:2952: GGML_ASSERT(inp != nullptr && "missing result_norm/result_embd tensor") failed

[mohamed-em2m]

same bug gemma 12b mtp not working
llama_model_load: error loading model: unknown model architecture: 'gemma4-assistant'

[mpetruc]

+1.
Also posted on the model's HF discussions:

llama-server -m ${models}/gemma4-12b/gemma-4-12b-it-UD-Q8_K_XL.gguf -fit off -c 16384 -ngl 999 -ub 1024 -b 512 --temp 1 --min-p 0.1 --top-p 0.95 --top-k 64 --no-mmap -np 1  --model-draft ${models}/gemma4-12b/MTP/gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 3 --verbose

Getting:

[...]
llama-swap-1  | 0.09.613.462 D sched_reserve: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
llama-swap-1  | 0.09.613.466 D graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
llama-swap-1  | 0.09.614.479 I sched_reserve: Flash Attention was auto, set to enabled
llama-swap-1  | 0.09.614.496 I sched_reserve: resolving fused Gated Delta Net support:
llama-swap-1  | 0.09.614.498 D graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
llama-swap-1  | 0.09.615.415 I sched_reserve: fused Gated Delta Net (autoregressive) enabled
llama-swap-1  | 0.09.615.437 D graph_reserve: reserving a graph for ubatch with n_tokens =   16, n_seqs =  1, n_outputs =   16
llama-swap-1  | 0.09.616.391 I sched_reserve: fused Gated Delta Net (chunked) enabled
llama-swap-1  | 0.09.616.415 D graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  1, n_outputs =    4
llama-swap-1  | 0.09.692.762 D ggml_cuda_host_malloc: failed to allocate 32.52 MiB of pinned memory: out of memory
llama-swap-1  | 0.09.692.925 D graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
llama-swap-1  | 0.09.694.765 D graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  1, n_outputs =    4
llama-swap-1  | 0.09.696.225 I sched_reserve:      CUDA0 compute buffer size =   139.52 MiB
llama-swap-1  | 0.09.696.250 I sched_reserve:  CUDA_Host compute buffer size =    32.52 MiB
[...]
llama-swap-1  | 0.10.149.165 I llama_prepare_model_devices: using device CUDA0 (NVIDIA GeForce RTX 3080 Laptop GPU) (0000:01:00.0) - 0 MiB free
[...]
llama-swap-1  | 0.10.699.523 E llama_model_load: error loading model: vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)
llama-swap-1  | 0.10.699.559 E llama_model_load_from_file_impl: failed to load model
llama-swap-1  | 0.10.699.564 E srv    load_model: failed to load draft model, '/models/gemma4-12b/MTP/gemma-4-12b-it-Q8_0-MTP.gguf'
llama-swap-1  | 0.10.699.575 I srv    operator(): operator(): cleaning up before exit...
llama-swap-1  | 0.10.701.546 E srv  llama_server: exiting due to model loading error
llama-swap-1  | 0.10.701.672 D ~llama_context:      CUDA0 compute buffer size is 139.5177 MiB, matches expectation of 139.5177 MiB
llama-swap-1  | 0.10.701.674 D ~llama_context:  CUDA_Host compute buffer size is  32.5196 MiB, matches expectation of  32.5196 MiB
[...]

./llama-server --version
version: 9590 (d2462f8f7)
built with GNU 11.4.0 for Linux x86_64

Nvidia RTX 3080 t1, 16Gb VRAM

Edit:

Please disregard my message above: turns out there was something using VRAM that i was not aware of. After restarting MTP works correctly. Getting 44tps vs 25 without MTP. Nice. Thanks you!

[mohamed-em2m]

i fixed it by download latest commit of repo it work so good from 30 to 69 t/s on l4 gpu

[sheldonrobinson]

@mohamed-em2m @mpetruc which model are you using? This issue occurs on Windows.

[mohamed-em2m]

i use unsloth/gemma-4-12b-it-GGUF:UD-Q4_K_XL on ubuntu 22 gpu l4 24 vram

[ServeurpersoCom]

This model works perfectly for me.

[MoE-Vision-Gemma-4-26B-A4B-IT-MTP]
m = unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q8_K_XL.gguf
mm = unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf
md = unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-MTP-BF16.gguf
spec-type = draft-mtp
spec-draft-n-max = 2
c = 262144

-> EDIT: To be tested under Windows

[mpapili]

I can recreate this bug today on main as of a few minutes ago (latest commit 1593d56) - built for ROCm.

Qwen3.5-397B with MTP from unsloth (worked fine 2 days ago with Vulkan build).

[sheldonrobinson]

I'm beginning to think this is hardware related.
Vulkan, unsloth/gemma-4-e2b-it-gguf:UD-Q4_K_XL
Does not work on T2000
Works on Quadro RTX 5000

Is anyone else running an eGPU setup?

systeminfo
OS Name:                       Microsoft Windows 11 Pro for Workstations
OS Version:                    10.0.26200 N/A Build 26200
OS Manufacturer:               Microsoft Corporation
OS Configuration:              Standalone Workstation
OS Build Type:                 Multiprocessor Free
System Manufacturer:           LENOVO
System Model:                  20QTCTO1WW
System Type:                   x64-based PC
Processor(s):                  1 Processor(s) Installed.
                               [01]: Intel64 Family 6 Model 158 Stepping 13 GenuineIntel ~2808 Mhz
BIOS Version:                  LENOVO N2OET70W (1.57 ), 3/24/2026

nvidia-smi
Sat Jun 13 15:56:06 2026
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 516.40       Driver Version: 516.40       CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro RTX 5000    WDDM  | 00000000:0A:00.0  On |                  Off |
| 34%   31C    P8    12W / 230W |  11401MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     29752    C+G   ...llamacpp\llama-server.exe    N/A      |
+-----------------------------------------------------------------------------+

[gitdode]

Following Llama-server deployment with unsloth/gemma-4-26B-A4B-it-GGUF and version: 9430 (d48a56eff) like this:

llama-server --model gemma-4-26B-A4B-it-UD-Q4_K_M.gguf --mmproj mmproj-F16.gguf --model-draft mtp-gemma-4-26B-A4B-it.gguf --temp 1.0 --top-p 0.95 --top-k 64 --spec-type draft-mtp --spec-draft-n-max 4 --host 0.0.0.0

I also got:

E llama_model_load: error loading model: unknown model architecture: 'gemma4-assistant'

With a version: 9737 (67e9fd3b7) just built I get:

E llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this warning is normal during memory fitting)

which is obviously okay, and then:

I srv load_model: speculative decoding context initialized

And now I get ~15 instead of ~8 t/s 🥳

This is on a Framework 13 with 16 × AMD Ryzen AI 7 350 w/ Radeon 860M and 32 GB of RAM running Debian forky/sid.

[liminfei-amd]

Root cause: the server builds the speculative draft context and the MTP context from the target context params, which carry embeddings = true (and a pooling type). The draft/MTP graphs only emit draft logits and have no embeddings output, so creating that context with embeddings/pooling enabled fails and the server aborts in load_model. llama-cli does not copy those target-side params into the speculative context, which is why it loads there. (This lines up with @CISC's earlier note on #24480 that MTP contexts do not need to set the embeddings tensor.)

Fix (#24942): before llama_init_from_model() for the draft and MTP contexts, set embeddings = false and pooling_type = LLAMA_POOLING_TYPE_NONE, so the speculative context is built for logits only. The target context is unchanged.

Validation (gfx1151, current master): unpatched server aborts in load_model; patched server loads and serves MTP speculative decoding end to end (output matching the llama-cli run). git apply --check is clean.

Scope: this fixes the embeddings/pooling load failure on the draft/MTP path, which is the common failure in this report. A few comments here mention hardware-specific variants/segfaults that look like separate problems — if your case is not fixed by this branch, that is likely a different issue and worth a focused report. Testing on the various platforms in this thread would help confirm coverage.