Eval bug: Nemotron-H-47B-Reasoning always reprocesses prompt (even after #16382)

[aoleg]

Name and Version

llama-cli --version
load_backend: loaded RPC backend from s:\llm\llama.vulkan\ggml-rpc.dll
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (Intel Corporation) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from s:\llm\llama.vulkan\ggml-vulkan.dll
load_backend: loaded CPU backend from s:\llm\llama.vulkan\ggml-cpu-alderlake.dll
version: 6686 (128d522)
built with clang version 19.1.5 for x86_64-pc-windows-msvc

Operating systems

Windows

GGML backends

Vulkan

Hardware

Intel i9-12900K + 1x RTX 5090 32GB

Models

bartowski\nvidia_Nemotron-H-47B-Reasoning-128K-GGUF\nvidia_Nemotron-H-47B-Reasoning-128K-Q4_K_M.gguf
https://huggingface.co/bartowski/nvidia_Nemotron-H-47B-Reasoning-128K-GGUF

Problem description & steps to reproduce

PR #16382 was supposed to fix prompt reprocessing for hybrid models, but Nemotron-H-47B-Reasoning is still reprocessing every time; context caching does not seem to be working. I originally opened this issue in #16033; it was merged with #15677, but PR #16382 does not seem to fix it for Nemotron-H-47B.

When I run llama-server, it does not appear to respect the values for kv-cache; moreover, it reprocesses the prompt every time, which gets quite long in the end.

Particularly unrealistic values for kv cache (32768 context size):
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 32768
llama_context: n_ctx_per_seq = 32768
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = enabled
llama_context: kv_unified = false
llama_context: freq_base = 10000.0
llama_context: freq_scale = 1
llama_context: n_ctx_per_seq (32768) < n_ctx_train (1048576) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host output buffer size = 0.50 MiB
llama_kv_cache: Vulkan0 KV buffer size = 640.00 MiB
llama_kv_cache: size = 640.00 MiB ( 32768 cells, 5 layers, 1/1 seqs), K (f16): 320.00 MiB, V (f16): 320.00 MiB
llama_memory_recurrent: Vulkan0 RS buffer size = 730.55 MiB
llama_memory_recurrent: size = 730.55 MiB ( 1 cells, 98 layers, 1 seqs), R (f32): 10.55 MiB, S (f32): 720.00 MiB
llama_context: Vulkan0 compute buffer size = 304.23 MiB
llama_context: Vulkan_Host compute buffer size = 248.36 MiB
llama_context: graph nodes = 2750
llama_context: graph splits = 182

And reprocessing every time despite saving context (here, I simply re-generated the reply, which normally causes no reprocessing at all):
slot update_slots: id 0 | task 0 | saved context checkpoint 1 of 3 (pos_min = 8508, pos_max = 8508, size = 730.548 MiB)
...
slot update_slots: id 0 | task 169 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 8509
slot update_slots: id 0 | task 169 | n_past = 8509, cache_tokens.size() = 8672, seq_id = 0, pos_min = 8671, n_swa = 1
slot update_slots: id 0 | task 169 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see #13194 (comment))

main: server is listening on http://127.0.0.1:8081 - starting the main loop
srv update_slots: all slots are idle
common_sampler_types_from_names: unable to match sampler by name 'tfs_z'
common_sampler_types_from_names: unable to match sampler by name 'typical_p'
slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 0 | task 0 | processing task
slot update_slots: id 0 | task 0 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 8509
slot update_slots: id 0 | task 0 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.240686
slot update_slots: id 0 | task 0 | n_past = 2048, memory_seq_rm [2048, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 4096, n_tokens = 2048, progress = 0.481373
slot update_slots: id 0 | task 0 | n_past = 4096, memory_seq_rm [4096, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 6144, n_tokens = 2048, progress = 0.722059
slot update_slots: id 0 | task 0 | n_past = 6144, memory_seq_rm [6144, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 8192, n_tokens = 2048, progress = 0.962745
slot update_slots: id 0 | task 0 | n_past = 8192, memory_seq_rm [8192, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 8509, n_tokens = 317, progress = 1.000000
slot update_slots: id 0 | task 0 | prompt done, n_past = 8509, n_tokens = 317
slot update_slots: id 0 | task 0 | saved context checkpoint 1 of 3 (pos_min = 8508, pos_max = 8508, size = 730.548 MiB)
slot release: id 0 | task 0 | stop processing: n_past = 8672, truncated = 0
slot print_timing: id 0 | task 0 |
prompt eval time = 172848.79 ms / 8509 tokens ( 20.31 ms per token, 49.23 tokens per second)
eval time = 58931.25 ms / 164 tokens ( 359.34 ms per token, 2.78 tokens per second)
total time = 231780.04 ms / 8673 tokens
srv update_slots: all slots are idle
srv log_server_r: request: POST /completion 127.0.0.1 200
common_sampler_types_from_names: unable to match sampler by name 'tfs_z'
common_sampler_types_from_names: unable to match sampler by name 'typical_p'
slot get_availabl: id 0 | task 0 | selected slot by lcs similarity, lcs_len = 8509, similarity = 0.981 (> 0.100 thold)
slot launch_slot_: id 0 | task 169 | processing task
slot update_slots: id 0 | task 169 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 8509
slot update_slots: id 0 | task 169 | n_past = 8509, cache_tokens.size() = 8672, seq_id = 0, pos_min = 8671, n_swa = 1
slot update_slots: id 0 | task 169 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com//pull/13194#issuecomment-2868343055)
slot update_slots: id 0 | task 169 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id 0 | task 169 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.240686

First Bad Commit

No response

Relevant log output

s:\llm\llama.vulkan>llama-server     --model "s:\lmstudio\models\bartowski\nvidia_Nemotron-H-47B-Reasoning-128K-GGUF\nvidia_Nemotron-H-47B-Reasoning-128K-Q4_K_M.gguf"     --alias "nvidia/Nemotron-H-47B-Reasoning-128K-Q4_K_M"     --ctx-size 32768     -fa on     -ctk f16 -ctv f16     -ngl 999     --host 127.0.0.1     --port 8081     --batch-size 2048     --no-mmap
load_backend: loaded RPC backend from s:\llm\llama.vulkan\ggml-rpc.dll
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (Intel Corporation) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from s:\llm\llama.vulkan\ggml-vulkan.dll
load_backend: loaded CPU backend from s:\llm\llama.vulkan\ggml-cpu-alderlake.dll
build: 6686 (128d522c) with clang version 19.1.5 for x86_64-pc-windows-msvc
system info: n_threads = 16, n_threads_batch = 16, total_threads = 24

system_info: n_threads = 16 (n_threads_batch = 16) / 24 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 8081, http threads: 23
main: loading model
srv    load_model: loading model 's:\lmstudio\models\bartowski\nvidia_Nemotron-H-47B-Reasoning-128K-GGUF\nvidia_Nemotron-H-47B-Reasoning-128K-Q4_K_M.gguf'
llama_model_load_from_file_impl: using device Vulkan0 (NVIDIA GeForce RTX 5090) (0000:01:00.0) - 31333 MiB free
llama_model_loader: loaded meta data with 48 key-value pairs and 577 tensors from s:\lmstudio\models\bartowski\nvidia_Nemotron-H-47B-Reasoning-128K-GGUF\nvidia_Nemotron-H-47B-Reasoning-128K-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = nemotron_h
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Nemotron H 47B Reasoning 128K
llama_model_loader: - kv   3:                           general.finetune str              = Reasoning-128k
llama_model_loader: - kv   4:                           general.basename str              = Nemotron-H
llama_model_loader: - kv   5:                         general.size_label str              = 47B
llama_model_loader: - kv   6:                            general.license str              = other
llama_model_loader: - kv   7:                       general.license.name str              = nvidia-internal-scientific-research-a...
llama_model_loader: - kv   8:                       general.license.link str              = https://www.nvidia.com/en-us/agreemen...
llama_model_loader: - kv   9:                               general.tags arr[str,3]       = ["nvidia", "pytorch", "text-generation"]
llama_model_loader: - kv  10:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv  11:                     nemotron_h.block_count u32              = 98
llama_model_loader: - kv  12:                  nemotron_h.context_length u32              = 1048576
llama_model_loader: - kv  13:                nemotron_h.embedding_length u32              = 8192
llama_model_loader: - kv  14:             nemotron_h.feed_forward_length arr[i32,98]      = [0, 30720, 0, 30720, 0, 30720, 0, 307...
llama_model_loader: - kv  15:            nemotron_h.attention.head_count u32              = 64
llama_model_loader: - kv  16:         nemotron_h.attention.head_count_kv arr[i32,98]      = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
llama_model_loader: - kv  17: nemotron_h.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  18:    nemotron_h.attention.layer_norm_epsilon f32              = 0.000010
llama_model_loader: - kv  19:                      nemotron_h.vocab_size u32              = 131072
llama_model_loader: - kv  20:            nemotron_h.rope.dimension_count u32              = 128
llama_model_loader: - kv  21:                 nemotron_h.ssm.conv_kernel u32              = 4
llama_model_loader: - kv  22:                  nemotron_h.ssm.state_size u32              = 256
llama_model_loader: - kv  23:                 nemotron_h.ssm.group_count u32              = 8
llama_model_loader: - kv  24:                  nemotron_h.ssm.inner_size u32              = 16384
llama_model_loader: - kv  25:              nemotron_h.ssm.time_step_rank u32              = 256
llama_model_loader: - kv  26:          nemotron_h.rope.scaling.finetuned bool             = false
llama_model_loader: - kv  27:            nemotron_h.attention.key_length u32              = 128
llama_model_loader: - kv  28:          nemotron_h.attention.value_length u32              = 128
llama_model_loader: - kv  29:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  30:                         tokenizer.ggml.pre str              = tekken
llama_model_loader: - kv  31:                      tokenizer.ggml.tokens arr[str,131072]  = ["<unk>", "<s>", "</s>", "[INST]", "[...
llama_model_loader: - kv  32:                  tokenizer.ggml.token_type arr[i32,131072]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  33:                      tokenizer.ggml.merges arr[str,269443]  = ["─а ─а", "─а t", "e r", "i n", "─а ─...
llama_model_loader: - kv  34:                tokenizer.ggml.eos_token_id u32              = 11
llama_model_loader: - kv  35:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  36:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  37:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  38:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  39:               tokenizer.ggml.add_sep_token bool             = false
llama_model_loader: - kv  40:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  41:                    tokenizer.chat_template str              = {{ '<SPECIAL_10>System\n' }}{%- if mes...
llama_model_loader: - kv  42:               general.quantization_version u32              = 2
llama_model_loader: - kv  43:                          general.file_type u32              = 15
llama_model_loader: - kv  44:                      quantize.imatrix.file str              = /models_out/Nemotron-H-47B-Reasoning-...
llama_model_loader: - kv  45:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav5.txt
llama_model_loader: - kv  46:             quantize.imatrix.entries_count u32              = 206
llama_model_loader: - kv  47:              quantize.imatrix.chunks_count u32              = 822
llama_model_loader: - type  f32:  369 tensors
llama_model_loader: - type q4_K:  181 tensors
llama_model_loader: - type q6_K:   27 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 26.24 GiB (4.82 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 11 ('<SPECIAL_11>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8499 MB
print_info: arch             = nemotron_h
print_info: vocab_only       = 0
print_info: n_ctx_train      = 1048576
print_info: n_embd           = 8192
print_info: n_layer          = 98
print_info: n_head           = 64
print_info: n_head_kv        = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_embd_k_gqa     = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_embd_v_gqa     = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = [0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 30720, 30720, 0, 0, 30720, 30720, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720]
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = -1
print_info: rope scaling     = linear
print_info: freq_base_train  = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 1048576
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 4
print_info: ssm_d_inner      = 16384
print_info: ssm_d_state      = 256
print_info: ssm_dt_rank      = 256
print_info: ssm_n_group      = 8
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = ?B
print_info: model params     = 46.79 B
print_info: general.name     = Nemotron H 47B Reasoning 128K
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 11 '<SPECIAL_11>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 0 '<unk>'
print_info: LF token         = 1010 '─К'
print_info: EOG token        = 11 '<SPECIAL_11>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 98 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 99/99 layers to GPU
load_tensors:      Vulkan0 model buffer size = 26284.26 MiB
load_tensors:  Vulkan_Host model buffer size =   576.00 MiB
load_tensors:          CPU model buffer size =    14.11 MiB
.................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 32768
llama_context: n_ctx_per_seq = 32768
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = false
llama_context: freq_base     = 10000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (32768) < n_ctx_train (1048576) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host  output buffer size =     0.50 MiB
llama_kv_cache:    Vulkan0 KV buffer size =   640.00 MiB
llama_kv_cache: size =  640.00 MiB ( 32768 cells,   5 layers,  1/1 seqs), K (f16):  320.00 MiB, V (f16):  320.00 MiB
llama_memory_recurrent:    Vulkan0 RS buffer size =   730.55 MiB
llama_memory_recurrent: size =  730.55 MiB (     1 cells,  98 layers,  1 seqs), R (f32):   10.55 MiB, S (f32):  720.00 MiB
llama_context:    Vulkan0 compute buffer size =   304.23 MiB
llama_context: Vulkan_Host compute buffer size =   248.36 MiB
llama_context: graph nodes  = 2750
llama_context: graph splits = 182
common_init_from_params: added <SPECIAL_11> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 32768
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv    load_model: load_model: Chat template parsing error: this custom template is not supported, try using --jinja
srv    load_model: load_model: The chat template that comes with this model is not yet supported, falling back to chatml. This may cause the model to output suboptimal responses
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 32768
srv          init: Enable thinking? 0
main: model loaded
main: chat template, chat_template: {%- for message in messages -%}
  {{- '<|im_start|>' + message.role + '
' + message.content + '<|im_end|>
' -}}
{%- endfor -%}
{%- if add_generation_prompt -%}
  {{- '<|im_start|>assistant
' -}}
{%- endif -%}, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://127.0.0.1:8081 - starting the main loop
srv  update_slots: all slots are idle
common_sampler_types_from_names: unable to match sampler by name 'tfs_z'
common_sampler_types_from_names: unable to match sampler by name 'typical_p'
slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 8509
slot update_slots: id  0 | task 0 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.240686
slot update_slots: id  0 | task 0 | n_past = 2048, memory_seq_rm [2048, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 4096, n_tokens = 2048, progress = 0.481373
slot update_slots: id  0 | task 0 | n_past = 4096, memory_seq_rm [4096, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 6144, n_tokens = 2048, progress = 0.722059
slot update_slots: id  0 | task 0 | n_past = 6144, memory_seq_rm [6144, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 8192, n_tokens = 2048, progress = 0.962745
slot update_slots: id  0 | task 0 | n_past = 8192, memory_seq_rm [8192, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 8509, n_tokens = 317, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 8509, n_tokens = 317
slot update_slots: id  0 | task 0 | saved context checkpoint 1 of 3 (pos_min = 8508, pos_max = 8508, size = 730.548 MiB)
slot      release: id  0 | task 0 | stop processing: n_past = 8672, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =  172848.79 ms /  8509 tokens (   20.31 ms per token,    49.23 tokens per second)
       eval time =   58931.25 ms /   164 tokens (  359.34 ms per token,     2.78 tokens per second)
      total time =  231780.04 ms /  8673 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /completion 127.0.0.1 200
common_sampler_types_from_names: unable to match sampler by name 'tfs_z'
common_sampler_types_from_names: unable to match sampler by name 'typical_p'
slot get_availabl: id  0 | task 0 | selected slot by lcs similarity, lcs_len = 8509, similarity = 0.981 (> 0.100 thold)
slot launch_slot_: id  0 | task 169 | processing task
slot update_slots: id  0 | task 169 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 8509
slot update_slots: id  0 | task 169 | n_past = 8509, cache_tokens.size() = 8672, seq_id = 0, pos_min = 8671, n_swa = 1
slot update_slots: id  0 | task 169 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 169 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 169 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.240686
slot update_slots: id  0 | task 169 | n_past = 2048, memory_seq_rm [2048, end)
slot update_slots: id  0 | task 169 | prompt processing progress, n_past = 4096, n_tokens = 2048, progress = 0.481373
slot update_slots: id  0 | task 169 | n_past = 4096, memory_seq_rm [4096, end)
slot update_slots: id  0 | task 169 | prompt processing progress, n_past = 6144, n_tokens = 2048, progress = 0.722059
slot update_slots: id  0 | task 169 | n_past = 6144, memory_seq_rm [6144, end)
slot update_slots: id  0 | task 169 | prompt processing progress, n_past = 8192, n_tokens = 2048, progress = 0.962745
slot update_slots: id  0 | task 169 | n_past = 8192, memory_seq_rm [8192, end)
slot update_slots: id  0 | task 169 | prompt processing progress, n_past = 8509, n_tokens = 317, progress = 1.000000
slot update_slots: id  0 | task 169 | prompt done, n_past = 8509, n_tokens = 317
slot update_slots: id  0 | task 169 | saved context checkpoint 2 of 3 (pos_min = 8508, pos_max = 8508, size = 730.548 MiB)
slot      release: id  0 | task 169 | stop processing: n_past = 8718, truncated = 0
slot print_timing: id  0 | task 169 |
prompt eval time =  174215.46 ms /  8509 tokens (   20.47 ms per token,    48.84 tokens per second)
       eval time =   75767.02 ms /   210 tokens (  360.80 ms per token,     2.77 tokens per second)
      total time =  249982.48 ms /  8719 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /completion 127.0.0.1 200
common_sampler_types_from_names: unable to match sampler by name 'tfs_z'
common_sampler_types_from_names: unable to match sampler by name 'typical_p'
slot get_availabl: id  0 | task 169 | selected slot by lcs similarity, lcs_len = 8509, similarity = 0.976 (> 0.100 thold)
slot launch_slot_: id  0 | task 384 | processing task
slot update_slots: id  0 | task 384 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 8509
slot update_slots: id  0 | task 384 | n_past = 8509, cache_tokens.size() = 8718, seq_id = 0, pos_min = 8717, n_swa = 1
slot update_slots: id  0 | task 384 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 384 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 384 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.240686

[aoleg]

I just tested, and the same issue still applies to Jamba Mini 1.7 as well, model https://huggingface.co/bartowski/ai21labs_AI21-Jamba-Mini-1.7-GGUF
bartowski\ai21labs_AI21-Jamba-Mini-1.7-GGUF\ai21labs_AI21-Jamba-Mini-1.7-Q6_K_L.gguf

The SWA checkpoint is saved but never reused, and I am still getting full prompt reprocessing every time.

[Kyleg142]

Can confirm the issue persists with NVIDIA Nemotron 12B v2.

Model: bartowski/nvidia_NVIDIA-Nemotron-Nano-12B-v2-GGUF:Q8_0

slot update_slots: id  0 | task 3666 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 3666 | erased invalidated context checkpoint (pos_min = 5049, pos_max = 5049, n_swa = 1, size = 143.938 MiB)

[ddh0]

To be honest, I'm not sure what could be causing this issue. Could either of you (@aoleg or @Kyleg142) please explain more about how you're using the models when the issue arises? Are you using the built-in llama-server WebUI, or an external frontend? Are you starting a fresh conversation, or are you loading up an existing chat?

When I run llama-server, it does not appear to respect the values for kv-cache

@aoleg - could you please explain what you mean by this?

My understanding is that the current implementation of context checkpointing might still run into some edge-cases where it's not as seamless as a regular transformer model (where it's easy to resume from any point in the context) but it definitely should not have to re-process the entire context for every new message.

Paging @ggerganov - if you have any idea what might be causing this, and if you would be so kind as to point me in the right direction, I would be happy to start working on a fix.

[Kyleg142]

@ddh0 I am simply using the llamacpp webUI at http://localhost:8080, the issue occurs every time the model is prompted - there is no specific instance where it occurs. Here are some more details of my setup for reference:

Version: llama.cpp-hip b6686-1 (AMD ROCm)
GPU: 7900xt
VRAM Usage: 17403MB/20464MB

Doesn't look like it's bound to GPU hardware or API since OP is getting the same issue on Nvidia with Vulkan.

Here is my launch command for llama-server:
llama-server -hf bartowski/nvidia_NVIDIA-Nemotron-Nano-12B-v2-GGUF:Q8_0 -c 131072 --jinja --repeat-penalty 1.05 --temp 0.6 --top-p 0.95

[aoleg]

To be honest, I'm not sure what could be causing this issue. Could either of you (@aoleg or @Kyleg142) please explain more about how you're using the models when the issue arises? Are you using the built-in llama-server WebUI, or an external frontend? Are you starting a fresh conversation, or are you loading up an existing chat?

I am using SillyTavern, loading up an existing chat with existing history and trying to continue. When I do this, each new message I send causes full prompt reprocessing. This does not happen with non-hybrid models. Even worse, simply re-generating a message (zero changes in context) still causes full reprocessing.

If there is anything particular that I can do to help isolate the issue, please let me know.

When I run llama-server, it does not appear to respect the values for kv-cache

@aoleg - could you please explain what you mean by this?

What I was meaning to say: there is no way for 32K context to occupy that little VRAM; please correct me if I am wrong.
llama_kv_cache: size = 640.00 MiB ( 32768 cells, 5 layers, 1/1 seqs), K (f16): 320.00 MiB, V (f16): 320.00 MiB

[ddh0]

I am using SillyTavern, loading up an existing chat with existing history and trying to continue. When I do this, each new message I send causes full prompt reprocessing.

Yeah, that would do it. In order for the context checkpointing to be maximally useful you need to start from a fresh chat so that it can save checkpoints at every message. Since you started from an existing chat, it had no opportunity to save checkpoints for the existing messages. SillyTavern might also be doing other things behind the scenes that can invalidate checkpoints (lorebooks, author's notes,...).

Could you try starting a brand new chat with minimal additional features, so that the code has a chance to make checkpoints as it goes?

What I was meaning to say: there is no way for 32K context to occupy that little VRAM; please correct me if I am wrong.
llama_kv_cache: size = 640.00 MiB ( 32768 cells, 5 layers, 1/1 seqs), K (f16): 320.00 MiB, V (f16): 320.00 MiB

That looks fine :) the KV cache is smaller than you're used to seeing becuase it's a hybrid model, and it only has to keep a standard KV cache for the 5 transformer layers in Nemotron-H-47B. The rest of the layers are recurrent.

[ddh0]

For anyone reading this who is having issues with context checkpointing not being as helpful as they would like, in addition to my previous comment, you can also try setting --ctx-checkpoints 32 or some other high value (the default is 3 at the time of writing). This lets the server keep more checkpoints in memory. The checkpoints are pretty small and they're kept in RAM, not VRAM, so there's barely any downside.

[ddh0]

@Kyleg142

I am now able to reproduce your issue with bartowski's nvidia_NVIDIA-Nemotron-Nano-12B-v2-Q8_0.gguf. The issue goes away if you remove the --jinja argument, but that leads to broken chat templating. So it seems that something about their custom chat template causes checkpoints to be continually invalidated. I will need to look more into this.

The easiest fix for this problem, IMO, would be to simplify their jinja chat template by hand, keeping the same core functionality but removing the problematic parts. I don't know which parts are problematic yet, though.

[aoleg]

@ddh0 I started a new chat in SillyTavern. Here's the log:

llama-server     --model "s:\lmstudio\models\bartowski\nvidia_Nemotron-H-47B-Reasoning-128K-GGUF\nvidia_Nemotron-H-47B-Reasoning-128K-Q4_K_M.gguf"     --alias "nvidia/Nemotron-H-47B-Reasoning-128K-Q4_K_M"     --ctx-size 32768     --ctx-checkpoints 32     -ngl 999     -fa on     -ctk f16 -ctv f16     --host 127.0.0.1     --port 8081     --no-mmap     --jinja     --threads 12
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from S:\llm\llama\ggml-cuda.dll
load_backend: loaded RPC backend from S:\llm\llama\ggml-rpc.dll
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (Intel Corporation) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from S:\llm\llama\ggml-vulkan.dll
load_backend: loaded CPU backend from S:\llm\llama\ggml-cpu-alderlake.dll
build: 6898 (d1d250eb) with clang version 19.1.5 for x86_64-pc-windows-msvc
system info: n_threads = 12, n_threads_batch = 12, total_threads = 24

system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CUDA : ARCHS = 500,610,700,750,800,860,890,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 8081, http threads: 23
main: loading model
srv    load_model: loading model 's:\lmstudio\models\bartowski\nvidia_Nemotron-H-47B-Reasoning-128K-GGUF\nvidia_Nemotron-H-47B-Reasoning-128K-Q4_K_M.gguf'
llama_model_load_from_file_impl: skipping device Vulkan0 (NVIDIA GeForce RTX 5090) with id 0000:01:00.0 - already using device CUDA0 (NVIDIA GeForce RTX 5090) with the same id
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 5090) (0000:01:00.0) - 30841 MiB free
llama_model_loader: max stdio successfully set to 2048
llama_model_loader: loaded meta data with 48 key-value pairs and 577 tensors from s:\lmstudio\models\bartowski\nvidia_Nemotron-H-47B-Reasoning-128K-GGUF\nvidia_Nemotron-H-47B-Reasoning-128K-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = nemotron_h
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Nemotron H 47B Reasoning 128K
llama_model_loader: - kv   3:                           general.finetune str              = Reasoning-128k
llama_model_loader: - kv   4:                           general.basename str              = Nemotron-H
llama_model_loader: - kv   5:                         general.size_label str              = 47B
llama_model_loader: - kv   6:                            general.license str              = other
llama_model_loader: - kv   7:                       general.license.name str              = nvidia-internal-scientific-research-a...
llama_model_loader: - kv   8:                       general.license.link str              = https://www.nvidia.com/en-us/agreemen...
llama_model_loader: - kv   9:                               general.tags arr[str,3]       = ["nvidia", "pytorch", "text-generation"]
llama_model_loader: - kv  10:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv  11:                     nemotron_h.block_count u32              = 98
llama_model_loader: - kv  12:                  nemotron_h.context_length u32              = 1048576
llama_model_loader: - kv  13:                nemotron_h.embedding_length u32              = 8192
llama_model_loader: - kv  14:             nemotron_h.feed_forward_length arr[i32,98]      = [0, 30720, 0, 30720, 0, 30720, 0, 307...
llama_model_loader: - kv  15:            nemotron_h.attention.head_count u32              = 64
llama_model_loader: - kv  16:         nemotron_h.attention.head_count_kv arr[i32,98]      = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
llama_model_loader: - kv  17: nemotron_h.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  18:    nemotron_h.attention.layer_norm_epsilon f32              = 0.000010
llama_model_loader: - kv  19:                      nemotron_h.vocab_size u32              = 131072
llama_model_loader: - kv  20:            nemotron_h.rope.dimension_count u32              = 128
llama_model_loader: - kv  21:                 nemotron_h.ssm.conv_kernel u32              = 4
llama_model_loader: - kv  22:                  nemotron_h.ssm.state_size u32              = 256
llama_model_loader: - kv  23:                 nemotron_h.ssm.group_count u32              = 8
llama_model_loader: - kv  24:                  nemotron_h.ssm.inner_size u32              = 16384
llama_model_loader: - kv  25:              nemotron_h.ssm.time_step_rank u32              = 256
llama_model_loader: - kv  26:          nemotron_h.rope.scaling.finetuned bool             = false
llama_model_loader: - kv  27:            nemotron_h.attention.key_length u32              = 128
llama_model_loader: - kv  28:          nemotron_h.attention.value_length u32              = 128
llama_model_loader: - kv  29:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  30:                         tokenizer.ggml.pre str              = tekken
llama_model_loader: - kv  31:                      tokenizer.ggml.tokens arr[str,131072]  = ["<unk>", "<s>", "</s>", "[INST]", "[...
llama_model_loader: - kv  32:                  tokenizer.ggml.token_type arr[i32,131072]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  33:                      tokenizer.ggml.merges arr[str,269443]  = ["─а ─а", "─а t", "e r", "i n", "─а ─...
llama_model_loader: - kv  34:                tokenizer.ggml.eos_token_id u32              = 11
llama_model_loader: - kv  35:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  36:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  37:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  38:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  39:               tokenizer.ggml.add_sep_token bool             = false
llama_model_loader: - kv  40:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  41:                    tokenizer.chat_template str              = {{ '<SPECIAL_10>System\n' }}{%- if mes...
llama_model_loader: - kv  42:               general.quantization_version u32              = 2
llama_model_loader: - kv  43:                          general.file_type u32              = 15
llama_model_loader: - kv  44:                      quantize.imatrix.file str              = /models_out/Nemotron-H-47B-Reasoning-...
llama_model_loader: - kv  45:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav5.txt
llama_model_loader: - kv  46:             quantize.imatrix.entries_count u32              = 206
llama_model_loader: - kv  47:              quantize.imatrix.chunks_count u32              = 822
llama_model_loader: - type  f32:  369 tensors
llama_model_loader: - type q4_K:  181 tensors
llama_model_loader: - type q6_K:   27 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 26.24 GiB (4.82 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 11 ('<SPECIAL_11>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8499 MB
print_info: arch             = nemotron_h
print_info: vocab_only       = 0
print_info: n_ctx_train      = 1048576
print_info: n_embd           = 8192
print_info: n_layer          = 98
print_info: n_head           = 64
print_info: n_head_kv        = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_embd_k_gqa     = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_embd_v_gqa     = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = [0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 30720, 30720, 0, 0, 30720, 30720, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720]
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = -1
print_info: rope scaling     = linear
print_info: freq_base_train  = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 1048576
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 4
print_info: ssm_d_inner      = 16384
print_info: ssm_d_state      = 256
print_info: ssm_dt_rank      = 256
print_info: ssm_n_group      = 8
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = ?B
print_info: model params     = 46.79 B
print_info: general.name     = Nemotron H 47B Reasoning 128K
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 11 '<SPECIAL_11>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 0 '<unk>'
print_info: LF token         = 1010 '─К'
print_info: EOG token        = 11 '<SPECIAL_11>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 98 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 99/99 layers to GPU
load_tensors:        CUDA0 model buffer size = 26298.37 MiB
load_tensors:          CPU model buffer size =   576.00 MiB
................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 32768
llama_context: n_ctx_per_seq = 32768
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = false
llama_context: freq_base     = 10000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (32768) < n_ctx_train (1048576) -- the full capacity of the model will not be utilized
llama_context:  CUDA_Host  output buffer size =     0.50 MiB
llama_kv_cache:      CUDA0 KV buffer size =   640.00 MiB
llama_kv_cache: size =  640.00 MiB ( 32768 cells,   5 layers,  1/1 seqs), K (f16):  320.00 MiB, V (f16):  320.00 MiB
llama_memory_recurrent:      CUDA0 RS buffer size =   730.55 MiB
llama_memory_recurrent: size =  730.55 MiB (     1 cells,  98 layers,  1 seqs), R (f32):   10.55 MiB, S (f32):  720.00 MiB
llama_context:      CUDA0 compute buffer size =   304.23 MiB
llama_context:  CUDA_Host compute buffer size =    80.01 MiB
llama_context: graph nodes  = 2750
llama_context: graph splits = 2
common_init_from_params: added <SPECIAL_11> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 32768
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 32768
srv          init: Enable thinking? 0
main: model loaded
main: chat template, chat_template: {{ '<SPECIAL_10>System
' }}{%- if messages and messages[0]['role'] == 'system' -%}{{ messages[0]['content'].strip() }}{%- endif -%}{% for message in (messages[1:] if messages[0]['role'] == 'system' else messages) %}{%- if message['role'] == 'user' -%}{{ '
<SPECIAL_11>User
' + message['content'].strip() + '
<SPECIAL_11>Assistant
' }}{%- if loop.last -%}{%- if messages[0]['role'] == 'system' -%}{%- if "{'reasoning': True}" in messages[0]['content'] -%}{{ '<think>
' }}{%- elif "{'reasoning': False}" in messages[0]['content'] -%}{{ '<think></think>' }}{%- endif -%}{%- endif -%}{%- endif -%}{%- elif message['role'] == 'assistant' -%}{{ message['content'].strip() }}{%- endif -%}{%- endfor -%}, example_format: '<SPECIAL_10>System
You are a helpful assistant
<SPECIAL_11>User
Hello
<SPECIAL_11>Assistant
Hi there
<SPECIAL_11>User
How are you?
<SPECIAL_11>Assistant
'
main: server is listening on http://127.0.0.1:8081 - starting the main loop
srv  update_slots: all slots are idle
slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 1894
slot update_slots: id  0 | task 0 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 1894, n_tokens = 1894, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 1894, n_tokens = 1894
slot update_slots: id  0 | task 0 | saved context checkpoint 1 of 32 (pos_min = 1893, pos_max = 1893, size = 730.548 MiB)
srv  cancel_tasks: cancel task, id_task = 0
srv  log_server_r: request: POST /completion 127.0.0.1 200
slot      release: id  0 | task 0 | stop processing: n_past = 2553, truncated = 0
srv  update_slots: all slots are idle
slot get_availabl: id  0 | task 0 | selected slot by lcs similarity, lcs_len = 1893, similarity = 0.741 (> 0.100 thold)
slot launch_slot_: id  0 | task 662 | processing task
slot update_slots: id  0 | task 662 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 2539
slot update_slots: id  0 | task 662 | n_past = 1893, cache_tokens.size() = 2553, seq_id = 0, pos_min = 2552, n_swa = 1
slot update_slots: id  0 | task 662 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 662 | erased invalidated context checkpoint (pos_min = 1893, pos_max = 1893, n_swa = 1, size = 730.548 MiB)
slot update_slots: id  0 | task 662 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 662 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.806617
slot update_slots: id  0 | task 662 | n_past = 2048, memory_seq_rm [2048, end)
slot update_slots: id  0 | task 662 | prompt processing progress, n_past = 2539, n_tokens = 491, progress = 1.000000
slot update_slots: id  0 | task 662 | prompt done, n_past = 2539, n_tokens = 491
slot update_slots: id  0 | task 662 | saved context checkpoint 1 of 32 (pos_min = 2538, pos_max = 2538, size = 730.548 MiB)
srv  cancel_tasks: cancel task, id_task = 662
srv  log_server_r: request: POST /completion 127.0.0.1 200
slot      release: id  0 | task 662 | stop processing: n_past = 2981, truncated = 0
srv  update_slots: all slots are idle
slot get_availabl: id  0 | task 662 | selected slot by lcs similarity, lcs_len = 1626, similarity = 0.545 (> 0.100 thold)
slot launch_slot_: id  0 | task 1108 | processing task
slot update_slots: id  0 | task 1108 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 2995
slot update_slots: id  0 | task 1108 | n_past = 1626, cache_tokens.size() = 2981, seq_id = 0, pos_min = 2980, n_swa = 1
slot update_slots: id  0 | task 1108 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 1108 | erased invalidated context checkpoint (pos_min = 2538, pos_max = 2538, n_swa = 1, size = 730.548 MiB)
slot update_slots: id  0 | task 1108 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 1108 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.683806
slot update_slots: id  0 | task 1108 | n_past = 2048, memory_seq_rm [2048, end)
slot update_slots: id  0 | task 1108 | prompt processing progress, n_past = 2995, n_tokens = 947, progress = 1.000000
slot update_slots: id  0 | task 1108 | prompt done, n_past = 2995, n_tokens = 947
slot update_slots: id  0 | task 1108 | saved context checkpoint 1 of 32 (pos_min = 2994, pos_max = 2994, size = 730.548 MiB)
srv  cancel_tasks: cancel task, id_task = 1108
srv  log_server_r: request: POST /completion 127.0.0.1 200
slot      release: id  0 | task 1108 | stop processing: n_past = 3699, truncated = 0
srv  update_slots: all slots are idle
slot get_availabl: id  0 | task 1108 | selected slot by lcs similarity, lcs_len = 1808, similarity = 0.489 (> 0.100 thold)
slot launch_slot_: id  0 | task 1816 | processing task
slot update_slots: id  0 | task 1816 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 3684
slot update_slots: id  0 | task 1816 | n_past = 1808, cache_tokens.size() = 3699, seq_id = 0, pos_min = 3698, n_swa = 1
slot update_slots: id  0 | task 1816 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 1816 | erased invalidated context checkpoint (pos_min = 2994, pos_max = 2994, n_swa = 1, size = 730.548 MiB)
slot update_slots: id  0 | task 1816 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 1816 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.555918
slot update_slots: id  0 | task 1816 | n_past = 2048, memory_seq_rm [2048, end)
slot update_slots: id  0 | task 1816 | prompt processing progress, n_past = 3684, n_tokens = 1636, progress = 1.000000
slot update_slots: id  0 | task 1816 | prompt done, n_past = 3684, n_tokens = 1636
slot update_slots: id  0 | task 1816 | saved context checkpoint 1 of 32 (pos_min = 3683, pos_max = 3683, size = 730.548 MiB)
srv  cancel_tasks: cancel task, id_task = 1816
srv  log_server_r: request: POST /completion 127.0.0.1 200
slot      release: id  0 | task 1816 | stop processing: n_past = 4071, truncated = 0
srv  update_slots: all slots are idle
slot get_availabl: id  0 | task 1816 | selected slot by lcs similarity, lcs_len = 2453, similarity = 0.603 (> 0.100 thold)
slot launch_slot_: id  0 | task 2207 | processing task
slot update_slots: id  0 | task 2207 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 4082
slot update_slots: id  0 | task 2207 | n_past = 2453, cache_tokens.size() = 4071, seq_id = 0, pos_min = 4070, n_swa = 1
slot update_slots: id  0 | task 2207 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 2207 | erased invalidated context checkpoint (pos_min = 3683, pos_max = 3683, n_swa = 1, size = 730.548 MiB)
slot update_slots: id  0 | task 2207 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 2207 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.501715
slot update_slots: id  0 | task 2207 | n_past = 2048, memory_seq_rm [2048, end)
slot update_slots: id  0 | task 2207 | prompt processing progress, n_past = 4082, n_tokens = 2034, progress = 1.000000
slot update_slots: id  0 | task 2207 | prompt done, n_past = 4082, n_tokens = 2034
slot update_slots: id  0 | task 2207 | saved context checkpoint 1 of 32 (pos_min = 4081, pos_max = 4081, size = 730.548 MiB)
srv  cancel_tasks: cancel task, id_task = 2207
srv  log_server_r: request: POST /completion 127.0.0.1 200
slot      release: id  0 | task 2207 | stop processing: n_past = 4673, truncated = 0
srv  update_slots: all slots are idle
slot get_availabl: id  0 | task 2207 | selected slot by lcs similarity, lcs_len = 2909, similarity = 0.623 (> 0.100 thold)
slot launch_slot_: id  0 | task 2802 | processing task
slot update_slots: id  0 | task 2802 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 4683
slot update_slots: id  0 | task 2802 | n_past = 2909, cache_tokens.size() = 4673, seq_id = 0, pos_min = 4672, n_swa = 1
slot update_slots: id  0 | task 2802 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 2802 | erased invalidated context checkpoint (pos_min = 4081, pos_max = 4081, n_swa = 1, size = 730.548 MiB)
slot update_slots: id  0 | task 2802 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 2802 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.437326
slot update_slots: id  0 | task 2802 | n_past = 2048, memory_seq_rm [2048, end)
slot update_slots: id  0 | task 2802 | prompt processing progress, n_past = 4096, n_tokens = 2048, progress = 0.874653
slot update_slots: id  0 | task 2802 | n_past = 4096, memory_seq_rm [4096, end)
slot update_slots: id  0 | task 2802 | prompt processing progress, n_past = 4683, n_tokens = 587, progress = 1.000000
slot update_slots: id  0 | task 2802 | prompt done, n_past = 4683, n_tokens = 587
slot update_slots: id  0 | task 2802 | saved context checkpoint 1 of 32 (pos_min = 4682, pos_max = 4682, size = 730.548 MiB)
slot      release: id  0 | task 2802 | stop processing: n_past = 4891, truncated = 0
slot print_timing: id  0 | task 2802 |
prompt eval time =    3405.50 ms /  4683 tokens (    0.73 ms per token,  1375.13 tokens per second)
       eval time =    6189.38 ms /   209 tokens (   29.61 ms per token,    33.77 tokens per second)
      total time =    9594.87 ms /  4892 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /completion 127.0.0.1 200

[aoleg]

It is still forcing full prompt reprocessing every time:

slot update_slots: id 0 | task 2207 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see #13194 (comment))

Here's another log, this time with the vulkan build:

llama-server     --model "s:\lmstudio\models\bartowski\nvidia_Nemotron-H-47B-Reasoning-128K-GGUF\nvidia_Nemotron-H-47B-Reasoning-128K-Q4_K_M.gguf"     --alias "nvidia/Nemotron-H-47B-Reasoning-128K-Q4_K_M"     --ctx-size 32768     --ctx-checkpoints 32     -ngl 999     -fa on     -ctk f16 -ctv f16     --host 127.0.0.1     --port 8081     --no-mmap     --jinja     --threads 12
load_backend: loaded RPC backend from S:\llm\llama.vulkan\ggml-rpc.dll
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (Intel Corporation) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from S:\llm\llama.vulkan\ggml-vulkan.dll
load_backend: loaded CPU backend from S:\llm\llama.vulkan\ggml-cpu-alderlake.dll
build: 6687 (e29acf74) with clang version 19.1.5 for x86_64-pc-windows-msvc
system info: n_threads = 12, n_threads_batch = 12, total_threads = 24

system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 8081, http threads: 23
main: loading model
srv    load_model: loading model 's:\lmstudio\models\bartowski\nvidia_Nemotron-H-47B-Reasoning-128K-GGUF\nvidia_Nemotron-H-47B-Reasoning-128K-Q4_K_M.gguf'
llama_model_load_from_file_impl: using device Vulkan0 (NVIDIA GeForce RTX 5090) (0000:01:00.0) - 31333 MiB free
llama_model_loader: loaded meta data with 48 key-value pairs and 577 tensors from s:\lmstudio\models\bartowski\nvidia_Nemotron-H-47B-Reasoning-128K-GGUF\nvidia_Nemotron-H-47B-Reasoning-128K-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = nemotron_h
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Nemotron H 47B Reasoning 128K
llama_model_loader: - kv   3:                           general.finetune str              = Reasoning-128k
llama_model_loader: - kv   4:                           general.basename str              = Nemotron-H
llama_model_loader: - kv   5:                         general.size_label str              = 47B
llama_model_loader: - kv   6:                            general.license str              = other
llama_model_loader: - kv   7:                       general.license.name str              = nvidia-internal-scientific-research-a...
llama_model_loader: - kv   8:                       general.license.link str              = https://www.nvidia.com/en-us/agreemen...
llama_model_loader: - kv   9:                               general.tags arr[str,3]       = ["nvidia", "pytorch", "text-generation"]
llama_model_loader: - kv  10:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv  11:                     nemotron_h.block_count u32              = 98
llama_model_loader: - kv  12:                  nemotron_h.context_length u32              = 1048576
llama_model_loader: - kv  13:                nemotron_h.embedding_length u32              = 8192
llama_model_loader: - kv  14:             nemotron_h.feed_forward_length arr[i32,98]      = [0, 30720, 0, 30720, 0, 30720, 0, 307...
llama_model_loader: - kv  15:            nemotron_h.attention.head_count u32              = 64
llama_model_loader: - kv  16:         nemotron_h.attention.head_count_kv arr[i32,98]      = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
llama_model_loader: - kv  17: nemotron_h.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  18:    nemotron_h.attention.layer_norm_epsilon f32              = 0.000010
llama_model_loader: - kv  19:                      nemotron_h.vocab_size u32              = 131072
llama_model_loader: - kv  20:            nemotron_h.rope.dimension_count u32              = 128
llama_model_loader: - kv  21:                 nemotron_h.ssm.conv_kernel u32              = 4
llama_model_loader: - kv  22:                  nemotron_h.ssm.state_size u32              = 256
llama_model_loader: - kv  23:                 nemotron_h.ssm.group_count u32              = 8
llama_model_loader: - kv  24:                  nemotron_h.ssm.inner_size u32              = 16384
llama_model_loader: - kv  25:              nemotron_h.ssm.time_step_rank u32              = 256
llama_model_loader: - kv  26:          nemotron_h.rope.scaling.finetuned bool             = false
llama_model_loader: - kv  27:            nemotron_h.attention.key_length u32              = 128
llama_model_loader: - kv  28:          nemotron_h.attention.value_length u32              = 128
llama_model_loader: - kv  29:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  30:                         tokenizer.ggml.pre str              = tekken
llama_model_loader: - kv  31:                      tokenizer.ggml.tokens arr[str,131072]  = ["<unk>", "<s>", "</s>", "[INST]", "[...
llama_model_loader: - kv  32:                  tokenizer.ggml.token_type arr[i32,131072]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  33:                      tokenizer.ggml.merges arr[str,269443]  = ["─а ─а", "─а t", "e r", "i n", "─а ─...
llama_model_loader: - kv  34:                tokenizer.ggml.eos_token_id u32              = 11
llama_model_loader: - kv  35:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  36:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  37:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  38:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  39:               tokenizer.ggml.add_sep_token bool             = false
llama_model_loader: - kv  40:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  41:                    tokenizer.chat_template str              = {{ '<SPECIAL_10>System\n' }}{%- if mes...
llama_model_loader: - kv  42:               general.quantization_version u32              = 2
llama_model_loader: - kv  43:                          general.file_type u32              = 15
llama_model_loader: - kv  44:                      quantize.imatrix.file str              = /models_out/Nemotron-H-47B-Reasoning-...
llama_model_loader: - kv  45:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav5.txt
llama_model_loader: - kv  46:             quantize.imatrix.entries_count u32              = 206
llama_model_loader: - kv  47:              quantize.imatrix.chunks_count u32              = 822
llama_model_loader: - type  f32:  369 tensors
llama_model_loader: - type q4_K:  181 tensors
llama_model_loader: - type q6_K:   27 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 26.24 GiB (4.82 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 11 ('<SPECIAL_11>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8499 MB
print_info: arch             = nemotron_h
print_info: vocab_only       = 0
print_info: n_ctx_train      = 1048576
print_info: n_embd           = 8192
print_info: n_layer          = 98
print_info: n_head           = 64
print_info: n_head_kv        = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_embd_k_gqa     = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_embd_v_gqa     = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = [0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 30720, 30720, 0, 0, 30720, 30720, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720]
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = -1
print_info: rope scaling     = linear
print_info: freq_base_train  = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 1048576
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 4
print_info: ssm_d_inner      = 16384
print_info: ssm_d_state      = 256
print_info: ssm_dt_rank      = 256
print_info: ssm_n_group      = 8
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = ?B
print_info: model params     = 46.79 B
print_info: general.name     = Nemotron H 47B Reasoning 128K
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 11 '<SPECIAL_11>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 0 '<unk>'
print_info: LF token         = 1010 '─К'
print_info: EOG token        = 11 '<SPECIAL_11>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 98 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 99/99 layers to GPU
load_tensors:      Vulkan0 model buffer size = 26284.26 MiB
load_tensors:  Vulkan_Host model buffer size =   576.00 MiB
load_tensors:          CPU model buffer size =    14.11 MiB
.................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 32768
llama_context: n_ctx_per_seq = 32768
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = false
llama_context: freq_base     = 10000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (32768) < n_ctx_train (1048576) -- the full capacity of the model will not be utilized
llama_context: Vulkan_Host  output buffer size =     0.50 MiB
llama_kv_cache:    Vulkan0 KV buffer size =   640.00 MiB
llama_kv_cache: size =  640.00 MiB ( 32768 cells,   5 layers,  1/1 seqs), K (f16):  320.00 MiB, V (f16):  320.00 MiB
llama_memory_recurrent:    Vulkan0 RS buffer size =   730.55 MiB
llama_memory_recurrent: size =  730.55 MiB (     1 cells,  98 layers,  1 seqs), R (f32):   10.55 MiB, S (f32):  720.00 MiB
llama_context:    Vulkan0 compute buffer size =   304.23 MiB
llama_context: Vulkan_Host compute buffer size =   248.36 MiB
llama_context: graph nodes  = 2750
llama_context: graph splits = 182
common_init_from_params: added <SPECIAL_11> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 32768
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 32768
srv          init: Enable thinking? 0
main: model loaded
main: chat template, chat_template: {{ '<SPECIAL_10>System
' }}{%- if messages and messages[0]['role'] == 'system' -%}{{ messages[0]['content'].strip() }}{%- endif -%}{% for message in (messages[1:] if messages[0]['role'] == 'system' else messages) %}{%- if message['role'] == 'user' -%}{{ '
<SPECIAL_11>User
' + message['content'].strip() + '
<SPECIAL_11>Assistant
' }}{%- if loop.last -%}{%- if messages[0]['role'] == 'system' -%}{%- if "{'reasoning': True}" in messages[0]['content'] -%}{{ '<think>
' }}{%- elif "{'reasoning': False}" in messages[0]['content'] -%}{{ '<think></think>' }}{%- endif -%}{%- endif -%}{%- endif -%}{%- elif message['role'] == 'assistant' -%}{{ message['content'].strip() }}{%- endif -%}{%- endfor -%}, example_format: '<SPECIAL_10>System
You are a helpful assistant
<SPECIAL_11>User
Hello
<SPECIAL_11>Assistant
Hi there
<SPECIAL_11>User
How are you?
<SPECIAL_11>Assistant
'
main: server is listening on http://127.0.0.1:8081 - starting the main loop
srv  update_slots: all slots are idle
srv  log_server_r: request: GET /v1/models 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 1894
slot update_slots: id  0 | task 0 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 1894, n_tokens = 1894, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 1894, n_tokens = 1894
slot update_slots: id  0 | task 0 | saved context checkpoint 1 of 32 (pos_min = 1893, pos_max = 1893, size = 730.548 MiB)
slot      release: id  0 | task 0 | stop processing: n_past = 2004, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =   44076.23 ms /  1894 tokens (   23.27 ms per token,    42.97 tokens per second)
       eval time =   23863.95 ms /   111 tokens (  214.99 ms per token,     4.65 tokens per second)
      total time =   67940.18 ms /  2005 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /completion 127.0.0.1 200
slot get_availabl: id  0 | task 0 | selected slot by lcs similarity, lcs_len = 1893, similarity = 0.945 (> 0.100 thold)
slot launch_slot_: id  0 | task 112 | processing task
slot update_slots: id  0 | task 112 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 2018
slot update_slots: id  0 | task 112 | n_past = 1893, cache_tokens.size() = 2004, seq_id = 0, pos_min = 2003, n_swa = 1
slot update_slots: id  0 | task 112 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 112 | erased invalidated context checkpoint (pos_min = 1893, pos_max = 1893, n_swa = 1, size = 730.548 MiB)
slot update_slots: id  0 | task 112 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 112 | prompt processing progress, n_past = 2018, n_tokens = 2018, progress = 1.000000
slot update_slots: id  0 | task 112 | prompt done, n_past = 2018, n_tokens = 2018
slot update_slots: id  0 | task 112 | saved context checkpoint 1 of 32 (pos_min = 2017, pos_max = 2017, size = 730.548 MiB)
slot      release: id  0 | task 112 | stop processing: n_past = 2129, truncated = 0
slot print_timing: id  0 | task 112 |
prompt eval time =   46067.51 ms /  2018 tokens (   22.83 ms per token,    43.81 tokens per second)
       eval time =   22507.58 ms /   112 tokens (  200.96 ms per token,     4.98 tokens per second)
      total time =   68575.09 ms /  2130 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /completion 127.0.0.1 200

[ddh0]

The issue is due to the chat template of that model as I mentioned above. To verify this you can start it without --jinja and you should observe that it works correctly (except for broken output due to the wrong chat template). I'm going to look at the chat template now to see if I can simplify it to fix this issue.

[aoleg]

Started without --jinja, still the same result (I am using SillyTavern with its own chat template, so this should not affect it anyway):

llama-server     --model "s:\lmstudio\models\bartowski\nvidia_Nemotron-H-47B-Reasoning-128K-GGUF\nvidia_Nemotron-H-47B-Reasoning-128K-Q4_K_M.gguf"     --alias "nvidia/Nemotron-H-47B-Reasoning-128K-Q4_K_M"     --ctx-size 32768     --ctx-checkpoints 32     -ngl 999     -fa on     -ctk f16 -ctv f16     --host 127.0.0.1     --port 8081     --no-mmap     --threads 12
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from S:\llm\llama\ggml-cuda.dll
load_backend: loaded RPC backend from S:\llm\llama\ggml-rpc.dll
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 5090 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = Intel(R) UHD Graphics 770 (Intel Corporation) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
load_backend: loaded Vulkan backend from S:\llm\llama\ggml-vulkan.dll
load_backend: loaded CPU backend from S:\llm\llama\ggml-cpu-alderlake.dll
build: 6898 (d1d250eb) with clang version 19.1.5 for x86_64-pc-windows-msvc
system info: n_threads = 12, n_threads_batch = 12, total_threads = 24

system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CUDA : ARCHS = 500,610,700,750,800,860,890,1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 8081, http threads: 23
main: loading model
srv    load_model: loading model 's:\lmstudio\models\bartowski\nvidia_Nemotron-H-47B-Reasoning-128K-GGUF\nvidia_Nemotron-H-47B-Reasoning-128K-Q4_K_M.gguf'
llama_model_load_from_file_impl: skipping device Vulkan0 (NVIDIA GeForce RTX 5090) with id 0000:01:00.0 - already using device CUDA0 (NVIDIA GeForce RTX 5090) with the same id
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 5090) (0000:01:00.0) - 28997 MiB free
llama_model_loader: max stdio successfully set to 2048
llama_model_loader: loaded meta data with 48 key-value pairs and 577 tensors from s:\lmstudio\models\bartowski\nvidia_Nemotron-H-47B-Reasoning-128K-GGUF\nvidia_Nemotron-H-47B-Reasoning-128K-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = nemotron_h
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Nemotron H 47B Reasoning 128K
llama_model_loader: - kv   3:                           general.finetune str              = Reasoning-128k
llama_model_loader: - kv   4:                           general.basename str              = Nemotron-H
llama_model_loader: - kv   5:                         general.size_label str              = 47B
llama_model_loader: - kv   6:                            general.license str              = other
llama_model_loader: - kv   7:                       general.license.name str              = nvidia-internal-scientific-research-a...
llama_model_loader: - kv   8:                       general.license.link str              = https://www.nvidia.com/en-us/agreemen...
llama_model_loader: - kv   9:                               general.tags arr[str,3]       = ["nvidia", "pytorch", "text-generation"]
llama_model_loader: - kv  10:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv  11:                     nemotron_h.block_count u32              = 98
llama_model_loader: - kv  12:                  nemotron_h.context_length u32              = 1048576
llama_model_loader: - kv  13:                nemotron_h.embedding_length u32              = 8192
llama_model_loader: - kv  14:             nemotron_h.feed_forward_length arr[i32,98]      = [0, 30720, 0, 30720, 0, 30720, 0, 307...
llama_model_loader: - kv  15:            nemotron_h.attention.head_count u32              = 64
llama_model_loader: - kv  16:         nemotron_h.attention.head_count_kv arr[i32,98]      = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
llama_model_loader: - kv  17: nemotron_h.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  18:    nemotron_h.attention.layer_norm_epsilon f32              = 0.000010
llama_model_loader: - kv  19:                      nemotron_h.vocab_size u32              = 131072
llama_model_loader: - kv  20:            nemotron_h.rope.dimension_count u32              = 128
llama_model_loader: - kv  21:                 nemotron_h.ssm.conv_kernel u32              = 4
llama_model_loader: - kv  22:                  nemotron_h.ssm.state_size u32              = 256
llama_model_loader: - kv  23:                 nemotron_h.ssm.group_count u32              = 8
llama_model_loader: - kv  24:                  nemotron_h.ssm.inner_size u32              = 16384
llama_model_loader: - kv  25:              nemotron_h.ssm.time_step_rank u32              = 256
llama_model_loader: - kv  26:          nemotron_h.rope.scaling.finetuned bool             = false
llama_model_loader: - kv  27:            nemotron_h.attention.key_length u32              = 128
llama_model_loader: - kv  28:          nemotron_h.attention.value_length u32              = 128
llama_model_loader: - kv  29:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  30:                         tokenizer.ggml.pre str              = tekken
llama_model_loader: - kv  31:                      tokenizer.ggml.tokens arr[str,131072]  = ["<unk>", "<s>", "</s>", "[INST]", "[...
llama_model_loader: - kv  32:                  tokenizer.ggml.token_type arr[i32,131072]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  33:                      tokenizer.ggml.merges arr[str,269443]  = ["─а ─а", "─а t", "e r", "i n", "─а ─...
llama_model_loader: - kv  34:                tokenizer.ggml.eos_token_id u32              = 11
llama_model_loader: - kv  35:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  36:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  37:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  38:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  39:               tokenizer.ggml.add_sep_token bool             = false
llama_model_loader: - kv  40:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  41:                    tokenizer.chat_template str              = {{ '<SPECIAL_10>System\n' }}{%- if mes...
llama_model_loader: - kv  42:               general.quantization_version u32              = 2
llama_model_loader: - kv  43:                          general.file_type u32              = 15
llama_model_loader: - kv  44:                      quantize.imatrix.file str              = /models_out/Nemotron-H-47B-Reasoning-...
llama_model_loader: - kv  45:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav5.txt
llama_model_loader: - kv  46:             quantize.imatrix.entries_count u32              = 206
llama_model_loader: - kv  47:              quantize.imatrix.chunks_count u32              = 822
llama_model_loader: - type  f32:  369 tensors
llama_model_loader: - type q4_K:  181 tensors
llama_model_loader: - type q6_K:   27 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 26.24 GiB (4.82 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 11 ('<SPECIAL_11>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8499 MB
print_info: arch             = nemotron_h
print_info: vocab_only       = 0
print_info: n_ctx_train      = 1048576
print_info: n_embd           = 8192
print_info: n_layer          = 98
print_info: n_head           = 64
print_info: n_head_kv        = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_embd_k_gqa     = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: n_embd_v_gqa     = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = [0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 30720, 30720, 0, 0, 30720, 30720, 30720, 0, 30720, 0, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720, 0, 30720]
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = -1
print_info: rope scaling     = linear
print_info: freq_base_train  = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 1048576
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 4
print_info: ssm_d_inner      = 16384
print_info: ssm_d_state      = 256
print_info: ssm_dt_rank      = 256
print_info: ssm_n_group      = 8
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = ?B
print_info: model params     = 46.79 B
print_info: general.name     = Nemotron H 47B Reasoning 128K
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 11 '<SPECIAL_11>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 0 '<unk>'
print_info: LF token         = 1010 '─К'
print_info: EOG token        = 11 '<SPECIAL_11>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 98 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 99/99 layers to GPU
load_tensors:        CUDA0 model buffer size = 26298.37 MiB
load_tensors:          CPU model buffer size =   576.00 MiB
................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 32768
llama_context: n_ctx_per_seq = 32768
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = false
llama_context: freq_base     = 10000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (32768) < n_ctx_train (1048576) -- the full capacity of the model will not be utilized
llama_context:  CUDA_Host  output buffer size =     0.50 MiB
llama_kv_cache:      CUDA0 KV buffer size =   640.00 MiB
llama_kv_cache: size =  640.00 MiB ( 32768 cells,   5 layers,  1/1 seqs), K (f16):  320.00 MiB, V (f16):  320.00 MiB
llama_memory_recurrent:      CUDA0 RS buffer size =   730.55 MiB
llama_memory_recurrent: size =  730.55 MiB (     1 cells,  98 layers,  1 seqs), R (f32):   10.55 MiB, S (f32):  720.00 MiB
llama_context:      CUDA0 compute buffer size =   304.23 MiB
llama_context:  CUDA_Host compute buffer size =    80.01 MiB
llama_context: graph nodes  = 2750
llama_context: graph splits = 2
common_init_from_params: added <SPECIAL_11> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 32768
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv    load_model: load_model: Chat template parsing error: this custom template is not supported, try using --jinja
srv    load_model: load_model: The chat template that comes with this model is not yet supported, falling back to chatml. This may cause the model to output suboptimal responses
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 32768
srv          init: Enable thinking? 0
main: model loaded
main: chat template, chat_template: {%- for message in messages -%}
  {{- '<|im_start|>' + message.role + '
' + message.content + '<|im_end|>
' -}}
{%- endfor -%}
{%- if add_generation_prompt -%}
  {{- '<|im_start|>assistant
' -}}
{%- endif -%}, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://127.0.0.1:8081 - starting the main loop
srv  update_slots: all slots are idle
slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 1894
slot update_slots: id  0 | task 0 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 1894, n_tokens = 1894, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 1894, n_tokens = 1894
slot update_slots: id  0 | task 0 | saved context checkpoint 1 of 32 (pos_min = 1893, pos_max = 1893, size = 730.548 MiB)
slot      release: id  0 | task 0 | stop processing: n_past = 1996, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =    1691.77 ms /  1894 tokens (    0.89 ms per token,  1119.54 tokens per second)
       eval time =    3004.32 ms /   103 tokens (   29.17 ms per token,    34.28 tokens per second)
      total time =    4696.09 ms /  1997 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /completion 127.0.0.1 200
slot get_availabl: id  0 | task 0 | selected slot by lcs similarity, lcs_len = 1893, similarity = 0.948 (> 0.100 thold)
slot launch_slot_: id  0 | task 104 | processing task
slot update_slots: id  0 | task 104 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 2010
slot update_slots: id  0 | task 104 | n_past = 1893, cache_tokens.size() = 1996, seq_id = 0, pos_min = 1995, n_swa = 1
slot update_slots: id  0 | task 104 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 104 | erased invalidated context checkpoint (pos_min = 1893, pos_max = 1893, n_swa = 1, size = 730.548 MiB)
slot update_slots: id  0 | task 104 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 104 | prompt processing progress, n_past = 2010, n_tokens = 2010, progress = 1.000000
slot update_slots: id  0 | task 104 | prompt done, n_past = 2010, n_tokens = 2010
slot update_slots: id  0 | task 104 | saved context checkpoint 1 of 32 (pos_min = 2009, pos_max = 2009, size = 730.548 MiB)
slot      release: id  0 | task 104 | stop processing: n_past = 2095, truncated = 0
slot print_timing: id  0 | task 104 |
prompt eval time =    1521.03 ms /  2010 tokens (    0.76 ms per token,  1321.48 tokens per second)
       eval time =    2525.93 ms /    86 tokens (   29.37 ms per token,    34.05 tokens per second)
      total time =    4046.96 ms /  2096 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /completion 127.0.0.1 200
slot get_availabl: id  0 | task 104 | selected slot by lcs similarity, lcs_len = 1626, similarity = 0.776 (> 0.100 thold)
slot launch_slot_: id  0 | task 191 | processing task
slot update_slots: id  0 | task 191 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 2105
slot update_slots: id  0 | task 191 | n_past = 1626, cache_tokens.size() = 2095, seq_id = 0, pos_min = 2094, n_swa = 1
slot update_slots: id  0 | task 191 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 191 | erased invalidated context checkpoint (pos_min = 2009, pos_max = 2009, n_swa = 1, size = 730.548 MiB)
slot update_slots: id  0 | task 191 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 191 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.972922
slot update_slots: id  0 | task 191 | n_past = 2048, memory_seq_rm [2048, end)
slot update_slots: id  0 | task 191 | prompt processing progress, n_past = 2105, n_tokens = 57, progress = 1.000000
slot update_slots: id  0 | task 191 | prompt done, n_past = 2105, n_tokens = 57
slot update_slots: id  0 | task 191 | saved context checkpoint 1 of 32 (pos_min = 2104, pos_max = 2104, size = 730.548 MiB)
slot      release: id  0 | task 191 | stop processing: n_past = 2171, truncated = 0
slot print_timing: id  0 | task 191 |
prompt eval time =    1943.00 ms /  2105 tokens (    0.92 ms per token,  1083.38 tokens per second)
       eval time =    1958.49 ms /    67 tokens (   29.23 ms per token,    34.21 tokens per second)
      total time =    3901.49 ms /  2172 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /completion 127.0.0.1 200

Here's the SillyTavern instruct+context template:

{
    "instruct": {
        "input_sequence": "<SPECIAL_11>User\n",
        "output_sequence": "<SPECIAL_11>Assistant\n",
        "last_output_sequence": "",
        "system_sequence": "<SPECIAL_10>System\n",
        "stop_sequence": "",
        "wrap": true,
        "macro": true,
        "names_behavior": "force",
        "activation_regex": "",
        "first_output_sequence": "",
        "skip_examples": false,
        "output_suffix": "\n",
        "input_suffix": "\n",
        "system_suffix": "\n",
        "user_alignment_message": "",
        "system_same_as_user": false,
        "last_system_sequence": "",
        "first_input_sequence": "",
        "last_input_sequence": "",
        "sequences_as_stop_strings": true,
        "story_string_prefix": "",
        "story_string_suffix": "",
        "name": "Nemotron-H 47B"
    },
    "context": {
        "story_string": "<SPECIAL_10>System\n{'reasoning': False}\n{{#if anchorBefore}}{{anchorBefore}}\n{{/if}}{{#if system}}{{system}}\n{{/if}}{{#if wiBefore}}{{wiBefore}}\n{{/if}}{{#if description}}{{description}}\n{{/if}}{{#if personality}}{{char}}'s personality: {{personality}}\n{{/if}}{{#if scenario}}Scenario: {{scenario}}\n{{/if}}{{#if wiAfter}}{{wiAfter}}\n{{/if}}{{#if persona}}{{persona}}\n{{/if}}{{#if anchorAfter}}{{anchorAfter}}\n{{/if}}",
        "example_separator": "***",
        "chat_start": "***",
        "use_stop_strings": false,
        "names_as_stop_strings": true,
        "story_string_position": 0,
        "story_string_depth": 1,
        "story_string_role": 0,
        "always_force_name2": true,
        "trim_sentences": false,
        "single_line": false,
        "name": "Nemotron-H 47B"
    }
}

[ddh0]

Hmm, I'm not very familiar with SillyTavern - are you using the chat completions endpoint provided by llama-server?

[aoleg]

Same session but using llamacpp localhost WebUI (no --jinja) saves and reuses the context correctly. So this part works just fine; the issue manifests itself when I use a third-party front-end such as SillyTavern. I posted my instruct+context config earlier, don't see anything fancy there that could throw off the caching.

srv  update_slots: all slots are idle
srv  log_server_r: request: POST /completion 127.0.0.1 200
srv  log_server_r: request: GET / 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
srv  update_slots: all slots are idle
srv  log_server_r: request: GET /slots 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
srv  log_server_r: request: GET /props 127.0.0.1 200
srv  params_from_: Chat format: Content-only
slot get_availabl: id  0 | task 191 | selected slot by LRU, t_last = 86857173
slot launch_slot_: id  0 | task 261 | processing task
slot update_slots: id  0 | task 261 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 32
slot update_slots: id  0 | task 261 | n_past = 1, cache_tokens.size() = 2171, seq_id = 0, pos_min = 2170, n_swa = 1
slot update_slots: id  0 | task 261 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
slot update_slots: id  0 | task 261 | erased invalidated context checkpoint (pos_min = 2104, pos_max = 2104, n_swa = 1, size = 730.548 MiB)
slot update_slots: id  0 | task 261 | n_past = 0, memory_seq_rm [0, end)
slot update_slots: id  0 | task 261 | prompt processing progress, n_past = 32, n_tokens = 32, progress = 1.000000
slot update_slots: id  0 | task 261 | prompt done, n_past = 32, n_tokens = 32
slot update_slots: id  0 | task 261 | saved context checkpoint 1 of 32 (pos_min = 31, pos_max = 31, size = 730.548 MiB)
srv  log_server_r: request: GET /slots 127.0.0.1 200
srv  log_server_r: request: GET /slots 127.0.0.1 200
srv  cancel_tasks: cancel task, id_task = 261
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
slot      release: id  0 | task 261 | stop processing: n_past = 553, truncated = 0
srv  update_slots: all slots are idle
srv  params_from_: Chat format: Content-only
slot get_availabl: id  0 | task 261 | selected slot by lcs similarity, lcs_len = 552, similarity = 0.998 (> 0.100 thold)
slot launch_slot_: id  0 | task 787 | processing task
slot update_slots: id  0 | task 787 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 585
slot update_slots: id  0 | task 787 | n_past = 552, cache_tokens.size() = 553, seq_id = 0, pos_min = 552, n_swa = 1
slot update_slots: id  0 | task 787 | restored context checkpoint (pos_min = 31, pos_max = 31, size = 730.548 MiB)
slot update_slots: id  0 | task 787 | n_past = 32, memory_seq_rm [32, end)
slot update_slots: id  0 | task 787 | prompt processing progress, n_past = 585, n_tokens = 553, progress = 0.945299
slot update_slots: id  0 | task 787 | prompt done, n_past = 585, n_tokens = 553
slot update_slots: id  0 | task 787 | saved context checkpoint 2 of 32 (pos_min = 584, pos_max = 584, size = 730.548 MiB)
srv  cancel_tasks: cancel task, id_task = 787
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
slot      release: id  0 | task 787 | stop processing: n_past = 949, truncated = 0
srv  update_slots: all slots are idle
srv  params_from_: Chat format: Content-only
slot get_availabl: id  0 | task 787 | selected slot by lcs similarity, lcs_len = 948, similarity = 0.999 (> 0.100 thold)
slot launch_slot_: id  0 | task 1154 | processing task
slot update_slots: id  0 | task 1154 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 979
slot update_slots: id  0 | task 1154 | n_past = 948, cache_tokens.size() = 949, seq_id = 0, pos_min = 948, n_swa = 1
slot update_slots: id  0 | task 1154 | restored context checkpoint (pos_min = 584, pos_max = 584, size = 730.548 MiB)
slot update_slots: id  0 | task 1154 | n_past = 585, memory_seq_rm [585, end)
slot update_slots: id  0 | task 1154 | prompt processing progress, n_past = 979, n_tokens = 394, progress = 0.402451
slot update_slots: id  0 | task 1154 | prompt done, n_past = 979, n_tokens = 394
slot update_slots: id  0 | task 1154 | saved context checkpoint 3 of 32 (pos_min = 978, pos_max = 978, size = 730.548 MiB)
srv  cancel_tasks: cancel task, id_task = 1154
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
slot      release: id  0 | task 1154 | stop processing: n_past = 1253, truncated = 0
srv  update_slots: all slots are idle