Eval bug: issues with draft model and Cline+VSCode

[Nepherpitou]

Name and Version

 .\llama-cli.exe --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
version: 1 (46c69e0)
built with MSVC 19.29.30157.0 for

Operating systems

Windows

GGML backends

CUDA

Hardware

Ryzen 7900X, 128G DDR5, RTX 3090 + RTX 4090

Models

Main - qwen2.5-coder-32b-instruct-q6_k.gguf
Draft - qwen2.5-coder-1.5b-instruct-q6_k.gguf

Problem description & steps to reproduce

When I run

.\llama-server.exe -m qwen2.5-coder-32b-instruct.gguf -ngl 99 --port 5000 -fa -c 32000 -md qwen2.5-coder-1.5b-instruct-q6_k.gguf -ngld 99 -ctk q8_0 -ctv q8_0 -a qwen2.5-coder-32b --log-timestamps -devd 'CUDA0' -ts '3,10' --draft 16 --draft-p-min 0.4 -nocb

and use Cline in VS Code, then it sends multiple requests at once (I suppose, not sure) and server exit with error:

slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 32000, n_keep = 0, n_prompt_tokens = 7056
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.290249
llama_decode_internal: invalid token[0] = -657584439
llama_decode: failed to decode, ret = -1
llama_get_logits_ith: invalid logits id 0, reason: batch.logits[0] != true

But sometimes it doesn't fail, but output is corrupted:

theThe task is to add mocked API requests to the application....

Here is doubled theThe at the begin of response.

First Bad Commit

No response

Relevant log output

.\llama-server.exe -m ..\aphrodite\qwen2.5-coder-32b-instruct.gguf -ngl 99 --port 5000 -fa -c 32000 -md ..\aphrodite\qwen2.5-coder-1.5b-instruct-q6_k.gguf -ngld 99 -ctk q8_0 -ctv q8_0 -a qwen2.5-coder-32b --log-timestamps -devd 'CUDA0' -ts '3,10' --draft 16 --draft-p-min 0.4
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
build: 1 (46c69e0) with MSVC 19.29.30157.0 for
system info: n_threads = 12, n_threads_batch = 12, total_threads = 24

system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CUDA : ARCHS = 520,610,700,750 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 |

main: HTTP server is listening, hostname: 127.0.0.1, port: 5000, http threads: 23
main: loading model
srv    load_model: loading model '..\aphrodite\qwen2.5-coder-32b-instruct.gguf'
llama_load_model_from_file: using device CUDA0 (NVIDIA GeForce RTX 4090) - 22994 MiB free
llama_load_model_from_file: using device CUDA1 (NVIDIA GeForce RTX 3090) - 23306 MiB free
llama_model_loader: loaded meta data with 38 key-value pairs and 771 tensors from ..\aphrodite\qwen2.5-coder-32b-instruct.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 Coder 32B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Qwen2.5-Coder
llama_model_loader: - kv   5:                         general.size_label str              = 32B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                       general.license.link str              = https://huggingface.co/Qwen/Qwen2.5-C...
llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
llama_model_loader: - kv   9:                  general.base_model.0.name str              = Qwen2.5 Coder 32B
llama_model_loader: - kv  10:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  11:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen2.5-C...
llama_model_loader: - kv  12:                               general.tags arr[str,6]       = ["code", "codeqwen", "chat", "qwen", ...
llama_model_loader: - kv  13:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv  14:                          qwen2.block_count u32              = 64
llama_model_loader: - kv  15:                       qwen2.context_length u32              = 32768
llama_model_loader: - kv  16:                     qwen2.embedding_length u32              = 5120
llama_model_loader: - kv  17:                  qwen2.feed_forward_length u32              = 27648
llama_model_loader: - kv  18:                 qwen2.attention.head_count u32              = 40
llama_model_loader: - kv  19:              qwen2.attention.head_count_kv u32              = 8
llama_model_loader: - kv  20:                       qwen2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  21:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  22:                          general.file_type u32              = 18
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,151387]  = ["─а ─а", "─а─а ─а─а", "i n", "─а t",...
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  30:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  33:               general.quantization_version u32              = 2
llama_model_loader: - kv  34:                      quantize.imatrix.file str              = /models_out/Qwen2.5-Coder-32B-Instruc...
llama_model_loader: - kv  35:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt
llama_model_loader: - kv  36:             quantize.imatrix.entries_count i32              = 448
llama_model_loader: - kv  37:              quantize.imatrix.chunks_count i32              = 128
llama_model_loader: - type  f32:  321 tensors
llama_model_loader: - type q6_K:  450 tensors
llm_load_vocab: special tokens cache size = 22
llm_load_vocab: token to piece cache size = 0.9310 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = qwen2
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 152064
llm_load_print_meta: n_merges         = 151387
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 5120
llm_load_print_meta: n_layer          = 64
llm_load_print_meta: n_head           = 40
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 5
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 27648
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = ?B
llm_load_print_meta: model ftype      = Q6_K
llm_load_print_meta: model params     = 32.76 B
llm_load_print_meta: model size       = 25.03 GiB (6.56 BPW)
llm_load_print_meta: general.name     = Qwen2.5 Coder 32B Instruct
llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOS token        = 151645 '<|im_end|>'
llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
llm_load_print_meta: PAD token        = 151643 '<|endoftext|>'
llm_load_print_meta: LF token         = 148848 '├Д─м'
llm_load_print_meta: FIM PRE token    = 151659 '<|fim_prefix|>'
llm_load_print_meta: FIM SUF token    = 151661 '<|fim_suffix|>'
llm_load_print_meta: FIM MID token    = 151660 '<|fim_middle|>'
llm_load_print_meta: FIM PAD token    = 151662 '<|fim_pad|>'
llm_load_print_meta: FIM REP token    = 151663 '<|repo_name|>'
llm_load_print_meta: FIM SEP token    = 151664 '<|file_sep|>'
llm_load_print_meta: EOG token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOG token        = 151645 '<|im_end|>'
llm_load_print_meta: EOG token        = 151662 '<|fim_pad|>'
llm_load_print_meta: EOG token        = 151663 '<|repo_name|>'
llm_load_print_meta: EOG token        = 151664 '<|file_sep|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: offloading 64 repeating layers to GPU
llm_load_tensors: offloading output layer to GPU
llm_load_tensors: offloaded 65/65 layers to GPU
llm_load_tensors:        CUDA0 model buffer size =  5722.68 MiB
llm_load_tensors:        CUDA1 model buffer size = 19303.18 MiB
llm_load_tensors:   CPU_Mapped model buffer size =   609.08 MiB
.................................................................................................
llama_new_context_with_model: n_seq_max     = 1
llama_new_context_with_model: n_ctx         = 32000
llama_new_context_with_model: n_ctx_per_seq = 32000
llama_new_context_with_model: n_batch       = 2048
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 1
llama_new_context_with_model: freq_base     = 1000000.0
llama_new_context_with_model: freq_scale    = 1
llama_new_context_with_model: n_ctx_per_seq (32000) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
llama_kv_cache_init:      CUDA0 KV buffer size =   996.09 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =  3253.91 MiB
llama_new_context_with_model: KV self size  = 4250.00 MiB, K (q8_0): 2125.00 MiB, V (q8_0): 2125.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.58 MiB
llama_new_context_with_model: pipeline parallelism enabled (n_copies=4)
llama_new_context_with_model:      CUDA0 compute buffer size =   459.26 MiB
llama_new_context_with_model:      CUDA1 compute buffer size =   472.02 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =   260.02 MiB
llama_new_context_with_model: graph nodes  = 1991
llama_new_context_with_model: graph splits = 3
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv    load_model: loading draft model '..\aphrodite\qwen2.5-coder-1.5b-instruct-q6_k.gguf'
llama_load_model_from_file: using device CUDA0 (NVIDIA GeForce RTX 4090) - 15768 MiB free
llama_model_loader: loaded meta data with 26 key-value pairs and 339 tensors from ..\aphrodite\qwen2.5-coder-1.5b-instruct-q6_k.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 Coder 1.5B Instruct GGUF
llama_model_loader: - kv   3:                           general.finetune str              = Instruct-GGUF
llama_model_loader: - kv   4:                           general.basename str              = Qwen2.5-Coder
llama_model_loader: - kv   5:                         general.size_label str              = 1.5B
llama_model_loader: - kv   6:                          qwen2.block_count u32              = 28
llama_model_loader: - kv   7:                       qwen2.context_length u32              = 32768
llama_model_loader: - kv   8:                     qwen2.embedding_length u32              = 1536
llama_model_loader: - kv   9:                  qwen2.feed_forward_length u32              = 8960
llama_model_loader: - kv  10:                 qwen2.attention.head_count u32              = 12
llama_model_loader: - kv  11:              qwen2.attention.head_count_kv u32              = 2
llama_model_loader: - kv  12:                       qwen2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  13:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  14:                          general.file_type u32              = 18
llama_model_loader: - kv  15:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  16:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  17:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  18:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  19:                      tokenizer.ggml.merges arr[str,151387]  = ["─а ─а", "─а─а ─а─а", "i n", "─а t",...
llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  22:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  23:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  25:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  141 tensors
llama_model_loader: - type q6_K:  198 tensors
llm_load_vocab: special tokens cache size = 22
llm_load_vocab: token to piece cache size = 0.9310 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = qwen2
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 151936
llm_load_print_meta: n_merges         = 151387
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 1536
llm_load_print_meta: n_layer          = 28
llm_load_print_meta: n_head           = 12
llm_load_print_meta: n_head_kv        = 2
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 6
llm_load_print_meta: n_embd_k_gqa     = 256
llm_load_print_meta: n_embd_v_gqa     = 256
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 8960
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 1.5B
llm_load_print_meta: model ftype      = Q6_K
llm_load_print_meta: model params     = 1.78 B
llm_load_print_meta: model size       = 1.36 GiB (6.56 BPW)
llm_load_print_meta: general.name     = Qwen2.5 Coder 1.5B Instruct GGUF
llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOS token        = 151645 '<|im_end|>'
llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
llm_load_print_meta: PAD token        = 151643 '<|endoftext|>'
llm_load_print_meta: LF token         = 148848 '├Д─м'
llm_load_print_meta: FIM PRE token    = 151659 '<|fim_prefix|>'
llm_load_print_meta: FIM SUF token    = 151661 '<|fim_suffix|>'
llm_load_print_meta: FIM MID token    = 151660 '<|fim_middle|>'
llm_load_print_meta: FIM PAD token    = 151662 '<|fim_pad|>'
llm_load_print_meta: FIM REP token    = 151663 '<|repo_name|>'
llm_load_print_meta: FIM SEP token    = 151664 '<|file_sep|>'
llm_load_print_meta: EOG token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOG token        = 151645 '<|im_end|>'
llm_load_print_meta: EOG token        = 151662 '<|fim_pad|>'
llm_load_print_meta: EOG token        = 151663 '<|repo_name|>'
llm_load_print_meta: EOG token        = 151664 '<|file_sep|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: offloading 28 repeating layers to GPU
llm_load_tensors: offloading output layer to GPU
llm_load_tensors: offloaded 29/29 layers to GPU
llm_load_tensors:        CUDA0 model buffer size =  1208.11 MiB
llm_load_tensors:   CPU_Mapped model buffer size =   182.57 MiB
...........................................................................
llama_new_context_with_model: n_seq_max     = 1
llama_new_context_with_model: n_ctx         = 32768
llama_new_context_with_model: n_ctx_per_seq = 32768
llama_new_context_with_model: n_batch       = 2048
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 1
llama_new_context_with_model: freq_base     = 1000000.0
llama_new_context_with_model: freq_scale    = 1
llama_kv_cache_init:      CUDA0 KV buffer size =   476.00 MiB
llama_new_context_with_model: KV self size  =  476.00 MiB, K (q8_0):  238.00 MiB, V (q8_0):  238.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.58 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   299.75 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    67.01 MiB
llama_new_context_with_model: graph nodes  = 875
llama_new_context_with_model: graph splits = 2
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv          init: initializing slots, n_slots = 1
llama_new_context_with_model: n_seq_max     = 1
llama_new_context_with_model: n_ctx         = 32000
llama_new_context_with_model: n_ctx_per_seq = 32000
llama_new_context_with_model: n_batch       = 32000
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 1
llama_new_context_with_model: freq_base     = 1000000.0
llama_new_context_with_model: freq_scale    = 1
llama_new_context_with_model: n_ctx_per_seq (32000) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
llama_kv_cache_init:      CUDA0 KV buffer size =   464.84 MiB
llama_new_context_with_model: KV self size  =  464.84 MiB, K (q8_0):  232.42 MiB, V (q8_0):  232.42 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.58 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   299.75 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    65.51 MiB
llama_new_context_with_model: graph nodes  = 875
llama_new_context_with_model: graph splits = 2
slot         init: id  0 | task -1 | new slot n_ctx_slot = 32000
main: model loaded
main: chat template, built_in: 1, chat_example: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://127.0.0.1:5000 - starting the main loop
srv  update_slots: all slots are idle
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 32000, n_keep = 0, n_prompt_tokens = 7056
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 2048, n_tokens = 2048, progress = 0.290249
llama_decode_internal: invalid token[0] = -657584439
llama_decode: failed to decode, ret = -1
llama_get_logits_ith: invalid logits id 0, reason: batch.logits[0] != true

[Nepherpitou]

Everyting works fine without -md draft model, -nocb doesn't help as well.

[Nepherpitou]

There is really strange behavior. It fails only if it's first request to fresh started server. And works almost fine if server already processed at least one request, but sometimes artifacts with first word in output are occurred.

[josharian]

I just hit an error that looks a lot like this, at commit 0cc6375, using draft model (gemma2-2b at Q8 for gemma2-27b, across 4 gpus). There is no concurrency involved, and it reproduces well, albeit slowly and in a way I can't share directly.

Tail of log:

slot launch_slot_: id  0 | task 1195 | processing task
slot update_slots: id  0 | task 1195 | new prompt, n_ctx_slot = 8192, n_keep = 0, n_prompt_tokens = 15769
slot update_slots: id  0 | task 1195 | input truncated, n_ctx = 8192, n_keep = 0, n_left = 8192, n_prompt_tokens = 7577
slot update_slots: id  0 | task 1195 | need to evaluate at least 1 token to generate logits, n_past = 7577, n_prompt_tokens = 7577
slot update_slots: id  0 | task 1195 | kv cache rm [7576, end)
slot update_slots: id  0 | task 1195 | prompt processing progress, n_past = 7577, n_tokens = 1, progress = 0.000132
slot update_slots: id  0 | task 1195 | prompt done, n_past = 7577, n_tokens = 1
llama_decode: failed to decode, ret = 1
llama_get_logits_ith: invalid logits id 0, reason: batch.logits[0] != true
Segmentation fault (core dumped)

Happy to add instrumentation and debugging and re-run, if I can get guidance about what would be useful to know.

[josharian]

Bisect points to:

47f931c8f9a26c072d71224bc8013cc66ea9e445 is the first bad commit
commit 47f931c8f9a26c072d71224bc8013cc66ea9e445
Author: Georgi Gerganov <ggerganov@gmail.com>
Date:   Mon Nov 25 21:50:07 2024 +0200

    server : enable cache_prompt by default (#10501)
    
    ggml-ci

 examples/server/README.md  | 2 +-
 examples/server/server.cpp | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

and reverting it at HEAD also resolves the issue.

Let me know what other information would be useful to provide.

cc @ggerganov

[ggerganov]

Would be helpful if you provide repro steps.

[josharian]

Unfortunately, the reliable repro steps involve a lot of data I can't share. My initial attempts to minimize failed. My apologies. I will try again, but each repro is very slow, and the setup is complicated, so minimization is pretty non-trivial.

The overall repro approach is: start the server, make a lot of requests.

FWIW, I ran the git bisect like this:

make clean
GGML_CUDA=1 make -j24
./llama-server -m gemma2-27b-instruct.gguf -ngl 100 -c 0 -md gemma2-2b-instruct-Q8.gguf -ngld 100 -devd CUDA0,CUDA1,CUDA2,CUDA3

I'm using a 4x4090 setup.

[ggerganov]

The bisect information is irrelevant - this commit simply enables speculative decoding. Before that commit, there is no speculative decoding (unless the client sets cache_prompt: true explicitly, which most likely it does not).

[josharian]

I have found no version that reliably reproduces in less than a half hour. But inspecting the code, it looks like the fix may be as simple as:

diff --git a/examples/server/server.cpp b/examples/server/server.cpp
index 9c86407c..2fe33b18 100644
--- a/examples/server/server.cpp
+++ b/examples/server/server.cpp
@@ -2330,7 +2330,10 @@ struct server_context {
                     common_batch_add(slot.batch_spec, draft[i], slot.n_past + 1 + i, { slot.id }, true);
                 }
 
-                llama_decode(ctx, slot.batch_spec);
+                int ret = llama_decode(ctx, slot.batch_spec);
+                if (ret != 0) {
+                    continue;
+                }
 
                 // the accepted tokens from the speculation
                 const auto ids = common_sampler_sample_and_accept_n(slot.smpl, ctx, draft);

That at least appears to fix things for me. But I'm not sure what cleanup, if any, might be required before the continue.

[unclemusclez]

commit 9f912511bc9414fa7a3c521378b6388cd932b58d (HEAD, tag: b4202, origin/master, origin/HEAD, master)

I have this same issue. It seems to trigger toward the end of a response, breaking the response and crashing.
WebUI

I messed with a few different options trying to regenerate responses, but seemingly all you have to do is use a decode model.

Dec 02 21:50:47 kamala llama-server[24689]: [100B blob data]
Dec 02 21:50:47 kamala llama-server[24689]: [100B blob data]
Dec 02 21:50:47 kamala llama-server[24689]: [48B blob data]
Dec 02 21:50:47 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 21:50:47 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 21:50:47 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 21:50:47 kamala llama-server[24689]: slot launch_slot_: id  0 | task 5123 | processing task
Dec 02 21:50:47 kamala llama-server[24689]: slot update_slots: id  0 | task 5123 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 7377
Dec 02 21:50:47 kamala llama-server[24689]: slot update_slots: id  0 | task 5123 | input truncated, n_ctx = 4096, n_keep = 0, n_left = 4096, n_prompt_tokens = 3281
Dec 02 21:50:47 kamala llama-server[24689]: slot update_slots: id  0 | task 5123 | kv cache rm [0, end)
Dec 02 21:50:47 kamala llama-server[24689]: slot update_slots: id  0 | task 5123 | prompt processing progress, n_past = 3281, n_tokens = 3281, progress = 1.000000
Dec 02 21:50:47 kamala llama-server[24689]: slot update_slots: id  0 | task 5123 | prompt done, n_past = 3281, n_tokens = 3281
Dec 02 21:51:35 kamala llama-server[24689]: slot      release: id  0 | task 5123 | stop processing: n_past = 3343, truncated = 1
Dec 02 21:51:35 kamala llama-server[24689]: slot print_timing: id  0 | task 5123 |
Dec 02 21:51:35 kamala llama-server[24689]: [100B blob data]
Dec 02 21:51:35 kamala llama-server[24689]: [100B blob data]
Dec 02 21:51:35 kamala llama-server[24689]: [48B blob data]
Dec 02 21:51:35 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 21:51:35 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 21:51:35 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 21:51:36 kamala llama-server[24689]: slot launch_slot_: id  0 | task 5136 | processing task
Dec 02 21:51:36 kamala llama-server[24689]: slot update_slots: id  0 | task 5136 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 7437
Dec 02 21:51:36 kamala llama-server[24689]: slot update_slots: id  0 | task 5136 | input truncated, n_ctx = 4096, n_keep = 0, n_left = 4096, n_prompt_tokens = 3341
Dec 02 21:51:36 kamala llama-server[24689]: slot update_slots: id  0 | task 5136 | kv cache rm [0, end)
Dec 02 21:51:36 kamala llama-server[24689]: slot update_slots: id  0 | task 5136 | prompt processing progress, n_past = 3341, n_tokens = 3341, progress = 1.000000
Dec 02 21:51:36 kamala llama-server[24689]: slot update_slots: id  0 | task 5136 | prompt done, n_past = 3341, n_tokens = 3341
Dec 02 21:53:38 kamala llama-server[24689]: llama_decode: failed to decode, ret = 1

exec command:
/usr/local/bin/llama-server -m models/QwQ-32B-Preview-Q4_K_M.gguf -a qwq-32b-preview -md models/Qwen2.5-Coder-3B-Q4_K_L.gguf --port 11430 --host 0.0.0.0 -t 4 -fa -ngl 999 -ngld 999 -b 8092 --no-mmap --threads-http 4 --api-key XXX -to 3600 -sm none -sp --slot

[unclemusclez]

@josharian are you using the Unsloth versions of the GGUFs?

[josharian]

@unclemusclez nope. Downloaded from hf, converted with the python script. I'm pretty confident that this doesn't have to do with the precise model in use.

[unclemusclez]

@unclemusclez nope. Downloaded from hf, converted with the python script. I'm pretty confident that this doesn't have to do with the precise model in use.
I AM using Unsloth, so that's good, I would agree it's independent of model.

Are you also using embeddings? I use WebUI with the ChromaDB.
I run a separate instance of llama.cpp with the embedding model (BGE-m3) only.

If I use the instruct model with WebUI and the request is a large amount of tokens, it will potentially crash. So I would say, that any normal use case will trigger this. I've had it trigger the 3rd prompt and sometimes I've had it trigger the first prompt, so I think it is related to tokens.

I am on AMD/HIP ROCm 6.2.4. GFX906, which that eliminates the GPU issue.

Ubuntu 24.04

We are both using Flash Attention.

It will trigger mid response, but usually later in the response, or if it is giving back nonsense repetitively.

Other than "Model not Found" all I see in the logs server side is llama_decode: failed to decode, ret = 1 when it crashes.

some logs:

Dec 02 11:28:28 kamala systemd[1]: Started llama.service - Llama Service.
░░ Subject: A start job for unit llama.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit llama.service has finished successfully.
░░
░░ The job identifier is 1014499.
Dec 02 11:28:30 kamala llama-server[14451]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Dec 02 11:28:30 kamala llama-server[14451]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Dec 02 11:28:30 kamala llama-server[14451]: ggml_cuda_init: found 1 ROCm devices:
Dec 02 11:28:30 kamala llama-server[14451]:   Device 0: AMD Radeon Graphics, compute capability 9.0, VMM: no
Dec 02 11:28:30 kamala llama-server[14451]: build: 4202 (9f912511) with cc (Ubuntu 13.2.0-23ubuntu4) 13.2.0 for x86_64-linux-gnu
Dec 02 11:28:30 kamala llama-server[14451]: system info: n_threads = 4, n_threads_batch = 4, total_threads = 8
Dec 02 11:28:30 kamala llama-server[14451]: system_info: n_threads = 4 (n_threads_batch = 4) / 8 | ROCm : PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 |
Dec 02 11:28:30 kamala llama-server[14451]: main: HTTP server is listening, hostname: 0.0.0.0, port: 11430, http threads: 4
Dec 02 11:28:30 kamala llama-server[14451]: main: loading model
Dec 02 11:28:30 kamala llama-server[14451]: srv    load_model: loading model 'models/QwQ-32B-Preview-Q4_K_M.gguf'
Dec 02 11:28:30 kamala llama-server[14451]: llama_load_model_from_file: using device ROCm0 (AMD Radeon Graphics) - 28596 MiB free
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: loaded meta data with 27 key-value pairs and 771 tensors from models/QwQ-32B-Preview-Q4_K_M.gguf (version GGUF V3 (latest))
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv   0:                       general.architecture str              = qwen2
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv   1:                               general.type str              = model
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv   2:                               general.name str              = QwQ 32B Preview
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv   3:                       general.organization str              = Qwen
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv   4:                           general.finetune str              = Preview
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv   5:                           general.basename str              = QwQ
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv   6:                         general.size_label str              = 32B
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv   7:                          qwen2.block_count u32              = 64
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv   8:                       qwen2.context_length u32              = 32768
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv   9:                     qwen2.embedding_length u32              = 5120
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  10:                  qwen2.feed_forward_length u32              = 27648
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  11:                 qwen2.attention.head_count u32              = 40
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  12:              qwen2.attention.head_count_kv u32              = 8
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  13:                       qwen2.rope.freq_base f32              = 1000000.000000
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  14:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000010
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  15:                          general.file_type u32              = 15
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  16:                       tokenizer.ggml.model str              = gpt2
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  17:                         tokenizer.ggml.pre str              = qwen2
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  18:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  19:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  20:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  21:                tokenizer.ggml.eos_token_id u32              = 151645
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  22:            tokenizer.ggml.padding_token_id u32              = 151654
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  23:                tokenizer.ggml.bos_token_id u32              = 151643
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  24:               tokenizer.ggml.add_bos_token bool             = false
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  25:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - kv  26:               general.quantization_version u32              = 2
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - type  f32:  321 tensors
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - type q4_K:  385 tensors
Dec 02 11:28:30 kamala llama-server[14451]: llama_model_loader: - type q6_K:   65 tensors
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_vocab: special tokens cache size = 22
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_vocab: token to piece cache size = 0.9310 MB
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: format           = GGUF V3 (latest)
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: arch             = qwen2
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: vocab type       = BPE
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_vocab          = 152064
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_merges         = 151387
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: vocab_only       = 0
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_ctx_train      = 32768
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_embd           = 5120
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_layer          = 64
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_head           = 40
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_head_kv        = 8
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_rot            = 128
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_swa            = 0
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_embd_head_k    = 128
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_embd_head_v    = 128
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_gqa            = 5
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_embd_k_gqa     = 1024
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_embd_v_gqa     = 1024
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: f_norm_eps       = 0.0e+00
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: f_logit_scale    = 0.0e+00
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_ff             = 27648
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_expert         = 0
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_expert_used    = 0
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: causal attn      = 1
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: pooling type     = 0
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: rope type        = 2
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: rope scaling     = linear
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: freq_base_train  = 1000000.0
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: freq_scale_train = 1
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: n_ctx_orig_yarn  = 32768
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: rope_finetuned   = unknown
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: ssm_d_conv       = 0
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: ssm_d_inner      = 0
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: ssm_d_state      = 0
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: ssm_dt_rank      = 0
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: ssm_dt_b_c_rms   = 0
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: model type       = ?B
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: model ftype      = Q4_K - Medium
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: model params     = 32.76 B
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: model size       = 18.48 GiB (4.85 BPW)
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: general.name     = QwQ 32B Preview
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: EOS token        = 151645 '<|im_end|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: PAD token        = 151654 '<|vision_pad|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: LF token         = 148848 'ÄĬ'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: FIM PRE token    = 151659 '<|fim_prefix|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: FIM SUF token    = 151661 '<|fim_suffix|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: FIM MID token    = 151660 '<|fim_middle|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: FIM PAD token    = 151662 '<|fim_pad|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: FIM REP token    = 151663 '<|repo_name|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: FIM SEP token    = 151664 '<|file_sep|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: EOG token        = 151643 '<|endoftext|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: EOG token        = 151645 '<|im_end|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: EOG token        = 151662 '<|fim_pad|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: EOG token        = 151663 '<|repo_name|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: EOG token        = 151664 '<|file_sep|>'
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_print_meta: max token length = 256
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_tensors: offloading 64 repeating layers to GPU
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_tensors: offloading output layer to GPU
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_tensors: offloaded 65/65 layers to GPU
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_tensors:          CPU model buffer size =   417.66 MiB
Dec 02 11:28:30 kamala llama-server[14451]: llm_load_tensors:        ROCm0 model buffer size = 18508.35 MiB
Dec 02 11:29:52 kamala llama-server[14451]: .................................................................................................
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model: n_seq_max     = 1
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model: n_ctx         = 4096
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model: n_ctx_per_seq = 4096
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model: n_batch       = 4096
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model: n_ubatch      = 512
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model: flash_attn    = 1
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model: freq_base     = 1000000.0
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model: freq_scale    = 1
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
Dec 02 11:29:52 kamala llama-server[14451]: llama_kv_cache_init:      ROCm0 KV buffer size =  1024.00 MiB
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model:  ROCm_Host  output buffer size =     0.58 MiB
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model:      ROCm0 compute buffer size =   307.00 MiB
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model:  ROCm_Host compute buffer size =    18.01 MiB
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model: graph nodes  = 1991
Dec 02 11:29:52 kamala llama-server[14451]: llama_new_context_with_model: graph splits = 2
Dec 02 11:29:52 kamala llama-server[14451]: common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
Dec 02 11:29:52 kamala llama-server[14451]: srv    load_model: loading draft model 'models/Qwen2.5-Coder-3B-Q4_K_L.gguf'
Dec 02 11:29:52 kamala llama-server[14451]: llama_load_model_from_file: using device ROCm0 (AMD Radeon Graphics) - 8700 MiB free
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: loaded meta data with 38 key-value pairs and 434 tensors from models/Qwen2.5-Coder-3B-Q4_K_L.gguf (version GGUF V3 (latest))
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv   0:                       general.architecture str              = qwen2
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv   1:                               general.type str              = model
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 Coder 3B
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv   3:                           general.basename str              = Qwen2.5-Coder
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv   4:                         general.size_label str              = 3B
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv   5:                            general.license str              = other
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv   6:                       general.license.name str              = qwen-research
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv   7:                       general.license.link str              = https://huggingface.co/Qwen/Qwen2.5-C...
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv   9:                  general.base_model.0.name str              = Qwen2.5 3B
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  10:          general.base_model.0.organization str              = Qwen
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  11:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen2.5-3B
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  12:                               general.tags arr[str,5]       = ["code", "qwen", "qwen-coder", "codeq...
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  13:                          general.languages arr[str,1]       = ["en"]
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  14:                          qwen2.block_count u32              = 36
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  15:                       qwen2.context_length u32              = 32768
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  16:                     qwen2.embedding_length u32              = 2048
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  17:                  qwen2.feed_forward_length u32              = 11008
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  18:                 qwen2.attention.head_count u32              = 16
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  19:              qwen2.attention.head_count_kv u32              = 2
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  20:                       qwen2.rope.freq_base f32              = 1000000.000000
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  21:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  22:                          general.file_type u32              = 15
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = qwen2
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 151645
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 151643
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  30:                tokenizer.ggml.bos_token_id u32              = 151643
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = false
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  33:               general.quantization_version u32              = 2
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  34:                      quantize.imatrix.file str              = /models_out/Qwen2.5-Coder-3B-GGUF/Qwe...
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  35:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  36:             quantize.imatrix.entries_count i32              = 252
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - kv  37:              quantize.imatrix.chunks_count i32              = 128
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - type  f32:  181 tensors
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - type q8_0:    1 tensors
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - type q4_K:  216 tensors
Dec 02 11:29:53 kamala llama-server[14451]: llama_model_loader: - type q6_K:   36 tensors
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_vocab: special tokens cache size = 22
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_vocab: token to piece cache size = 0.9310 MB
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: format           = GGUF V3 (latest)
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: arch             = qwen2
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: vocab type       = BPE
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_vocab          = 151936
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_merges         = 151387
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: vocab_only       = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_ctx_train      = 32768
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_embd           = 2048
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_layer          = 36
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_head           = 16
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_head_kv        = 2
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_rot            = 128
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_swa            = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_embd_head_k    = 128
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_embd_head_v    = 128
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_gqa            = 8
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_embd_k_gqa     = 256
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_embd_v_gqa     = 256
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: f_norm_eps       = 0.0e+00
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: f_logit_scale    = 0.0e+00
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_ff             = 11008
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_expert         = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_expert_used    = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: causal attn      = 1
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: pooling type     = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: rope type        = 2
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: rope scaling     = linear
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: freq_base_train  = 1000000.0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: freq_scale_train = 1
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_ctx_orig_yarn  = 32768
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: rope_finetuned   = unknown
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: ssm_d_conv       = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: ssm_d_inner      = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: ssm_d_state      = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: ssm_dt_rank      = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: ssm_dt_b_c_rms   = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: model type       = ?B
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: model ftype      = Q4_K - Medium
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: model params     = 3.09 B
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: model size       = 1.86 GiB (5.18 BPW)
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: general.name     = Qwen2.5 Coder 3B
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: f_norm_eps       = 0.0e+00
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: f_logit_scale    = 0.0e+00
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_ff             = 11008
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_expert         = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_expert_used    = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: causal attn      = 1
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: pooling type     = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: rope type        = 2
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: rope scaling     = linear
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: freq_base_train  = 1000000.0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: freq_scale_train = 1
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_ctx_orig_yarn  = 32768
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: rope_finetuned   = unknown
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: ssm_d_conv       = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: ssm_d_inner      = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: rope type        = 2
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: rope scaling     = linear
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: freq_base_train  = 1000000.0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: freq_scale_train = 1
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: n_ctx_orig_yarn  = 32768
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: rope_finetuned   = unknown
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: ssm_d_conv       = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: ssm_d_inner      = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: ssm_d_state      = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: ssm_dt_rank      = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: ssm_dt_b_c_rms   = 0
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: model type       = ?B
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: model ftype      = Q4_K - Medium
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: model params     = 3.09 B
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: model size       = 1.86 GiB (5.18 BPW)
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: general.name     = Qwen2.5 Coder 3B
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: EOS token        = 151645 '<|im_end|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: PAD token        = 151643 '<|endoftext|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: LF token         = 148848 'ÄĬ'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: FIM PRE token    = 151659 '<|fim_prefix|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: FIM SUF token    = 151661 '<|fim_suffix|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: FIM MID token    = 151660 '<|fim_middle|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: FIM PAD token    = 151662 '<|fim_pad|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: FIM REP token    = 151663 '<|repo_name|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: FIM SEP token    = 151664 '<|file_sep|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: EOG token        = 151643 '<|endoftext|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: EOG token        = 151645 '<|im_end|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: EOG token        = 151662 '<|fim_pad|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: EOG token        = 151663 '<|repo_name|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: EOG token        = 151664 '<|file_sep|>'
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_print_meta: max token length = 256
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_tensors: offloading 36 repeating layers to GPU
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_tensors: offloading output layer to GPU
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_tensors: offloaded 37/37 layers to GPU
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_tensors:        ROCm0 model buffer size =  1906.70 MiB
Dec 02 11:29:53 kamala llama-server[14451]: llm_load_tensors:    ROCm_Host model buffer size =   315.30 MiB
Dec 02 11:30:03 kamala llama-server[14451]: .........................................................................
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: n_seq_max     = 1
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: n_ctx         = 32768
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: n_ctx_per_seq = 32768
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: n_batch       = 8092
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: n_ubatch      = 512
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: flash_attn    = 1
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: freq_base     = 1000000.0
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: freq_scale    = 1
Dec 02 11:30:03 kamala llama-server[14451]: llama_kv_cache_init:      ROCm0 KV buffer size =  1152.00 MiB
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: KV self size  = 1152.00 MiB, K (f16):  576.00 MiB, V (f16):  576.00 MiB
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model:  ROCm_Host  output buffer size =     0.58 MiB
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model:      ROCm0 compute buffer size =   300.75 MiB
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model:  ROCm_Host compute buffer size =    68.01 MiB
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: graph nodes  = 1123
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: graph splits = 2
Dec 02 11:30:03 kamala llama-server[14451]: common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
Dec 02 11:30:03 kamala llama-server[14451]: srv          init: initializing slots, n_slots = 1
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: n_seq_max     = 1
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: n_ctx         = 4096
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: n_ctx_per_seq = 4096
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: n_batch       = 4096
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: n_ubatch      = 512
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: flash_attn    = 1
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: freq_base     = 1000000.0
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: freq_scale    = 1
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
Dec 02 11:30:03 kamala llama-server[14451]: llama_kv_cache_init:      ROCm0 KV buffer size =   144.00 MiB
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: KV self size  =  144.00 MiB, K (f16):   72.00 MiB, V (f16):   72.00 MiB
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model:  ROCm_Host  output buffer size =     0.58 MiB
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model:      ROCm0 compute buffer size =   300.75 MiB
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model:  ROCm_Host compute buffer size =    12.01 MiB
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: graph nodes  = 1123
Dec 02 11:30:03 kamala llama-server[14451]: llama_new_context_with_model: graph splits = 2
Dec 02 11:30:03 kamala llama-server[14451]: slot         init: id  0 | task -1 | new slot n_ctx_slot = 4096
Dec 02 11:30:03 kamala llama-server[14451]: main: model loaded
Dec 02 11:30:03 kamala llama-server[14451]: main: chat template, built_in: 1, chat_example: '<|im_start|>system
Dec 02 11:30:03 kamala llama-server[14451]: You are a helpful assistant<|im_end|>
Dec 02 11:30:03 kamala llama-server[14451]: <|im_start|>user
Dec 02 11:30:03 kamala llama-server[14451]: Hello<|im_end|>
Dec 02 11:30:03 kamala llama-server[14451]: <|im_start|>assistant
Dec 02 11:30:03 kamala llama-server[14451]: Hi there<|im_end|>
Dec 02 11:30:03 kamala llama-server[14451]: <|im_start|>user
Dec 02 11:30:03 kamala llama-server[14451]: How are you?<|im_end|>
Dec 02 11:30:03 kamala llama-server[14451]: <|im_start|>assistant
Dec 02 11:30:03 kamala llama-server[14451]: '
Dec 02 11:30:03 kamala llama-server[14451]: main: server is listening on http://0.0.0.0:11430 - starting the main loop
Dec 02 11:30:03 kamala llama-server[14451]: srv  update_slots: all slots are idle
Dec 02 11:31:11 kamala llama-server[14451]: request: GET /models 127.0.0.1 200
Dec 02 11:31:12 kamala llama-server[14451]: slot launch_slot_: id  0 | task 0 | processing task
Dec 02 11:31:12 kamala llama-server[14451]: slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 24305
Dec 02 11:31:12 kamala llama-server[14451]: slot update_slots: id  0 | task 0 | input truncated, n_ctx = 4096, n_keep = 0, n_left = 4096, n_prompt_tokens = 3825
Dec 02 11:31:12 kamala llama-server[14451]: slot update_slots: id  0 | task 0 | kv cache rm [0, end)
Dec 02 11:31:12 kamala llama-server[14451]: slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 3825, n_tokens = 3825, progress = 1.000000
Dec 02 11:31:12 kamala llama-server[14451]: slot update_slots: id  0 | task 0 | prompt done, n_past = 3825, n_tokens = 3825
Dec 02 11:32:07 kamala llama-server[14451]: slot      release: id  0 | task 0 | stop processing: n_past = 3852, truncated = 1
Dec 02 11:32:07 kamala llama-server[14451]: slot print_timing: id  0 | task 0 |
Dec 02 11:32:07 kamala llama-server[14451]: [100B blob data]
Dec 02 11:32:07 kamala llama-server[14451]: [100B blob data]
Dec 02 11:32:07 kamala llama-server[14451]: [48B blob data]
Dec 02 11:32:07 kamala llama-server[14451]: srv  update_slots: all slots are idle
Dec 02 11:32:07 kamala llama-server[14451]: request: POST /chat/completions 127.0.0.1 200
Dec 02 11:32:07 kamala llama-server[14451]: request: GET /models 127.0.0.1 200
Dec 02 11:32:07 kamala llama-server[14451]: slot launch_slot_: id  0 | task 29 | processing task
Dec 02 11:32:07 kamala llama-server[14451]: slot update_slots: id  0 | task 29 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 34390
Dec 02 11:32:07 kamala llama-server[14451]: slot update_slots: id  0 | task 29 | input truncated, n_ctx = 4096, n_keep = 0, n_left = 4096, n_prompt_tokens = 3670
Dec 02 11:32:07 kamala llama-server[14451]: slot update_slots: id  0 | task 29 | kv cache rm [0, end)
Dec 02 11:32:07 kamala llama-server[14451]: slot update_slots: id  0 | task 29 | prompt processing progress, n_past = 3670, n_tokens = 3670, progress = 1.000000
Dec 02 11:32:07 kamala llama-server[14451]: slot update_slots: id  0 | task 29 | prompt done, n_past = 3670, n_tokens = 3670
Dec 02 11:33:48 kamala llama-server[14451]: slot update_slots: id  0 | task 29 | slot context shift, n_keep = 0, n_left = 4095, n_discard = 2047
Dec 02 11:36:42 kamala llama-server[14451]: llama_decode: failed to decode, ret = 1
Dec 02 11:36:43 kamala systemd[1]: llama.service: Main process exited, code=dumped, status=11/SEGV

Dec 02 11:55:26 kamala systemd[1]: Started llama.service.
░░ Subject: A start job for unit llama.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit llama.service has finished successfully.
░░
░░ The job identifier is 1054885.
Dec 02 11:55:27 kamala llama-server[24689]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Dec 02 11:55:27 kamala llama-server[24689]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Dec 02 11:55:27 kamala llama-server[24689]: ggml_cuda_init: found 1 ROCm devices:
Dec 02 11:55:27 kamala llama-server[24689]:   Device 0: AMD Radeon Graphics, compute capability 9.0, VMM: no
Dec 02 11:55:27 kamala llama-server[24689]: build: 4202 (9f912511) with cc (Ubuntu 13.2.0-23ubuntu4) 13.2.0 for x86_64-linux-gnu
Dec 02 11:55:27 kamala llama-server[24689]: system info: n_threads = 4, n_threads_batch = 4, total_threads = 8
Dec 02 11:55:27 kamala llama-server[24689]: system_info: n_threads = 4 (n_threads_batch = 4) / 8 | ROCm : PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | AARCH64_REPACK = 1 |
Dec 02 11:55:27 kamala llama-server[24689]: main: HTTP server is listening, hostname: 0.0.0.0, port: 11430, http threads: 4
Dec 02 11:55:27 kamala llama-server[24689]: main: loading model
Dec 02 11:55:27 kamala llama-server[24689]: srv    load_model: loading model 'models/QwQ-32B-Preview-Q4_K_M.gguf'
Dec 02 11:55:27 kamala llama-server[24689]: llama_load_model_from_file: using device ROCm0 (AMD Radeon Graphics) - 28596 MiB free
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: loaded meta data with 27 key-value pairs and 771 tensors from models/QwQ-32B-Preview-Q4_K_M.gguf (version GGUF V3 (latest))
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv   0:                       general.architecture str              = qwen2
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv   1:                               general.type str              = model
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv   2:                               general.name str              = QwQ 32B Preview
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv   3:                       general.organization str              = Qwen
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv   4:                           general.finetune str              = Preview
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv   5:                           general.basename str              = QwQ
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv   6:                         general.size_label str              = 32B
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv   7:                          qwen2.block_count u32              = 64
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv   8:                       qwen2.context_length u32              = 32768
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv   9:                     qwen2.embedding_length u32              = 5120
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv  10:                  qwen2.feed_forward_length u32              = 27648
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv  11:                 qwen2.attention.head_count u32              = 40
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv  12:              qwen2.attention.head_count_kv u32              = 8
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv  13:                       qwen2.rope.freq_base f32              = 1000000.000000
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv  14:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000010
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv  15:                          general.file_type u32              = 15
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv  16:                       tokenizer.ggml.model str              = gpt2
Dec 02 11:55:27 kamala llama-server[24689]: llama_model_loader: - kv  17:                         tokenizer.ggml.pre str              = qwen2
Dec 02 11:55:28 kamala llama-server[24689]: llama_model_loader: - kv  18:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
Dec 02 11:55:28 kamala llama-server[24689]: llama_model_loader: - kv  19:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Dec 02 11:55:28 kamala llama-server[24689]: llama_model_loader: - kv  20:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
Dec 02 11:55:28 kamala llama-server[24689]: llama_model_loader: - kv  21:                tokenizer.ggml.eos_token_id u32              = 151645
Dec 02 11:55:28 kamala llama-server[24689]: llama_model_loader: - kv  22:            tokenizer.ggml.padding_token_id u32              = 151654
Dec 02 11:55:28 kamala llama-server[24689]: llama_model_loader: - kv  23:                tokenizer.ggml.bos_token_id u32              = 151643
Dec 02 11:55:28 kamala llama-server[24689]: llama_model_loader: - kv  24:               tokenizer.ggml.add_bos_token bool             = false
Dec 02 11:55:28 kamala llama-server[24689]: llama_model_loader: - kv  25:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
Dec 02 11:55:28 kamala llama-server[24689]: llama_model_loader: - kv  26:               general.quantization_version u32              = 2
Dec 02 11:55:28 kamala llama-server[24689]: llama_model_loader: - type  f32:  321 tensors
Dec 02 11:55:28 kamala llama-server[24689]: llama_model_loader: - type q4_K:  385 tensors
Dec 02 11:55:28 kamala llama-server[24689]: llama_model_loader: - type q6_K:   65 tensors
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_vocab: special tokens cache size = 22
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_vocab: token to piece cache size = 0.9310 MB
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: format           = GGUF V3 (latest)
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: arch             = qwen2
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: vocab type       = BPE
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_vocab          = 152064
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_merges         = 151387
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: vocab_only       = 0
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_ctx_train      = 32768
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_embd           = 5120
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_layer          = 64
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_head           = 40
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_head_kv        = 8
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_rot            = 128
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_swa            = 0
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_embd_head_k    = 128
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_embd_head_v    = 128
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_gqa            = 5
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_embd_k_gqa     = 1024
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_embd_v_gqa     = 1024
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: f_norm_eps       = 0.0e+00
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: f_logit_scale    = 0.0e+00
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_ff             = 27648
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_expert         = 0
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_expert_used    = 0
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: causal attn      = 1
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: pooling type     = 0
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: rope type        = 2
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: rope scaling     = linear
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: freq_base_train  = 1000000.0
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: freq_scale_train = 1
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: n_ctx_orig_yarn  = 32768
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: rope_finetuned   = unknown
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: ssm_d_conv       = 0
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: ssm_d_inner      = 0
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: ssm_d_state      = 0
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: ssm_dt_rank      = 0
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: ssm_dt_b_c_rms   = 0
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: model type       = ?B
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: model ftype      = Q4_K - Medium
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: model params     = 32.76 B
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: model size       = 18.48 GiB (4.85 BPW)
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: general.name     = QwQ 32B Preview
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: EOS token        = 151645 '<|im_end|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: PAD token        = 151654 '<|vision_pad|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: LF token         = 148848 'ÄĬ'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: FIM PRE token    = 151659 '<|fim_prefix|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: FIM SUF token    = 151661 '<|fim_suffix|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: FIM MID token    = 151660 '<|fim_middle|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: FIM PAD token    = 151662 '<|fim_pad|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: FIM REP token    = 151663 '<|repo_name|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: FIM SEP token    = 151664 '<|file_sep|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: EOG token        = 151643 '<|endoftext|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: EOG token        = 151645 '<|im_end|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: EOG token        = 151662 '<|fim_pad|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: EOG token        = 151663 '<|repo_name|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: EOG token        = 151664 '<|file_sep|>'
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_print_meta: max token length = 256
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_tensors: offloading 64 repeating layers to GPU
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_tensors: offloading output layer to GPU
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_tensors: offloaded 65/65 layers to GPU
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_tensors:          CPU model buffer size =   417.66 MiB
Dec 02 11:55:28 kamala llama-server[24689]: llm_load_tensors:        ROCm0 model buffer size = 18508.35 MiB
Dec 02 11:56:33 kamala llama-server[24689]: ............................................................................request: GET /models 127.0.0.1 503
Dec 02 11:56:50 kamala llama-server[24689]: .....................
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model: n_seq_max     = 1
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model: n_ctx         = 4096
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model: n_ctx_per_seq = 4096
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model: n_batch       = 4096
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model: n_ubatch      = 512
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model: flash_attn    = 1
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model: freq_base     = 1000000.0
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model: freq_scale    = 1
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
Dec 02 11:56:50 kamala llama-server[24689]: llama_kv_cache_init:      ROCm0 KV buffer size =  1024.00 MiB
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model:  ROCm_Host  output buffer size =     0.58 MiB
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model:      ROCm0 compute buffer size =   307.00 MiB
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model:  ROCm_Host compute buffer size =    18.01 MiB
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model: graph nodes  = 1991
Dec 02 11:56:50 kamala llama-server[24689]: llama_new_context_with_model: graph splits = 2
Dec 02 11:56:50 kamala llama-server[24689]: common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
Dec 02 11:56:50 kamala llama-server[24689]: srv    load_model: loading draft model 'models/Qwen2.5-Coder-3B-Q4_K_L.gguf'
Dec 02 11:56:50 kamala llama-server[24689]: llama_load_model_from_file: using device ROCm0 (AMD Radeon Graphics) - 8700 MiB free
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: loaded meta data with 38 key-value pairs and 434 tensors from models/Qwen2.5-Coder-3B-Q4_K_L.gguf (version GGUF V3 (latest))
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv   0:                       general.architecture str              = qwen2
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv   1:                               general.type str              = model
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 Coder 3B
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv   3:                           general.basename str              = Qwen2.5-Coder
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv   4:                         general.size_label str              = 3B
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv   5:                            general.license str              = other
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv   6:                       general.license.name str              = qwen-research
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv   7:                       general.license.link str              = https://huggingface.co/Qwen/Qwen2.5-C...
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv   9:                  general.base_model.0.name str              = Qwen2.5 3B
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  10:          general.base_model.0.organization str              = Qwen
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  11:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen2.5-3B
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  12:                               general.tags arr[str,5]       = ["code", "qwen", "qwen-coder", "codeq...
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  13:                          general.languages arr[str,1]       = ["en"]
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  14:                          qwen2.block_count u32              = 36
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  15:                       qwen2.context_length u32              = 32768
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  16:                     qwen2.embedding_length u32              = 2048
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  17:                  qwen2.feed_forward_length u32              = 11008
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  18:                 qwen2.attention.head_count u32              = 16
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  19:              qwen2.attention.head_count_kv u32              = 2
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  20:                       qwen2.rope.freq_base f32              = 1000000.000000
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  21:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  22:                          general.file_type u32              = 15
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = qwen2
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 151645
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 151643
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  30:                tokenizer.ggml.bos_token_id u32              = 151643
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = false
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  33:               general.quantization_version u32              = 2
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  34:                      quantize.imatrix.file str              = /models_out/Qwen2.5-Coder-3B-GGUF/Qwe...
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  35:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  36:             quantize.imatrix.entries_count i32              = 252
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - kv  37:              quantize.imatrix.chunks_count i32              = 128
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - type  f32:  181 tensors
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - type q8_0:    1 tensors
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - type q4_K:  216 tensors
Dec 02 11:56:51 kamala llama-server[24689]: llama_model_loader: - type q6_K:   36 tensors
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_vocab: special tokens cache size = 22
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_vocab: token to piece cache size = 0.9310 MB
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: format           = GGUF V3 (latest)
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: arch             = qwen2
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: vocab type       = BPE
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_vocab          = 151936
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_merges         = 151387
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: vocab_only       = 0
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_ctx_train      = 32768
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_embd           = 2048
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_layer          = 36
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_head           = 16
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_head_kv        = 2
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_rot            = 128
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_swa            = 0
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_embd_head_k    = 128
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_embd_head_v    = 128
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_gqa            = 8
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_embd_k_gqa     = 256
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_embd_v_gqa     = 256
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: f_norm_eps       = 0.0e+00
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: f_logit_scale    = 0.0e+00
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_ff             = 11008
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_expert         = 0
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_expert_used    = 0
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: causal attn      = 1
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: pooling type     = 0
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: rope type        = 2
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: rope scaling     = linear
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: freq_base_train  = 1000000.0
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: freq_scale_train = 1
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: n_ctx_orig_yarn  = 32768
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: rope_finetuned   = unknown
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: ssm_d_conv       = 0
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: ssm_d_inner      = 0
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: ssm_d_state      = 0
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: ssm_dt_rank      = 0
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: ssm_dt_b_c_rms   = 0
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: model type       = ?B
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: model ftype      = Q4_K - Medium
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: model params     = 3.09 B
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: model size       = 1.86 GiB (5.18 BPW)
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: general.name     = Qwen2.5 Coder 3B
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: EOS token        = 151645 '<|im_end|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: PAD token        = 151643 '<|endoftext|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: LF token         = 148848 'ÄĬ'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: FIM PRE token    = 151659 '<|fim_prefix|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: FIM SUF token    = 151661 '<|fim_suffix|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: FIM MID token    = 151660 '<|fim_middle|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: FIM PAD token    = 151662 '<|fim_pad|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: FIM REP token    = 151663 '<|repo_name|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: FIM SEP token    = 151664 '<|file_sep|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: EOG token        = 151643 '<|endoftext|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: EOG token        = 151645 '<|im_end|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: EOG token        = 151662 '<|fim_pad|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: EOG token        = 151663 '<|repo_name|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: EOG token        = 151664 '<|file_sep|>'
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_print_meta: max token length = 256
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_tensors: offloading 36 repeating layers to GPU
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_tensors: offloading output layer to GPU
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_tensors: offloaded 37/37 layers to GPU
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_tensors:        ROCm0 model buffer size =  1906.70 MiB
Dec 02 11:56:51 kamala llama-server[24689]: llm_load_tensors:    ROCm_Host model buffer size =   315.30 MiB
Dec 02 11:57:01 kamala llama-server[24689]: .........................................................................
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: n_seq_max     = 1
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: n_ctx         = 32768
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: n_ctx_per_seq = 32768
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: n_batch       = 8092
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: n_ubatch      = 512
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: flash_attn    = 1
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: freq_base     = 1000000.0
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: freq_scale    = 1
Dec 02 11:57:01 kamala llama-server[24689]: llama_kv_cache_init:      ROCm0 KV buffer size =  1152.00 MiB
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: KV self size  = 1152.00 MiB, K (f16):  576.00 MiB, V (f16):  576.00 MiB
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model:  ROCm_Host  output buffer size =     0.58 MiB
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model:      ROCm0 compute buffer size =   300.75 MiB
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model:  ROCm_Host compute buffer size =    68.01 MiB
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: graph nodes  = 1123
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: graph splits = 2
Dec 02 11:57:01 kamala llama-server[24689]: common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
Dec 02 11:57:01 kamala llama-server[24689]: srv          init: initializing slots, n_slots = 1
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: n_seq_max     = 1
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: n_ctx         = 4096
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: n_ctx_per_seq = 4096
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: n_batch       = 4096
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: n_ubatch      = 512
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: flash_attn    = 1
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: freq_base     = 1000000.0
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: freq_scale    = 1
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
Dec 02 11:57:01 kamala llama-server[24689]: llama_kv_cache_init:      ROCm0 KV buffer size =   144.00 MiB
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: KV self size  =  144.00 MiB, K (f16):   72.00 MiB, V (f16):   72.00 MiB
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model:  ROCm_Host  output buffer size =     0.58 MiB
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model:      ROCm0 compute buffer size =   300.75 MiB
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model:  ROCm_Host compute buffer size =    12.01 MiB
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: graph nodes  = 1123
Dec 02 11:57:01 kamala llama-server[24689]: llama_new_context_with_model: graph splits = 2
Dec 02 11:57:01 kamala llama-server[24689]: slot         init: id  0 | task -1 | new slot n_ctx_slot = 4096
Dec 02 11:57:01 kamala llama-server[24689]: main: model loaded
Dec 02 11:57:01 kamala llama-server[24689]: main: chat template, built_in: 1, chat_example: '<|im_start|>system
Dec 02 11:57:01 kamala llama-server[24689]: You are a helpful assistant<|im_end|>
Dec 02 11:57:01 kamala llama-server[24689]: <|im_start|>user
Dec 02 11:57:01 kamala llama-server[24689]: Hello<|im_end|>
Dec 02 11:57:01 kamala llama-server[24689]: <|im_start|>assistant
Dec 02 11:57:01 kamala llama-server[24689]: Hi there<|im_end|>
Dec 02 11:57:01 kamala llama-server[24689]: <|im_start|>user
Dec 02 11:57:01 kamala llama-server[24689]: How are you?<|im_end|>
Dec 02 11:57:01 kamala llama-server[24689]: <|im_start|>assistant
Dec 02 11:57:01 kamala llama-server[24689]: '
Dec 02 11:57:01 kamala llama-server[24689]: main: server is listening on http://0.0.0.0:11430 - starting the main loop
Dec 02 11:57:01 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 11:57:17 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 11:57:18 kamala llama-server[24689]: slot launch_slot_: id  0 | task 0 | processing task
Dec 02 11:57:18 kamala llama-server[24689]: slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 11456
Dec 02 11:57:18 kamala llama-server[24689]: slot update_slots: id  0 | task 0 | input truncated, n_ctx = 4096, n_keep = 0, n_left = 4096, n_prompt_tokens = 3264
Dec 02 11:57:18 kamala llama-server[24689]: slot update_slots: id  0 | task 0 | kv cache rm [0, end)
Dec 02 11:57:18 kamala llama-server[24689]: slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 3264, n_tokens = 3264, progress = 1.000000
Dec 02 11:57:18 kamala llama-server[24689]: slot update_slots: id  0 | task 0 | prompt done, n_past = 3264, n_tokens = 3264
Dec 02 11:58:00 kamala llama-server[24689]: slot      release: id  0 | task 0 | stop processing: n_past = 3266, truncated = 1
Dec 02 11:58:00 kamala llama-server[24689]: slot print_timing: id  0 | task 0 |
Dec 02 11:58:00 kamala llama-server[24689]: [100B blob data]
Dec 02 11:58:00 kamala llama-server[24689]: [100B blob data]
Dec 02 11:58:00 kamala llama-server[24689]: [48B blob data]
Dec 02 11:58:00 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 11:58:00 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 11:58:42 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 11:58:42 kamala llama-server[24689]: slot launch_slot_: id  0 | task 4 | processing task
Dec 02 11:58:42 kamala llama-server[24689]: slot update_slots: id  0 | task 4 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 14103
Dec 02 11:58:42 kamala llama-server[24689]: slot update_slots: id  0 | task 4 | input truncated, n_ctx = 4096, n_keep = 0, n_left = 4096, n_prompt_tokens = 3863
Dec 02 11:58:42 kamala llama-server[24689]: slot update_slots: id  0 | task 4 | kv cache rm [0, end)
Dec 02 11:58:42 kamala llama-server[24689]: slot update_slots: id  0 | task 4 | prompt processing progress, n_past = 3863, n_tokens = 3863, progress = 1.000000
Dec 02 11:58:42 kamala llama-server[24689]: slot update_slots: id  0 | task 4 | prompt done, n_past = 3863, n_tokens = 3863
Dec 02 12:00:05 kamala llama-server[24689]: slot update_slots: id  0 | task 4 | slot context shift, n_keep = 0, n_left = 4095, n_discard = 2047
Dec 02 12:00:38 kamala llama-server[24689]: slot      release: id  0 | task 4 | stop processing: n_past = 2352, truncated = 1
Dec 02 12:00:38 kamala llama-server[24689]: slot print_timing: id  0 | task 4 |
Dec 02 12:00:38 kamala llama-server[24689]: [100B blob data]
Dec 02 12:00:38 kamala llama-server[24689]: [100B blob data]
Dec 02 12:00:38 kamala llama-server[24689]: [48B blob data]
Dec 02 12:00:38 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 12:00:38 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 12:00:38 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 12:10:21 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 12:10:21 kamala llama-server[24689]: slot launch_slot_: id  0 | task 519 | processing task
Dec 02 12:10:21 kamala llama-server[24689]: slot update_slots: id  0 | task 519 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 339
Dec 02 12:10:21 kamala llama-server[24689]: slot update_slots: id  0 | task 519 | kv cache rm [0, end)
Dec 02 12:10:21 kamala llama-server[24689]: slot update_slots: id  0 | task 519 | prompt processing progress, n_past = 339, n_tokens = 339, progress = 1.000000
Dec 02 12:10:21 kamala llama-server[24689]: slot update_slots: id  0 | task 519 | prompt done, n_past = 339, n_tokens = 339
Dec 02 12:10:29 kamala llama-server[24689]: slot      release: id  0 | task 519 | stop processing: n_past = 380, truncated = 0
Dec 02 12:10:29 kamala llama-server[24689]: slot print_timing: id  0 | task 519 |
Dec 02 12:10:29 kamala llama-server[24689]: [100B blob data]
Dec 02 12:10:29 kamala llama-server[24689]: [100B blob data]
Dec 02 12:10:29 kamala llama-server[24689]: [48B blob data]
Dec 02 12:10:29 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 12:10:29 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 12:10:37 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 12:10:37 kamala llama-server[24689]: slot launch_slot_: id  0 | task 562 | processing task
Dec 02 12:10:37 kamala llama-server[24689]: slot update_slots: id  0 | task 562 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 342
Dec 02 12:10:37 kamala llama-server[24689]: slot update_slots: id  0 | task 562 | kv cache rm [326, end)
Dec 02 12:10:37 kamala llama-server[24689]: slot update_slots: id  0 | task 562 | prompt processing progress, n_past = 342, n_tokens = 16, progress = 0.046784
Dec 02 12:10:37 kamala llama-server[24689]: slot update_slots: id  0 | task 562 | prompt done, n_past = 342, n_tokens = 16
Dec 02 12:10:59 kamala llama-server[24689]: slot      release: id  0 | task 562 | stop processing: n_past = 678, truncated = 0
Dec 02 12:10:59 kamala llama-server[24689]: slot print_timing: id  0 | task 562 |
Dec 02 12:10:59 kamala llama-server[24689]: [100B blob data]
Dec 02 12:10:59 kamala llama-server[24689]: [100B blob data]
Dec 02 12:10:59 kamala llama-server[24689]: [48B blob data]
Dec 02 12:10:59 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 12:10:59 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 19:43:24 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 19:43:24 kamala llama-server[24689]: slot launch_slot_: id  0 | task 623 | processing task
Dec 02 19:43:24 kamala llama-server[24689]: slot update_slots: id  0 | task 623 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 342
Dec 02 19:43:24 kamala llama-server[24689]: slot update_slots: id  0 | task 623 | need to evaluate at least 1 token to generate logits, n_past = 342, n_prompt_tokens = 342
Dec 02 19:43:24 kamala llama-server[24689]: slot update_slots: id  0 | task 623 | kv cache rm [341, end)
Dec 02 19:43:24 kamala llama-server[24689]: slot update_slots: id  0 | task 623 | prompt processing progress, n_past = 342, n_tokens = 1, progress = 0.002924
Dec 02 19:43:24 kamala llama-server[24689]: slot update_slots: id  0 | task 623 | prompt done, n_past = 342, n_tokens = 1
Dec 02 19:43:36 kamala llama-server[24689]: slot      release: id  0 | task 623 | stop processing: n_past = 676, truncated = 0
Dec 02 19:43:36 kamala llama-server[24689]: slot print_timing: id  0 | task 623 |
Dec 02 19:43:36 kamala llama-server[24689]: [100B blob data]
Dec 02 19:43:36 kamala llama-server[24689]: [100B blob data]
Dec 02 19:43:36 kamala llama-server[24689]: [48B blob data]
Dec 02 19:43:36 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 19:43:36 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 21:37:36 kamala systemd[1]: /etc/systemd/system/llama.service:1: Assignment outside of section. Ignoring.
Dec 02 21:37:36 kamala systemd[1]: /etc/systemd/system/llama.service:2: Assignment outside of section. Ignoring.
Dec 02 21:37:36 kamala systemd[1]: /etc/systemd/system/llama.service:3: Assignment outside of section. Ignoring.
Dec 02 21:38:38 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 21:38:38 kamala llama-server[24689]: slot launch_slot_: id  0 | task 643 | processing task
Dec 02 21:38:38 kamala llama-server[24689]: slot update_slots: id  0 | task 643 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 339
Dec 02 21:38:38 kamala llama-server[24689]: slot update_slots: id  0 | task 643 | kv cache rm [326, end)
Dec 02 21:38:38 kamala llama-server[24689]: slot update_slots: id  0 | task 643 | prompt processing progress, n_past = 339, n_tokens = 13, progress = 0.038348
Dec 02 21:38:38 kamala llama-server[24689]: slot update_slots: id  0 | task 643 | prompt done, n_past = 339, n_tokens = 13
Dec 02 21:39:21 kamala llama-server[24689]: slot      release: id  0 | task 643 | stop processing: n_past = 832, truncated = 0
Dec 02 21:39:21 kamala llama-server[24689]: slot print_timing: id  0 | task 643 |
Dec 02 21:39:21 kamala llama-server[24689]: [100B blob data]
Dec 02 21:39:21 kamala llama-server[24689]: [100B blob data]
Dec 02 21:39:21 kamala llama-server[24689]: [48B blob data]
Dec 02 21:39:21 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 21:39:21 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 21:39:27 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 21:39:28 kamala llama-server[24689]: slot launch_slot_: id  0 | task 1027 | processing task
Dec 02 21:39:28 kamala llama-server[24689]: slot update_slots: id  0 | task 1027 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 372
Dec 02 21:39:28 kamala llama-server[24689]: slot update_slots: id  0 | task 1027 | kv cache rm [326, end)
Dec 02 21:39:28 kamala llama-server[24689]: slot update_slots: id  0 | task 1027 | prompt processing progress, n_past = 372, n_tokens = 46, progress = 0.123656
Dec 02 21:39:28 kamala llama-server[24689]: slot update_slots: id  0 | task 1027 | prompt done, n_past = 372, n_tokens = 46
Dec 02 21:39:30 kamala llama-server[24689]: slot      release: id  0 | task 1027 | stop processing: n_past = 394, truncated = 0
Dec 02 21:39:30 kamala llama-server[24689]: slot print_timing: id  0 | task 1027 |
Dec 02 21:39:30 kamala llama-server[24689]: [100B blob data]
Dec 02 21:39:30 kamala llama-server[24689]: [100B blob data]
Dec 02 21:39:30 kamala llama-server[24689]: [48B blob data]
Dec 02 21:39:30 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 21:39:30 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 21:39:47 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 21:39:47 kamala llama-server[24689]: slot launch_slot_: id  0 | task 1051 | processing task
Dec 02 21:39:47 kamala llama-server[24689]: slot update_slots: id  0 | task 1051 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 397
Dec 02 21:39:47 kamala llama-server[24689]: slot update_slots: id  0 | task 1051 | kv cache rm [357, end)
Dec 02 21:39:47 kamala llama-server[24689]: slot update_slots: id  0 | task 1051 | prompt processing progress, n_past = 397, n_tokens = 40, progress = 0.100756
Dec 02 21:39:47 kamala llama-server[24689]: slot update_slots: id  0 | task 1051 | prompt done, n_past = 397, n_tokens = 40
Dec 02 21:41:27 kamala llama-server[24689]: slot      release: id  0 | task 1051 | stop processing: n_past = 1456, truncated = 0
Dec 02 21:41:27 kamala llama-server[24689]: slot print_timing: id  0 | task 1051 |
Dec 02 21:41:27 kamala llama-server[24689]: [100B blob data]
Dec 02 21:41:27 kamala llama-server[24689]: [100B blob data]
Dec 02 21:41:27 kamala llama-server[24689]: [48B blob data]
Dec 02 21:41:27 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 21:41:27 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 21:41:45 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 21:41:45 kamala llama-server[24689]: slot launch_slot_: id  0 | task 2061 | processing task
Dec 02 21:41:45 kamala llama-server[24689]: slot update_slots: id  0 | task 2061 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 422
Dec 02 21:41:45 kamala llama-server[24689]: slot update_slots: id  0 | task 2061 | kv cache rm [351, end)
Dec 02 21:41:45 kamala llama-server[24689]: slot update_slots: id  0 | task 2061 | prompt processing progress, n_past = 422, n_tokens = 71, progress = 0.168246
Dec 02 21:41:45 kamala llama-server[24689]: slot update_slots: id  0 | task 2061 | prompt done, n_past = 422, n_tokens = 71
Dec 02 21:44:21 kamala llama-server[24689]: slot      release: id  0 | task 2061 | stop processing: n_past = 2121, truncated = 0
Dec 02 21:44:21 kamala llama-server[24689]: slot print_timing: id  0 | task 2061 |
Dec 02 21:44:21 kamala llama-server[24689]: [100B blob data]
Dec 02 21:44:21 kamala llama-server[24689]: [100B blob data]
Dec 02 21:44:21 kamala llama-server[24689]: [48B blob data]
Dec 02 21:44:21 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 21:44:21 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 21:44:24 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 21:44:24 kamala llama-server[24689]: slot launch_slot_: id  0 | task 3140 | processing task
Dec 02 21:44:24 kamala llama-server[24689]: slot update_slots: id  0 | task 3140 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 502
Dec 02 21:44:24 kamala llama-server[24689]: slot update_slots: id  0 | task 3140 | kv cache rm [410, end)
Dec 02 21:44:24 kamala llama-server[24689]: slot update_slots: id  0 | task 3140 | prompt processing progress, n_past = 502, n_tokens = 92, progress = 0.183267
Dec 02 21:44:24 kamala llama-server[24689]: slot update_slots: id  0 | task 3140 | prompt done, n_past = 502, n_tokens = 92
Dec 02 21:46:00 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 21:46:02 kamala llama-server[24689]: slot      release: id  0 | task 3140 | stop processing: n_past = 1539, truncated = 0
Dec 02 21:46:02 kamala llama-server[24689]: slot print_timing: id  0 | task 3140 |
Dec 02 21:46:02 kamala llama-server[24689]: [100B blob data]
Dec 02 21:46:02 kamala llama-server[24689]: [100B blob data]
Dec 02 21:46:02 kamala llama-server[24689]: [48B blob data]
Dec 02 21:46:02 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 21:46:02 kamala llama-server[24689]: slot launch_slot_: id  0 | task 3743 | processing task
Dec 02 21:46:02 kamala llama-server[24689]: slot update_slots: id  0 | task 3743 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 6364
Dec 02 21:46:02 kamala llama-server[24689]: slot update_slots: id  0 | task 3743 | input truncated, n_ctx = 4096, n_keep = 0, n_left = 4096, n_prompt_tokens = 2268
Dec 02 21:46:02 kamala llama-server[24689]: slot update_slots: id  0 | task 3743 | kv cache rm [0, end)
Dec 02 21:46:02 kamala llama-server[24689]: slot update_slots: id  0 | task 3743 | prompt processing progress, n_past = 2268, n_tokens = 2268, progress = 1.000000
Dec 02 21:46:02 kamala llama-server[24689]: slot update_slots: id  0 | task 3743 | prompt done, n_past = 2268, n_tokens = 2268
Dec 02 21:48:05 kamala llama-server[24689]: slot      release: id  0 | task 3743 | stop processing: n_past = 3078, truncated = 1
Dec 02 21:48:05 kamala llama-server[24689]: slot print_timing: id  0 | task 3743 |
Dec 02 21:48:05 kamala llama-server[24689]: [100B blob data]
Dec 02 21:48:05 kamala llama-server[24689]: [100B blob data]
Dec 02 21:48:05 kamala llama-server[24689]: [48B blob data]
Dec 02 21:48:05 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 21:48:05 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 21:48:05 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 21:48:06 kamala llama-server[24689]: slot launch_slot_: id  0 | task 4494 | processing task
Dec 02 21:48:06 kamala llama-server[24689]: slot update_slots: id  0 | task 4494 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 12464
Dec 02 21:48:06 kamala llama-server[24689]: slot update_slots: id  0 | task 4494 | input truncated, n_ctx = 4096, n_keep = 0, n_left = 4096, n_prompt_tokens = 2224
Dec 02 21:48:06 kamala llama-server[24689]: slot update_slots: id  0 | task 4494 | kv cache rm [0, end)
Dec 02 21:48:06 kamala llama-server[24689]: slot update_slots: id  0 | task 4494 | prompt processing progress, n_past = 2224, n_tokens = 2224, progress = 1.000000
Dec 02 21:48:06 kamala llama-server[24689]: slot update_slots: id  0 | task 4494 | prompt done, n_past = 2224, n_tokens = 2224
Dec 02 21:50:47 kamala llama-server[24689]: slot      release: id  0 | task 4494 | stop processing: n_past = 3424, truncated = 1
Dec 02 21:50:47 kamala llama-server[24689]: slot print_timing: id  0 | task 4494 |
Dec 02 21:50:47 kamala llama-server[24689]: [100B blob data]
Dec 02 21:50:47 kamala llama-server[24689]: [100B blob data]
Dec 02 21:50:47 kamala llama-server[24689]: [48B blob data]
Dec 02 21:50:47 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 21:50:47 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 21:50:47 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 21:50:47 kamala llama-server[24689]: slot launch_slot_: id  0 | task 5123 | processing task
Dec 02 21:50:47 kamala llama-server[24689]: slot update_slots: id  0 | task 5123 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 7377
Dec 02 21:50:47 kamala llama-server[24689]: slot update_slots: id  0 | task 5123 | input truncated, n_ctx = 4096, n_keep = 0, n_left = 4096, n_prompt_tokens = 3281
Dec 02 21:50:47 kamala llama-server[24689]: slot update_slots: id  0 | task 5123 | kv cache rm [0, end)
Dec 02 21:50:47 kamala llama-server[24689]: slot update_slots: id  0 | task 5123 | prompt processing progress, n_past = 3281, n_tokens = 3281, progress = 1.000000
Dec 02 21:50:47 kamala llama-server[24689]: slot update_slots: id  0 | task 5123 | prompt done, n_past = 3281, n_tokens = 3281
Dec 02 21:51:35 kamala llama-server[24689]: slot      release: id  0 | task 5123 | stop processing: n_past = 3343, truncated = 1
Dec 02 21:51:35 kamala llama-server[24689]: slot print_timing: id  0 | task 5123 |
Dec 02 21:51:35 kamala llama-server[24689]: [100B blob data]
Dec 02 21:51:35 kamala llama-server[24689]: [100B blob data]
Dec 02 21:51:35 kamala llama-server[24689]: [48B blob data]
Dec 02 21:51:35 kamala llama-server[24689]: srv  update_slots: all slots are idle
Dec 02 21:51:35 kamala llama-server[24689]: request: POST /chat/completions 127.0.0.1 200
Dec 02 21:51:35 kamala llama-server[24689]: request: GET /models 127.0.0.1 200
Dec 02 21:51:36 kamala llama-server[24689]: slot launch_slot_: id  0 | task 5136 | processing task
Dec 02 21:51:36 kamala llama-server[24689]: slot update_slots: id  0 | task 5136 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 7437
Dec 02 21:51:36 kamala llama-server[24689]: slot update_slots: id  0 | task 5136 | input truncated, n_ctx = 4096, n_keep = 0, n_left = 4096, n_prompt_tokens = 3341
Dec 02 21:51:36 kamala llama-server[24689]: slot update_slots: id  0 | task 5136 | kv cache rm [0, end)
Dec 02 21:51:36 kamala llama-server[24689]: slot update_slots: id  0 | task 5136 | prompt processing progress, n_past = 3341, n_tokens = 3341, progress = 1.000000
Dec 02 21:51:36 kamala llama-server[24689]: slot update_slots: id  0 | task 5136 | prompt done, n_past = 3341, n_tokens = 3341
Dec 02 21:53:38 kamala llama-server[24689]: llama_decode: failed to decode, ret = 1
Dec 02 21:53:38 kamala systemd[1]: llama.service: Main process exited, code=dumped, status=11/SEGV

i left llama.cpp on here for a while, and once i initiated the prompt with a bunch of tokens, it crashed

[unclemusclez]

@josharian it is possible we are not using the correct models/model architecture for this process.

https://docs.vllm.ai/en/v0.5.5/models/spec_decode.html

https://huggingface.co/models?search=accelerator

i get these type of responses using your patch: