Bug: Slow response times with llama.cpp llama-server

[phly95]

What happened?

When running:
.\llama-cli -m gemma-2-2b-it-Q4_K_M.gguf --threads 16 -ngl 27 --mlock --port 11484 --host 0.0.0.0 --top_k 40 --repeat_penalty 1.1 --min_p 0.05 --top_p 0.95 --prompt-cache-all -cb -np 4 --batch-size 512 -cnv

The output is blazing fast. When I sent "write a story" these are my speed stats:
llama_print_timings: load time = 1027.95 ms
llama_print_timings: sample time = 683.92 ms / 618 runs ( 1.11 ms per token, 903.62 tokens per second)
llama_print_timings: prompt eval time = 3678.86 ms / 12 tokens ( 306.57 ms per token, 3.26 tokens per second)
llama_print_timings: eval time = 4744.65 ms / 617 runs ( 7.69 ms per token, 130.04 tokens per second)
llama_print_timings: total time = 15385.66 ms / 629 tokens

When I try the same with llama server:
.\llama-server -m gemma-2-2b-it-Q4_K_M.gguf --threads 16 -ngl 27 --mlock --port 11484 --host 0.0.0.0 --top_k 40 --repeat_penalty 1.1 --min_p 0.05 --top_p 0.95 --prompt-cache-all -cb -np 4 --batch-size 512 -cnv

It takes 2-3 seconds before responding to my same "write a story" prompt on localhost. So while it does write fast, it generated at about 103 t/s (a good amount slower than CLI), and it takes a very long time (for my use-case) to start writing.

If the prompt is extremely similar (up to about 1-3 tokens difference at the end) to my last prompt the next generation will start quick, otherwise, it takes 2-3 seconds, even if the prompt is very short. Only extremely short prompts like "hi" or "yo" respond instantly.

Not quite sure why I'm getting such amazing performance with llama-cli but much worse performance when I use it in server mode.

Name and Version

.\llama-cli --version
version: 3578 (1f67436)
built with MSVC 19.29.30154.0 for x64

What operating system are you seeing the problem on?

Windows 11

Relevant log output

For the server:

> .\llama-server -m gemma-2-2b-it-Q4_K_M.gguf --threads 16 -ngl 27 --mlock --port 11484 --top_k 40 --repeat_penalty 1.1 --min_p 0.05 --top_p 0.95 --prompt-cache-all -cb -np 4
 --batch-size 512 -cnv
INFO [                    main] build info | tid="11744" timestamp=1723514860 build=3578 commit="1f67436c"
INFO [                    main] system info | tid="11744" timestamp=1723514860 n_threads=16 n_threads_batch=-1 total_threads=16 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | "
llama_model_loader: loaded meta data with 39 key-value pairs and 288 tensors from gemma-2-2b-it-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Gemma 2 2b It
llama_model_loader: - kv   3:                           general.finetune str              = it
llama_model_loader: - kv   4:                           general.basename str              = gemma-2
llama_model_loader: - kv   5:                         general.size_label str              = 2B
llama_model_loader: - kv   6:                            general.license str              = gemma
llama_model_loader: - kv   7:                               general.tags arr[str,2]       = ["conversational", "text-generation"]
llama_model_loader: - kv   8:                      gemma2.context_length u32              = 8192
llama_model_loader: - kv   9:                    gemma2.embedding_length u32              = 2304
llama_model_loader: - kv  10:                         gemma2.block_count u32              = 26
llama_model_loader: - kv  11:                 gemma2.feed_forward_length u32              = 9216
llama_model_loader: - kv  12:                gemma2.attention.head_count u32              = 8
llama_model_loader: - kv  13:             gemma2.attention.head_count_kv u32              = 4
llama_model_loader: - kv  14:    gemma2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  15:                gemma2.attention.key_length u32              = 256
llama_model_loader: - kv  16:              gemma2.attention.value_length u32              = 256
llama_model_loader: - kv  17:                          general.file_type u32              = 15
llama_model_loader: - kv  18:              gemma2.attn_logit_softcapping f32              = 50.000000
llama_model_loader: - kv  19:             gemma2.final_logit_softcapping f32              = 30.000000
llama_model_loader: - kv  20:            gemma2.attention.sliding_window u32              = 4096
llama_model_loader: - kv  21:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  22:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  23:                      tokenizer.ggml.tokens arr[str,256000]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  24:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  26:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  27:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  28:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  30:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  31:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {{ bos_token }}{% if messages[0]['rol...
llama_model_loader: - kv  33:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  34:               general.quantization_version u32              = 2
llama_model_loader: - kv  35:                      quantize.imatrix.file str              = /models_out/gemma-2-2b-it-GGUF/gemma-...
llama_model_loader: - kv  36:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt
llama_model_loader: - kv  37:             quantize.imatrix.entries_count i32              = 182
llama_model_loader: - kv  38:              quantize.imatrix.chunks_count i32              = 128
llama_model_loader: - type  f32:  105 tensors
llama_model_loader: - type q4_K:  156 tensors
llama_model_loader: - type q6_K:   27 tensors
llm_load_vocab: special tokens cache size = 249
llm_load_vocab: token to piece cache size = 1.6014 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = gemma2
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 256000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 2304
llm_load_print_meta: n_layer          = 26
llm_load_print_meta: n_head           = 8
llm_load_print_meta: n_head_kv        = 4
llm_load_print_meta: n_rot            = 256
llm_load_print_meta: n_swa            = 4096
llm_load_print_meta: n_embd_head_k    = 256
llm_load_print_meta: n_embd_head_v    = 256
llm_load_print_meta: n_gqa            = 2
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 9216
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 2B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 2.61 B
llm_load_print_meta: model size       = 1.59 GiB (5.21 BPW)
llm_load_print_meta: general.name     = Gemma 2 2b It
llm_load_print_meta: BOS token        = 2 '<bos>'
llm_load_print_meta: EOS token        = 1 '<eos>'
llm_load_print_meta: UNK token        = 3 '<unk>'
llm_load_print_meta: PAD token        = 0 '<pad>'
llm_load_print_meta: LF token         = 227 '<0x0A>'
llm_load_print_meta: EOT token        = 107 '<end_of_turn>'
llm_load_print_meta: max token length = 48
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060 Ti, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.26 MiB
llm_load_tensors: offloading 26 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 27/27 layers to GPU
llm_load_tensors:        CPU buffer size =   461.43 MiB
llm_load_tensors:      CUDA0 buffer size =  1623.70 MiB
..........................................................
llama_new_context_with_model: n_ctx      = 8192
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =   832.00 MiB
llama_new_context_with_model: KV self size  =  832.00 MiB, K (f16):  416.00 MiB, V (f16):  416.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     4.88 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   504.50 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    36.51 MiB
llama_new_context_with_model: graph nodes  = 1050
llama_new_context_with_model: graph splits = 2
INFO [                    init] initializing slots | tid="11744" timestamp=1723514861 n_slots=4
INFO [                    init] new slot | tid="11744" timestamp=1723514861 id_slot=0 n_ctx_slot=2048
INFO [                    init] new slot | tid="11744" timestamp=1723514861 id_slot=1 n_ctx_slot=2048
INFO [                    init] new slot | tid="11744" timestamp=1723514861 id_slot=2 n_ctx_slot=2048
INFO [                    init] new slot | tid="11744" timestamp=1723514861 id_slot=3 n_ctx_slot=2048
INFO [                    main] model loaded | tid="11744" timestamp=1723514861
INFO [                    main] chat template | tid="11744" timestamp=1723514861 chat_example="<start_of_turn>user\nYou are a helpful assistant\n\nHello<end_of_turn>\n<start_of_turn>model\nHi there<end_of_turn>\n<start_of_turn>user\nHow are you?<end_of_turn>\n<start_of_turn>model\n" built_in=true
INFO [                    main] HTTP server listening | tid="11744" timestamp=1723514861 hostname="127.0.0.1" port="11484" n_threads_http="15"
INFO [            update_slots] all slots are idle | tid="11744" timestamp=1723514861
INFO [   launch_slot_with_task] slot is processing task | tid="11744" timestamp=1723514873 id_slot=0 id_task=0
INFO [            update_slots] kv cache rm [p0, end) | tid="11744" timestamp=1723514873 id_slot=0 id_task=0 p0=0
INFO [           print_timings] prompt eval time     =     222.95 ms /    12 tokens (   18.58 ms per token,    53.82 tokens per second) | tid="11744" timestamp=1723514879 id_slot=0 id_task=0 t_prompt_processing=222.948 n_prompt_tokens_processed=12 t_token=18.579 n_tokens_second=53.82421012971635
INFO [           print_timings] generation eval time =    6485.43 ms /   660 runs   (    9.83 ms per token,   101.77 tokens per second) | tid="11744" timestamp=1723514879 id_slot=0 id_task=0 t_token_generation=6485.426 n_decoded=660 t_token=9.82640303030303 n_tokens_second=101.76663799725723
INFO [           print_timings]           total time =    6708.37 ms | tid="11744" timestamp=1723514879 id_slot=0 id_task=0 t_prompt_processing=222.948 t_token_generation=6485.426 t_total=6708.374000000001
INFO [            update_slots] slot released | tid="11744" timestamp=1723514879 id_slot=0 id_task=0 n_ctx=8192 n_past=671 n_system_tokens=0 n_cache_tokens=0 truncated=false
INFO [            update_slots] all slots are idle | tid="11744" timestamp=1723514879
INFO [      log_server_request] request | tid="21256" timestamp=1723514879 remote_addr="127.0.0.1" remote_port=65028 status=200 method="POST" path="/v1/chat/completions" params={}
INFO [            update_slots] all slots are idle | tid="11744" timestamp=1723514879

For the cli:

> .\llama-cli -m gemma-2-2b-it-Q4_K_M.gguf --threads 16 -ngl 27 --mlock --port 11484 --top_k 40 --repeat_penalty 1.1 --min_p 0.05 --top_p 0.95 --prompt-cache-all -cb -np 4 --batch-size 512 -cnv
Log start
main: build = 3578 (1f67436c)
main: built with MSVC 19.29.30154.0 for x64
main: seed  = 1723514926
llama_model_loader: loaded meta data with 39 key-value pairs and 288 tensors from gemma-2-2b-it-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Gemma 2 2b It
llama_model_loader: - kv   3:                           general.finetune str              = it
llama_model_loader: - kv   4:                           general.basename str              = gemma-2
llama_model_loader: - kv   5:                         general.size_label str              = 2B
llama_model_loader: - kv   6:                            general.license str              = gemma
llama_model_loader: - kv   7:                               general.tags arr[str,2]       = ["conversational", "text-generation"]
llama_model_loader: - kv   8:                      gemma2.context_length u32              = 8192
llama_model_loader: - kv   9:                    gemma2.embedding_length u32              = 2304
llama_model_loader: - kv  10:                         gemma2.block_count u32              = 26
llama_model_loader: - kv  11:                 gemma2.feed_forward_length u32              = 9216
llama_model_loader: - kv  12:                gemma2.attention.head_count u32              = 8
llama_model_loader: - kv  13:             gemma2.attention.head_count_kv u32              = 4
llama_model_loader: - kv  14:    gemma2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  15:                gemma2.attention.key_length u32              = 256
llama_model_loader: - kv  16:              gemma2.attention.value_length u32              = 256
llama_model_loader: - kv  17:                          general.file_type u32              = 15
llama_model_loader: - kv  18:              gemma2.attn_logit_softcapping f32              = 50.000000
llama_model_loader: - kv  19:             gemma2.final_logit_softcapping f32              = 30.000000
llama_model_loader: - kv  20:            gemma2.attention.sliding_window u32              = 4096
llama_model_loader: - kv  21:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  22:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  23:                      tokenizer.ggml.tokens arr[str,256000]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  24:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  26:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  27:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  28:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  30:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  31:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {{ bos_token }}{% if messages[0]['rol...
llama_model_loader: - kv  33:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  34:               general.quantization_version u32              = 2
llama_model_loader: - kv  35:                      quantize.imatrix.file str              = /models_out/gemma-2-2b-it-GGUF/gemma-...
llama_model_loader: - kv  36:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt
llama_model_loader: - kv  37:             quantize.imatrix.entries_count i32              = 182
llama_model_loader: - kv  38:              quantize.imatrix.chunks_count i32              = 128
llama_model_loader: - type  f32:  105 tensors
llama_model_loader: - type q4_K:  156 tensors
llama_model_loader: - type q6_K:   27 tensors
llm_load_vocab: special tokens cache size = 249
llm_load_vocab: token to piece cache size = 1.6014 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = gemma2
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 256000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 2304
llm_load_print_meta: n_layer          = 26
llm_load_print_meta: n_head           = 8
llm_load_print_meta: n_head_kv        = 4
llm_load_print_meta: n_rot            = 256
llm_load_print_meta: n_swa            = 4096
llm_load_print_meta: n_embd_head_k    = 256
llm_load_print_meta: n_embd_head_v    = 256
llm_load_print_meta: n_gqa            = 2
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 9216
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 2B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 2.61 B
llm_load_print_meta: model size       = 1.59 GiB (5.21 BPW)
llm_load_print_meta: general.name     = Gemma 2 2b It
llm_load_print_meta: BOS token        = 2 '<bos>'
llm_load_print_meta: EOS token        = 1 '<eos>'
llm_load_print_meta: UNK token        = 3 '<unk>'
llm_load_print_meta: PAD token        = 0 '<pad>'
llm_load_print_meta: LF token         = 227 '<0x0A>'
llm_load_print_meta: EOT token        = 107 '<end_of_turn>'
llm_load_print_meta: max token length = 48
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060 Ti, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.26 MiB
llm_load_tensors: offloading 26 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 27/27 layers to GPU
llm_load_tensors:        CPU buffer size =   461.43 MiB
llm_load_tensors:      CUDA0 buffer size =  1623.70 MiB
..........................................................
llama_new_context_with_model: n_ctx      = 8192
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =   832.00 MiB
llama_new_context_with_model: KV self size  =  832.00 MiB, K (f16):  416.00 MiB, V (f16):  416.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     3.91 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   504.50 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    36.51 MiB
llama_new_context_with_model: graph nodes  = 1050
llama_new_context_with_model: graph splits = 2
main: chat template example: <start_of_turn>user
You are a helpful assistant

Hello<end_of_turn>
<start_of_turn>model
Hi there<end_of_turn>
<start_of_turn>user
How are you?<end_of_turn>
<start_of_turn>model


system_info: n_threads = 16 / 16 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
main: interactive mode on.
sampling:
        repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000
        top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 8192, n_batch = 512, n_predict = -1, n_keep = 1


== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to the AI.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.


> write a story
The old lighthouse keeper, Elias Thorne, squinted at the swirling grey sky. He'd seen countless storms in his seventy years, but this one felt different. The wind shrieked like a banshee, and waves crashed against the rocks with a ferocious fury. His heart echoed the rhythm of the storm, a deep, unsettling thump.

He climbed to the lantern room, the familiar creak of the wooden stairs a familiar comfort in his old bones. As he lit the giant lamp, a single beam sliced through the swirling darkness, illuminating the churning sea below. A flicker on the horizon caught his eye – a ship, tossed like a toy by the storm's relentless waves.

Elias watched with increasing concern. The ship was small, barely a shadow in the storm's grasp. He remembered the story his grandfather told him - a legend of the Sea Serpent, a creature so vast it could swallow ships whole. The whispers had always been dismissed as folklore, but now…

The fog seemed to thicken, swirling into something like a living entity. Elias saw shapes shifting within it – dark wings, sharp teeth, scales glinting with an unnatural light. Fear gripped him, constricting his chest like a vise.  But then, he saw the ship. Its sails were ripped to shreds, its bow broken against a monstrous wave. It was going down.

Elias felt a surge of duty, a primal instinct that had been dormant for years. The legend whispered of the Sea Serpent's vulnerability to the lighthouse beam – the light would pierce through the fog and deter it. He grabbed his oilskin coat, its worn fabric familiar against his calloused hands, and rushed to the top of the lighthouse tower.

With a heavy heart, he switched on the powerful lamp, the giant beam piercing through the swirling mist. A deafening roar echoed across the waves as the Sea Serpent lunged towards him, its eyes blazing with rage. The light seemed to illuminate something within the creature – a flicker of fear in its depths.  The monster faltered, momentarily caught between fury and terror.

Elias watched, his heart hammering against his ribs, as the Sea Serpent slowly turned away from the lighthouse, retreating back into the swirling fog. He could barely hear it anymore, only the sound of the waves crashing on the rocks below.

A few moments later, a lone boat appeared, battered but afloat.  Elias rushed down the stairs and watched in relief as the survivors scrambled onto the shore. They looked at him with gratitude, eyes wide with awe.

He knew he wasn't a hero. He was just Elias Thorne, a keeper of the light. But tonight, he had faced something ancient and monstrous, and lived to tell the tale.  The legend might be real after all. The Sea Serpent wouldn't be back for a while – at least not until he lit that lighthouse once more.




>

llama_print_timings:        load time =    1034.62 ms
llama_print_timings:      sample time =     665.55 ms /   591 runs   (    1.13 ms per token,   887.99 tokens per second)
llama_print_timings: prompt eval time =    2607.47 ms /    12 tokens (  217.29 ms per token,     4.60 tokens per second)
llama_print_timings:        eval time =    4569.65 ms /   590 runs   (    7.75 ms per token,   129.11 tokens per second)
llama_print_timings:       total time =    8817.04 ms /   602 tokens

[JohannesGaessler]

I am unable to reproduce this issue using LLaMA 3 q4_0, an RTX 3090, and Linux. I noticed that -np 4 causes an incorrect default context size but this should not be causing a slowdown.

[fairydreaming]

@JohannesGaessler I tried to reproduce the problem on Linux with 4090 using exactly the same options and here are the results:

llama-cli:

llama_print_timings:        load time =     404.27 ms
llama_print_timings:      sample time =     588.40 ms /   678 runs   (    0.87 ms per token,  1152.28 tokens per second)
llama_print_timings: prompt eval time =    2326.22 ms /    12 tokens (  193.85 ms per token,     5.16 tokens per second)
llama_print_timings:        eval time =    2637.09 ms /   677 runs   (    3.90 ms per token,   256.72 tokens per second)
llama_print_timings:       total time =    6388.78 ms /   689 tokens

llama-server:

INFO [           print_timings] prompt eval time     =      20.14 ms /    20 tokens (    1.01 ms per token,   992.95 tokens per second) | tid="140180616544256" timestamp=1723553140 id_slot=0 id_task=0 t_prompt_processing=20.142 n_prompt_tokens_processed=20 t_token=1.0070999999999999 n_tokens_second=992.950054612253
INFO [           print_timings] generation eval time =    3174.59 ms /   646 runs   (    4.91 ms per token,   203.49 tokens per second) | tid="140180616544256" timestamp=1723553140 id_slot=0 id_task=0 t_token_generation=3174.587 n_decoded=646 t_token=4.914221362229102 n_tokens_second=203.49103678683244
INFO [           print_timings]           total time =    3194.73 ms | tid="140180616544256" timestamp=1723553140 id_slot=0 id_task=0 t_prompt_processing=20.142 t_token_generation=3174.587 t_total=3194.729

So indeed it looks like there is an observable difference in generation performance (llama-cli: 256 t/s, llama-server: 203 t/s).

I played with the options and when I removed --repeat_penalty 1.1 from the server command line the token generation rate increased to 244 t/s:

INFO [           print_timings] prompt eval time     =       7.16 ms /    12 tokens (    0.60 ms per token,  1676.45 tokens per second) | tid="140182255595520" timestamp=1723555361 id_slot=0 id_task=1029 t_prompt_processing=7.158 n_prompt_tokens_processed=12 t_token=0.5965 n_tokens_second=1676.4459346186086
INFO [           print_timings] generation eval time =    2096.70 ms /   512 runs   (    4.10 ms per token,   244.19 tokens per second) | tid="140182255595520" timestamp=1723555361 id_slot=0 id_task=1029 t_token_generation=2096.7 n_decoded=512 t_token=4.0951171875 n_tokens_second=244.19325606906094
INFO [           print_timings]           total time =    2103.86 ms | tid="140182255595520" timestamp=1723555361 id_slot=0 id_task=1029 t_prompt_processing=7.158 t_token_generation=2096.7 t_total=2103.8579999999997

I removed all the other options to make sure. First llama-cli without repeat penalty:

$ ./llama-cli -m ~/projects/llama.cpp/models/gemma-2-2b-it-Q4_K_M.gguf --threads 1 -ngl 27 -f ../prompts/prompt-gemma-2.txt
...
llama_print_timings:        load time =     364.44 ms
llama_print_timings:      sample time =      52.63 ms /   692 runs   (    0.08 ms per token, 13148.89 tokens per second)
llama_print_timings: prompt eval time =      13.86 ms /    12 tokens (    1.15 ms per token,   866.05 tokens per second)
llama_print_timings:        eval time =    2676.49 ms /   691 runs   (    3.87 ms per token,   258.17 tokens per second)
llama_print_timings:       total time =    2837.59 ms /   703 tokens

Then llama-cli with repeat penalty:

$ ./llama-cli -m ~/projects/llama.cpp/models/gemma-2-2b-it-Q4_K_M.gguf --threads 1 -ngl 27 --repeat_penalty 1.1 -f ../prompts/prompt-gemma-2.txt
...
llama_print_timings:        load time =     361.23 ms
llama_print_timings:      sample time =     682.26 ms /   800 runs   (    0.85 ms per token,  1172.57 tokens per second)
llama_print_timings: prompt eval time =      13.92 ms /    12 tokens (    1.16 ms per token,   862.38 tokens per second)
llama_print_timings:        eval time =    3104.04 ms /   799 runs   (    3.88 ms per token,   257.41 tokens per second)
llama_print_timings:       total time =    3908.98 ms /   811 tokens

llama-server without repeat penalty:

$ ./llama-server -m ~/projects/llama.cpp/models/gemma-2-2b-it-Q4_K_M.gguf --threads 1 -ngl 27 --port 11484 --host 192.168.18.6
...
INFO [           print_timings] prompt eval time     =      22.92 ms /    12 tokens (    1.91 ms per token,   523.56 tokens per second) | tid="139791553105920" timestamp=1723555912 id_slot=0 id_task=0 t_prompt_processing=22.92 n_prompt_tokens_processed=12 t_token=1.9100000000000001 n_tokens_second=523.5602094240837
INFO [           print_timings] generation eval time =    2100.82 ms /   512 runs   (    4.10 ms per token,   243.71 tokens per second) | tid="139791553105920" timestamp=1723555912 id_slot=0 id_task=0 t_token_generation=2100.815 n_decoded=512 t_token=4.103154296875 n_tokens_second=243.71493920216676
INFO [           print_timings]           total time =    2123.74 ms | tid="139791553105920" timestamp=1723555912 id_slot=0 id_task=0 t_prompt_processing=22.92 t_token_generation=2100.815 t_total=2123.735

llama-server with repeat penalty:

$ ./llama-server -m ~/projects/llama.cpp/models/gemma-2-2b-it-Q4_K_M.gguf --threads 1 -ngl 27 --repeat_penalty 1.1 --port 11484 --host 192.168.18.6
...
INFO [           print_timings] prompt eval time     =      16.11 ms /    12 tokens (    1.34 ms per token,   744.93 tokens per second) | tid="139969563238400" timestamp=1723555991 id_slot=0 id_task=0 t_prompt_processing=16.109 n_prompt_tokens_processed=12 t_token=1.3424166666666668 n_tokens_second=744.9251970947917
INFO [           print_timings] generation eval time =    2497.39 ms /   512 runs   (    4.88 ms per token,   205.01 tokens per second) | tid="139969563238400" timestamp=1723555991 id_slot=0 id_task=0 t_token_generation=2497.389 n_decoded=512 t_token=4.877712890625 n_tokens_second=205.01411674352693
INFO [           print_timings]           total time =    2513.50 ms | tid="139969563238400" timestamp=1723555991 id_slot=0 id_task=0 t_prompt_processing=16.109 t_token_generation=2497.389 t_total=2513.498

It doesn't happen (the difference in performance is negligible) when using CPU, but with CUDA I see a significant difference when using --repeat-penalty option in the llama-server. Also even without --repeat-penalty the server is consistently slightly slower (244 t/s) compared to cli (258 t/s).

I checked all of this on current master.

TLDR: generation t/s is indeed slower in llama-server compared to llama-cli, especially when using --repeat-penalty option

[fairydreaming]

I think I solved the mystery - at least partially. Instead of comparing llama_print_timings() output with request logs of the server I added llama_print_timings() in the server to the SIGINT handler. After processing the request I pressed ctrl+c and the server printed:

...
INFO [           print_timings] prompt eval time     =      23.09 ms /    12 tokens (    1.92 ms per token,   519.80 tokens per second) | tid="140485971464192" timestamp=1723570150 id_slot=0 id_task=0 t_prompt_processing=23.086 n_prompt_tokens_processed=12 t_token=1.9238333333333333 n_tokens_second=519.7955470848133
INFO [           print_timings] generation eval time =    2569.91 ms /   531 runs   (    4.84 ms per token,   206.62 tokens per second) | tid="140485971464192" timestamp=1723570150 id_slot=0 id_task=0 t_token_generation=2569.91 n_decoded=531 t_token=4.839755178907721 n_tokens_second=206.622021782864
INFO [           print_timings]           total time =    2593.00 ms | tid="140485971464192" timestamp=1723570150 id_slot=0 id_task=0 t_prompt_processing=23.086 t_token_generation=2569.91 t_total=2592.9959999999996
...
^C
llama_print_timings:        load time =     350.26 ms
llama_print_timings:      sample time =     457.10 ms /   531 runs   (    0.86 ms per token,  1161.66 tokens per second)
llama_print_timings: prompt eval time =      21.43 ms /    12 tokens (    1.79 ms per token,   559.83 tokens per second)
llama_print_timings:        eval time =    2051.39 ms /   530 runs   (    3.87 ms per token,   258.36 tokens per second)
llama_print_timings:       total time =  348520.11 ms /   542 tokens

so the performance degradation is not visible when comparing llama_print_timings() output of llama-cli and llama-server. I guess n_tokens_second values from request logs are simply calculated differently.

Now the question is which values ​​can we trust more?

@phly95 As for the second problem

it takes a very long time (for my use-case) to start writing)

I didn't notice anything like that when doing my experiments. Are you sure it's not a problem with your LLM frontend?

[JohannesGaessler]

When I tried to reproduce the issue I was not looking at the server log but instead the timings returned by a HTTP request I sent via curl. Those values were consistent with what llama-cli reported.

[fairydreaming]

When I tried to reproduce the issue I was not looking at the server log but instead the timings returned by a HTTP request I sent via curl. Those values were consistent with what llama-cli reported.

@JohannesGaessler From what I see they are exactly the same values:

logs:

...
INFO [           print_timings] prompt eval time     =      15.73 ms /    12 tokens (    1.31 ms per token,   762.68 tokens per second) | tid="139750370267136" timestamp=1723575901 id_slot=0 id_task=0 t_prompt_processing=15.734 n_prompt_tokens_processed=12 t_token=1.3111666666666666 n_tokens_second=762.6795474768019
INFO [           print_timings] generation eval time =    2465.50 ms /   512 runs   (    4.82 ms per token,   207.67 tokens per second) | tid="139750370267136" timestamp=1723575901 id_slot=0 id_task=0 t_token_generation=2465.502 n_decoded=512 t_token=4.81543359375 n_tokens_second=207.6656194154375
INFO [           print_timings]           total time =    2481.24 ms | tid="139750370267136" timestamp=1723575901 id_slot=0 id_task=0 t_prompt_processing=15.734 t_token_generation=2465.502 t_total=2481.236
...

server response:

...
"timings":{"prompt_n":12,"prompt_ms":15.734,"prompt_per_token_ms":1.3111666666666666,"prompt_per_second":762.6795474768019,"predicted_n":512,"predicted_ms":2465.502,"predicted_per_token_ms":4.81543359375,"predicted_per_second":207.6656194154375}
...

llama_print_timings output:

llama_print_timings:        load time =     353.07 ms
llama_print_timings:      sample time =     434.95 ms /   512 runs   (    0.85 ms per token,  1177.15 tokens per second)
llama_print_timings: prompt eval time =      14.13 ms /    12 tokens (    1.18 ms per token,   849.26 tokens per second)
llama_print_timings:        eval time =    1974.29 ms /   511 runs   (    3.86 ms per token,   258.83 tokens per second)
llama_print_timings:       total time =    9495.81 ms /   523 tokens

print_timings logs are consistent with timings in server response, but they are both inconsistent with llama_print_timings output.

Did you test it with --repeat_penalty 1.1 option?

[fairydreaming]

It looks like llama_print_timings() uses only time spent between the start of llama_decode_internal() call and getting the decoder output (for example by calling llama_get_logits()) to calculate t/s values. It doesn't include the time spent doing other things like token sampling. So these timings show what the performance would be if there was no additional processing performed.

On the other hand, server print_timings logs (and timings in server response) use time from the start of the generation to the slot release, so it includes all the additional processing like prompt tokenization and token sampling during generation. These values directly correspond to the frequency with which tokens are generated.

To confirm this, I measured the time needed for the model to count from 1 to 1000. In llama-cli it took 27.406s, while in llama-server it took 26.468s. So it doesn't look like the server is slower.
Printed generation performance values were 224.91 t/s for llama-cli and 184.81 t/s for llama-server so they indicate falsely that the server is much slower.

@phly95 To sum up, there is no difference in performance between llama-cli and llama-server, there is only a difference in measurement of the timings. You should trust values printed by llama-server if you need real time performance values.

[YMMF007]

What happened? 发生了什么事？

When running: 运行时： .\llama-cli -m gemma-2-2b-it-Q4_K_M.gguf --threads 16 -ngl 27 --mlock --port 11484 --host 0.0.0.0 --top_k 40 --repeat_penalty 1.1 --min_p 0.05 --top_p 0.95 --prompt-cache-all -cb -np 4 --batch-size 512 -cnv

The output is blazing fast. When I sent "write a story" these are my speed stats:输出速度极快。当我发送“写一个故事”时，这些是我的速度统计数据： llama_print_timings: load time = 1027.95 msllama_print_timings：加载时间 = 1027.95 ms llama_print_timings: sample time = 683.92 ms / 618 runs ( 1.11 ms per token, 903.62 tokens per second)llama_print_timings：采样时间 = 683.92 毫秒/618 次运行（每个令牌 1.11 毫秒，每秒 903.62 个令牌） llama_print_timings: prompt eval time = 3678.86 ms / 12 tokens ( 306.57 ms per token, 3.26 tokens per second)llama_print_timings：提示评估时间 = 3678.86 毫秒 / 12 个令牌（每个令牌 306.57 毫秒，每秒 3.26 个令牌） llama_print_timings: eval time = 4744.65 ms / 617 runs ( 7.69 ms per token, 130.04 tokens per second)llama_print_timings：评估时间 = 4744.65 毫秒/617 次（每个令牌 7.69 毫秒，每秒 130.04 个令牌） llama_print_timings: total time = 15385.66 ms / 629 tokensllama_print_timings：总时间 = 15385.66 毫秒 / 629 个令牌

When I try the same with llama server:当我对 llama 服务器尝试相同的操作时： .\llama-server -m gemma-2-2b-it-Q4_K_M.gguf --threads 16 -ngl 27 --mlock --port 11484 --host 0.0.0.0 --top_k 40 --repeat_penalty 1.1 --min_p 0.05 --top_p 0.95 --prompt-cache-all -cb -np 4 --batch-size 512 -cnv

It takes 2-3 seconds before responding to my same "write a story" prompt on localhost. So while it does write fast, it generated at about 103 t/s (a good amount slower than CLI), and it takes a very long time (for my use-case) to start writing.在响应我在 localhost 上相同的“写故事”提示之前需要 2-3 秒。因此，虽然它确实写得很快，但它的生成速度约为 103 t/s（比 CLI 慢很多），并且需要很长时间（对于我的用例）才能开始编写。

If the prompt is extremely similar (up to about 1-3 tokens difference at the end) to my last prompt the next generation will start quick, otherwise, it takes 2-3 seconds, even if the prompt is very short. Only extremely short prompts like "hi" or "yo" respond instantly.如果提示与我的最后一个提示非常相似（最后最多相差 1-3 个标记），则下一代将快速开始，否则，即使提示很短，也需要 2-3 秒。只有“嗨”或“哟”等极短的提示才会立即响应。

Not quite sure why I'm getting such amazing performance with llama-cli but much worse performance when I use it in server mode.不太确定为什么我使用 llama-cli 获得如此惊人的性能，但当我在服务器模式下使用它时，性能要差得多。

Name and Version 名称和版本

.\llama-cli --version .\llama-cli --版本
version: 3578 (1f67436)版本： 3578 （ 1f67436 ）
built with MSVC 19.29.30154.0 for x64使用适用于 x64 的 MSVC 19.29.30154.0 构建

What operating system are you seeing the problem on?

您在哪个操作系统上看到问题？
Windows 11 窗户11

Relevant log output 相关日志输出

For the server:

> .\llama-server -m gemma-2-2b-it-Q4_K_M.gguf --threads 16 -ngl 27 --mlock --port 11484 --top_k 40 --repeat_penalty 1.1 --min_p 0.05 --top_p 0.95 --prompt-cache-all -cb -np 4
 --batch-size 512 -cnv
INFO [                    main] build info | tid="11744" timestamp=1723514860 build=3578 commit="1f67436c"
INFO [                    main] system info | tid="11744" timestamp=1723514860 n_threads=16 n_threads_batch=-1 total_threads=16 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | "
llama_model_loader: loaded meta data with 39 key-value pairs and 288 tensors from gemma-2-2b-it-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Gemma 2 2b It
llama_model_loader: - kv   3:                           general.finetune str              = it
llama_model_loader: - kv   4:                           general.basename str              = gemma-2
llama_model_loader: - kv   5:                         general.size_label str              = 2B
llama_model_loader: - kv   6:                            general.license str              = gemma
llama_model_loader: - kv   7:                               general.tags arr[str,2]       = ["conversational", "text-generation"]
llama_model_loader: - kv   8:                      gemma2.context_length u32              = 8192
llama_model_loader: - kv   9:                    gemma2.embedding_length u32              = 2304
llama_model_loader: - kv  10:                         gemma2.block_count u32              = 26
llama_model_loader: - kv  11:                 gemma2.feed_forward_length u32              = 9216
llama_model_loader: - kv  12:                gemma2.attention.head_count u32              = 8
llama_model_loader: - kv  13:             gemma2.attention.head_count_kv u32              = 4
llama_model_loader: - kv  14:    gemma2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  15:                gemma2.attention.key_length u32              = 256
llama_model_loader: - kv  16:              gemma2.attention.value_length u32              = 256
llama_model_loader: - kv  17:                          general.file_type u32              = 15
llama_model_loader: - kv  18:              gemma2.attn_logit_softcapping f32              = 50.000000
llama_model_loader: - kv  19:             gemma2.final_logit_softcapping f32              = 30.000000
llama_model_loader: - kv  20:            gemma2.attention.sliding_window u32              = 4096
llama_model_loader: - kv  21:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  22:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  23:                      tokenizer.ggml.tokens arr[str,256000]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  24:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  26:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  27:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  28:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  30:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  31:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {{ bos_token }}{% if messages[0]['rol...
llama_model_loader: - kv  33:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  34:               general.quantization_version u32              = 2
llama_model_loader: - kv  35:                      quantize.imatrix.file str              = /models_out/gemma-2-2b-it-GGUF/gemma-...
llama_model_loader: - kv  36:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt
llama_model_loader: - kv  37:             quantize.imatrix.entries_count i32              = 182
llama_model_loader: - kv  38:              quantize.imatrix.chunks_count i32              = 128
llama_model_loader: - type  f32:  105 tensors
llama_model_loader: - type q4_K:  156 tensors
llama_model_loader: - type q6_K:   27 tensors
llm_load_vocab: special tokens cache size = 249
llm_load_vocab: token to piece cache size = 1.6014 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = gemma2
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 256000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 2304
llm_load_print_meta: n_layer          = 26
llm_load_print_meta: n_head           = 8
llm_load_print_meta: n_head_kv        = 4
llm_load_print_meta: n_rot            = 256
llm_load_print_meta: n_swa            = 4096
llm_load_print_meta: n_embd_head_k    = 256
llm_load_print_meta: n_embd_head_v    = 256
llm_load_print_meta: n_gqa            = 2
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 9216
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 2B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 2.61 B
llm_load_print_meta: model size       = 1.59 GiB (5.21 BPW)
llm_load_print_meta: general.name     = Gemma 2 2b It
llm_load_print_meta: BOS token        = 2 '<bos>'
llm_load_print_meta: EOS token        = 1 '<eos>'
llm_load_print_meta: UNK token        = 3 '<unk>'
llm_load_print_meta: PAD token        = 0 '<pad>'
llm_load_print_meta: LF token         = 227 '<0x0A>'
llm_load_print_meta: EOT token        = 107 '<end_of_turn>'
llm_load_print_meta: max token length = 48
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060 Ti, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.26 MiB
llm_load_tensors: offloading 26 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 27/27 layers to GPU
llm_load_tensors:        CPU buffer size =   461.43 MiB
llm_load_tensors:      CUDA0 buffer size =  1623.70 MiB
..........................................................
llama_new_context_with_model: n_ctx      = 8192
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =   832.00 MiB
llama_new_context_with_model: KV self size  =  832.00 MiB, K (f16):  416.00 MiB, V (f16):  416.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     4.88 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   504.50 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    36.51 MiB
llama_new_context_with_model: graph nodes  = 1050
llama_new_context_with_model: graph splits = 2
INFO [                    init] initializing slots | tid="11744" timestamp=1723514861 n_slots=4
INFO [                    init] new slot | tid="11744" timestamp=1723514861 id_slot=0 n_ctx_slot=2048
INFO [                    init] new slot | tid="11744" timestamp=1723514861 id_slot=1 n_ctx_slot=2048
INFO [                    init] new slot | tid="11744" timestamp=1723514861 id_slot=2 n_ctx_slot=2048
INFO [                    init] new slot | tid="11744" timestamp=1723514861 id_slot=3 n_ctx_slot=2048
INFO [                    main] model loaded | tid="11744" timestamp=1723514861
INFO [                    main] chat template | tid="11744" timestamp=1723514861 chat_example="<start_of_turn>user\nYou are a helpful assistant\n\nHello<end_of_turn>\n<start_of_turn>model\nHi there<end_of_turn>\n<start_of_turn>user\nHow are you?<end_of_turn>\n<start_of_turn>model\n" built_in=true
INFO [                    main] HTTP server listening | tid="11744" timestamp=1723514861 hostname="127.0.0.1" port="11484" n_threads_http="15"
INFO [            update_slots] all slots are idle | tid="11744" timestamp=1723514861
INFO [   launch_slot_with_task] slot is processing task | tid="11744" timestamp=1723514873 id_slot=0 id_task=0
INFO [            update_slots] kv cache rm [p0, end) | tid="11744" timestamp=1723514873 id_slot=0 id_task=0 p0=0
INFO [           print_timings] prompt eval time     =     222.95 ms /    12 tokens (   18.58 ms per token,    53.82 tokens per second) | tid="11744" timestamp=1723514879 id_slot=0 id_task=0 t_prompt_processing=222.948 n_prompt_tokens_processed=12 t_token=18.579 n_tokens_second=53.82421012971635
INFO [           print_timings] generation eval time =    6485.43 ms /   660 runs   (    9.83 ms per token,   101.77 tokens per second) | tid="11744" timestamp=1723514879 id_slot=0 id_task=0 t_token_generation=6485.426 n_decoded=660 t_token=9.82640303030303 n_tokens_second=101.76663799725723
INFO [           print_timings]           total time =    6708.37 ms | tid="11744" timestamp=1723514879 id_slot=0 id_task=0 t_prompt_processing=222.948 t_token_generation=6485.426 t_total=6708.374000000001
INFO [            update_slots] slot released | tid="11744" timestamp=1723514879 id_slot=0 id_task=0 n_ctx=8192 n_past=671 n_system_tokens=0 n_cache_tokens=0 truncated=false
INFO [            update_slots] all slots are idle | tid="11744" timestamp=1723514879
INFO [      log_server_request] request | tid="21256" timestamp=1723514879 remote_addr="127.0.0.1" remote_port=65028 status=200 method="POST" path="/v1/chat/completions" params={}
INFO [            update_slots] all slots are idle | tid="11744" timestamp=1723514879

For the cli:

> .\llama-cli -m gemma-2-2b-it-Q4_K_M.gguf --threads 16 -ngl 27 --mlock --port 11484 --top_k 40 --repeat_penalty 1.1 --min_p 0.05 --top_p 0.95 --prompt-cache-all -cb -np 4 --batch-size 512 -cnv
Log start
main: build = 3578 (1f67436c)
main: built with MSVC 19.29.30154.0 for x64
main: seed  = 1723514926
llama_model_loader: loaded meta data with 39 key-value pairs and 288 tensors from gemma-2-2b-it-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Gemma 2 2b It
llama_model_loader: - kv   3:                           general.finetune str              = it
llama_model_loader: - kv   4:                           general.basename str              = gemma-2
llama_model_loader: - kv   5:                         general.size_label str              = 2B
llama_model_loader: - kv   6:                            general.license str              = gemma
llama_model_loader: - kv   7:                               general.tags arr[str,2]       = ["conversational", "text-generation"]
llama_model_loader: - kv   8:                      gemma2.context_length u32              = 8192
llama_model_loader: - kv   9:                    gemma2.embedding_length u32              = 2304
llama_model_loader: - kv  10:                         gemma2.block_count u32              = 26
llama_model_loader: - kv  11:                 gemma2.feed_forward_length u32              = 9216
llama_model_loader: - kv  12:                gemma2.attention.head_count u32              = 8
llama_model_loader: - kv  13:             gemma2.attention.head_count_kv u32              = 4
llama_model_loader: - kv  14:    gemma2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  15:                gemma2.attention.key_length u32              = 256
llama_model_loader: - kv  16:              gemma2.attention.value_length u32              = 256
llama_model_loader: - kv  17:                          general.file_type u32              = 15
llama_model_loader: - kv  18:              gemma2.attn_logit_softcapping f32              = 50.000000
llama_model_loader: - kv  19:             gemma2.final_logit_softcapping f32              = 30.000000
llama_model_loader: - kv  20:            gemma2.attention.sliding_window u32              = 4096
llama_model_loader: - kv  21:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  22:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  23:                      tokenizer.ggml.tokens arr[str,256000]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  24:                      tokenizer.ggml.scores arr[f32,256000]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,256000]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  26:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  27:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  28:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  30:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  31:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {{ bos_token }}{% if messages[0]['rol...
llama_model_loader: - kv  33:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  34:               general.quantization_version u32              = 2
llama_model_loader: - kv  35:                      quantize.imatrix.file str              = /models_out/gemma-2-2b-it-GGUF/gemma-...
llama_model_loader: - kv  36:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt
llama_model_loader: - kv  37:             quantize.imatrix.entries_count i32              = 182
llama_model_loader: - kv  38:              quantize.imatrix.chunks_count i32              = 128
llama_model_loader: - type  f32:  105 tensors
llama_model_loader: - type q4_K:  156 tensors
llama_model_loader: - type q6_K:   27 tensors
llm_load_vocab: special tokens cache size = 249
llm_load_vocab: token to piece cache size = 1.6014 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = gemma2
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 256000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 2304
llm_load_print_meta: n_layer          = 26
llm_load_print_meta: n_head           = 8
llm_load_print_meta: n_head_kv        = 4
llm_load_print_meta: n_rot            = 256
llm_load_print_meta: n_swa            = 4096
llm_load_print_meta: n_embd_head_k    = 256
llm_load_print_meta: n_embd_head_v    = 256
llm_load_print_meta: n_gqa            = 2
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 9216
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 2B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 2.61 B
llm_load_print_meta: model size       = 1.59 GiB (5.21 BPW)
llm_load_print_meta: general.name     = Gemma 2 2b It
llm_load_print_meta: BOS token        = 2 '<bos>'
llm_load_print_meta: EOS token        = 1 '<eos>'
llm_load_print_meta: UNK token        = 3 '<unk>'
llm_load_print_meta: PAD token        = 0 '<pad>'
llm_load_print_meta: LF token         = 227 '<0x0A>'
llm_load_print_meta: EOT token        = 107 '<end_of_turn>'
llm_load_print_meta: max token length = 48
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060 Ti, compute capability 8.6, VMM: yes
llm_load_tensors: ggml ctx size =    0.26 MiB
llm_load_tensors: offloading 26 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 27/27 layers to GPU
llm_load_tensors:        CPU buffer size =   461.43 MiB
llm_load_tensors:      CUDA0 buffer size =  1623.70 MiB
..........................................................
llama_new_context_with_model: n_ctx      = 8192
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =   832.00 MiB
llama_new_context_with_model: KV self size  =  832.00 MiB, K (f16):  416.00 MiB, V (f16):  416.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     3.91 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   504.50 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    36.51 MiB
llama_new_context_with_model: graph nodes  = 1050
llama_new_context_with_model: graph splits = 2
main: chat template example: <start_of_turn>user
You are a helpful assistant

Hello<end_of_turn>
<start_of_turn>model
Hi there<end_of_turn>
<start_of_turn>user
How are you?<end_of_turn>
<start_of_turn>model


system_info: n_threads = 16 / 16 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
main: interactive mode on.
sampling:
        repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000
        top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 8192, n_batch = 512, n_predict = -1, n_keep = 1


== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to the AI.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.


> write a story
The old lighthouse keeper, Elias Thorne, squinted at the swirling grey sky. He'd seen countless storms in his seventy years, but this one felt different. The wind shrieked like a banshee, and waves crashed against the rocks with a ferocious fury. His heart echoed the rhythm of the storm, a deep, unsettling thump.

He climbed to the lantern room, the familiar creak of the wooden stairs a familiar comfort in his old bones. As he lit the giant lamp, a single beam sliced through the swirling darkness, illuminating the churning sea below. A flicker on the horizon caught his eye – a ship, tossed like a toy by the storm's relentless waves.

Elias watched with increasing concern. The ship was small, barely a shadow in the storm's grasp. He remembered the story his grandfather told him - a legend of the Sea Serpent, a creature so vast it could swallow ships whole. The whispers had always been dismissed as folklore, but now…

The fog seemed to thicken, swirling into something like a living entity. Elias saw shapes shifting within it – dark wings, sharp teeth, scales glinting with an unnatural light. Fear gripped him, constricting his chest like a vise.  But then, he saw the ship. Its sails were ripped to shreds, its bow broken against a monstrous wave. It was going down.

Elias felt a surge of duty, a primal instinct that had been dormant for years. The legend whispered of the Sea Serpent's vulnerability to the lighthouse beam – the light would pierce through the fog and deter it. He grabbed his oilskin coat, its worn fabric familiar against his calloused hands, and rushed to the top of the lighthouse tower.

With a heavy heart, he switched on the powerful lamp, the giant beam piercing through the swirling mist. A deafening roar echoed across the waves as the Sea Serpent lunged towards him, its eyes blazing with rage. The light seemed to illuminate something within the creature – a flicker of fear in its depths.  The monster faltered, momentarily caught between fury and terror.

Elias watched, his heart hammering against his ribs, as the Sea Serpent slowly turned away from the lighthouse, retreating back into the swirling fog. He could barely hear it anymore, only the sound of the waves crashing on the rocks below.

A few moments later, a lone boat appeared, battered but afloat.  Elias rushed down the stairs and watched in relief as the survivors scrambled onto the shore. They looked at him with gratitude, eyes wide with awe.

He knew he wasn't a hero. He was just Elias Thorne, a keeper of the light. But tonight, he had faced something ancient and monstrous, and lived to tell the tale.  The legend might be real after all. The Sea Serpent wouldn't be back for a while – at least not until he lit that lighthouse once more.




>

llama_print_timings:        load time =    1034.62 ms
llama_print_timings:      sample time =     665.55 ms /   591 runs   (    1.13 ms per token,   887.99 tokens per second)
llama_print_timings: prompt eval time =    2607.47 ms /    12 tokens (  217.29 ms per token,     4.60 tokens per second)
llama_print_timings:        eval time =    4569.65 ms /   590 runs   (    7.75 ms per token,   129.11 tokens per second)
llama_print_timings:       total time =    8817.04 ms /   602 tokens

Hi,I'm trying to compile llama-server.exe on Windows using w64devkit, but I'm encountering issues when running make and cmake commands in the command prompt. Here are the errors I'm receiving:
make: *** No targets specified and no makefile found. Stop(w64devkit) or No rule to make target 'llama-server'. Stop.(make)
How did you compile llama-sever.exe in windows system?

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[sivansh11]

similar issue,
llama-cli

llama_perf_sampler_print:    sampling time =      24.62 ms /   227 runs   (    0.11 ms per token,  9220.90 tokens per second)
llama_perf_context_print:        load time =     618.92 ms
llama_perf_context_print: prompt eval time =     927.17 ms /    49 tokens (   18.92 ms per token,    52.85 tokens per second)
llama_perf_context_print:        eval time =   14085.85 ms /   178 runs   (   79.13 ms per token,    12.64 tokens per second)
llama_perf_context_print:       total time =   20699.49 ms /   227 tokens
Interrupted by user

llama-server

slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 51
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 51, n_tokens = 51, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 51, n_tokens = 51
slot      release: id  0 | task 0 | stop processing: n_past = 325, truncated = 0
slot print_timing: id  0 | task 0 |
prompt eval time =     915.13 ms /    51 tokens (   17.94 ms per token,    55.73 tokens per second)
       eval time =   23038.87 ms /   275 tokens (   83.78 ms per token,    11.94 tokens per second)
      total time =   23954.00 ms /   326 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200