Running older Pascal GPU with Llama.cpp #19248
gngglobetech
started this conversation in
General
Replies: 1 comment
|
Very nice! |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Thanks to the Llama.cpp team my Tesla P40 is still hanging on. I am able to get over 20 TPS on most smaller MOE models, and the largest one out there KIMI-K2.5 is giving 5+ TPS. Please keep support those of us who do not have RTX 30/40/50 gpu.
garfieldd@G250-G51-ZB:~/llama.cpp/build/bin$ #!/bin/bash
--- 1. Fix Library Path (Checks if .so.2 or .so.1 exists) ---
if [ -f "/usr/lib/x86_64-linux-gnu/libjemalloc.so.2" ]; then
export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so.2"
elif [ -f "/usr/lib/x86_64-linux-gnu/libjemalloc.so.1" ]; then
export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so.1"
fi
--- 2. Environment Variables ---
export LLAMA_CUDA_FORCE_DMMV=1
export LLAMA_SET_ROWS=1
--- The Revised Command ---
numactl --interleave=all ./llama-server -m /home/garfieldd/models/kimik2q2/Kimi-K2.5-UD-Q2_K_XL-00001-of-00008.gguf -ngl 999 -ot ".ffn_(up|down)_exps.=CPU" -np 1 -t 28 --numa distribute --no-mmap --mlock --fit on -fa on --jinja --special --temp 1.0 --top_p 0.95 --min_p 0.01 --top-k 20 --repeat-penalty 1.0 --kv-unified --host 192.168.1.69 --port 8088
ggml_cuda_init: found 8 CUDA devices:
Device 0: Tesla P40, compute capability 6.1, VMM: yes
Device 1: Tesla P40, compute capability 6.1, VMM: yes
Device 2: Tesla P40, compute capability 6.1, VMM: yes
Device 3: Tesla P40, compute capability 6.1, VMM: yes
Device 4: Tesla P40, compute capability 6.1, VMM: yes
Device 5: Tesla P40, compute capability 6.1, VMM: yes
Device 6: Tesla P40, compute capability 6.1, VMM: yes
Device 7: Tesla P40, compute capability 6.1, VMM: yes
build: 7840 (b0311c1) with GNU 12.4.0 for Linux x86_64
/proc/sys/kernel/numa_balancing is enabled, this has been observed to impair performance
system info: n_threads = 28, n_threads_batch = 28, total_threads = 56
system_info: n_threads = 28 (n_threads_batch = 28) / 56 | CUDA : ARCHS = 610 | FORCE_MMQ = 1 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
init: using 55 threads for HTTP server
start: binding port with default address family
main: loading model
srv load_model: loading model '/home/garfieldd/models/kimik2q2/Kimi-K2.5-UD-Q2_K_XL-00001-of-00008.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected memory use with initial parameters [MiB]:
llama_params_fit_impl: - CUDA0 (Tesla P40): 24445 total, 19703 used, 4594 free vs. target of 1024
llama_params_fit_impl: - CUDA1 (Tesla P40): 24445 total, 17609 used, 6688 free vs. target of 1024
llama_params_fit_impl: - CUDA2 (Tesla P40): 24445 total, 17568 used, 6729 free vs. target of 1024
llama_params_fit_impl: - CUDA3 (Tesla P40): 24445 total, 15422 used, 8875 free vs. target of 1024
llama_params_fit_impl: - CUDA4 (Tesla P40): 24445 total, 17559 used, 6738 free vs. target of 1024
llama_params_fit_impl: - CUDA5 (Tesla P40): 24445 total, 17554 used, 6742 free vs. target of 1024
llama_params_fit_impl: - CUDA6 (Tesla P40): 24445 total, 17554 used, 6742 free vs. target of 1024
llama_params_fit_impl: - CUDA7 (Tesla P40): 24445 total, 14208 used, 10088 free vs. target of 1024
llama_params_fit_impl: projected to use 137181 MiB of device memory vs. 194381 MiB of free device memory
llama_params_fit_impl: targets for free memory can be met on all devices, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 2.79 seconds
llama_model_load_from_file_impl: using device CUDA0 (Tesla P40) (0000:04:00.0) - 24297 MiB free
llama_model_load_from_file_impl: using device CUDA1 (Tesla P40) (0000:05:00.0) - 24297 MiB free
llama_model_load_from_file_impl: using device CUDA2 (Tesla P40) (0000:08:00.0) - 24297 MiB free
llama_model_load_from_file_impl: using device CUDA3 (Tesla P40) (0000:09:00.0) - 24297 MiB free
llama_model_load_from_file_impl: using device CUDA4 (Tesla P40) (0000:83:00.0) - 24297 MiB free
llama_model_load_from_file_impl: using device CUDA5 (Tesla P40) (0000:84:00.0) - 24297 MiB free
llama_model_load_from_file_impl: using device CUDA6 (Tesla P40) (0000:87:00.0) - 24297 MiB free
llama_model_load_from_file_impl: using device CUDA7 (Tesla P40) (0000:88:00.0) - 24297 MiB free
llama_model_loader: additional 7 GGUFs metadata loaded.
llama_model_loader: loaded meta data with 60 key-value pairs and 1096 tensors from /home/garfieldd/models/kimik2q2/Kimi-K2.5-UD-Q2_K_XL-00001-of-00008.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = deepseek2
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Kimi-K2.5
llama_model_loader: - kv 3: general.basename str = Kimi-K2.5
llama_model_loader: - kv 4: general.quantized_by str = Unsloth
llama_model_loader: - kv 5: general.size_label str = 384x14B
llama_model_loader: - kv 6: general.license str = other
llama_model_loader: - kv 7: general.license.name str = modified-mit
llama_model_loader: - kv 8: general.repo_url str = https://huggingface.co/unsloth
llama_model_loader: - kv 9: general.base_model.count u32 = 1
llama_model_loader: - kv 10: general.base_model.0.name str = Kimi K2.5
llama_model_loader: - kv 11: general.base_model.0.organization str = Moonshotai
llama_model_loader: - kv 12: general.base_model.0.repo_url str = https://huggingface.co/moonshotai/Kim...
llama_model_loader: - kv 13: general.tags arr[str,2] = ["unsloth", "image-text-to-text"]
llama_model_loader: - kv 14: deepseek2.block_count u32 = 61
llama_model_loader: - kv 15: deepseek2.context_length u32 = 262144
llama_model_loader: - kv 16: deepseek2.embedding_length u32 = 7168
llama_model_loader: - kv 17: deepseek2.feed_forward_length u32 = 18432
llama_model_loader: - kv 18: deepseek2.attention.head_count u32 = 64
llama_model_loader: - kv 19: deepseek2.attention.head_count_kv u32 = 1
llama_model_loader: - kv 20: deepseek2.rope.scaling.type str = yarn
llama_model_loader: - kv 21: deepseek2.rope.scaling.factor f32 = 64.000000
llama_model_loader: - kv 22: deepseek2.rope.scaling.original_context_length u32 = 4096
llama_model_loader: - kv 23: deepseek2.rope.scaling.yarn_beta_fast f32 = 32.000000
llama_model_loader: - kv 24: deepseek2.rope.scaling.yarn_beta_slow f32 = 1.000000
llama_model_loader: - kv 25: deepseek2.rope.freq_base f32 = 50000.000000
llama_model_loader: - kv 26: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 27: deepseek2.expert_used_count u32 = 8
llama_model_loader: - kv 28: deepseek2.expert_group_count u32 = 1
llama_model_loader: - kv 29: deepseek2.expert_group_used_count u32 = 1
llama_model_loader: - kv 30: deepseek2.expert_gating_func u32 = 2
llama_model_loader: - kv 31: deepseek2.leading_dense_block_count u32 = 1
llama_model_loader: - kv 32: deepseek2.vocab_size u32 = 163840
llama_model_loader: - kv 33: deepseek2.attention.q_lora_rank u32 = 1536
llama_model_loader: - kv 34: deepseek2.attention.kv_lora_rank u32 = 512
llama_model_loader: - kv 35: deepseek2.attention.key_length u32 = 576
llama_model_loader: - kv 36: deepseek2.attention.value_length u32 = 512
llama_model_loader: - kv 37: deepseek2.attention.key_length_mla u32 = 192
llama_model_loader: - kv 38: deepseek2.attention.value_length_mla u32 = 128
llama_model_loader: - kv 39: deepseek2.expert_feed_forward_length u32 = 2048
llama_model_loader: - kv 40: deepseek2.expert_count u32 = 384
llama_model_loader: - kv 41: deepseek2.expert_shared_count u32 = 1
llama_model_loader: - kv 42: deepseek2.expert_weights_scale f32 = 2.827000
llama_model_loader: - kv 43: deepseek2.expert_weights_norm bool = true
llama_model_loader: - kv 44: deepseek2.rope.dimension_count u32 = 64
llama_model_loader: - kv 45: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000
llama_model_loader: - kv 46: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 47: tokenizer.ggml.pre str = kimi-k2
llama_model_loader: - kv 48: tokenizer.ggml.tokens arr[str,163840] = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 49: tokenizer.ggml.token_type arr[i32,163840] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 50: tokenizer.ggml.merges arr[str,163328] = ["Ġ Ġ", "ĠĠ ĠĠ", "Ġ t", "i n",...
llama_model_loader: - kv 51: tokenizer.ggml.bos_token_id u32 = 163584
llama_model_loader: - kv 52: tokenizer.ggml.eos_token_id u32 = 163586
llama_model_loader: - kv 53: tokenizer.ggml.padding_token_id u32 = 163839
llama_model_loader: - kv 54: tokenizer.chat_template str = {%- macro render_content(msg) -%}\n ...
llama_model_loader: - kv 55: general.quantization_version u32 = 2
llama_model_loader: - kv 56: general.file_type u32 = 10
llama_model_loader: - kv 57: split.no u16 = 0
llama_model_loader: - kv 58: split.tensors.count i32 = 1096
llama_model_loader: - kv 59: split.count u16 = 8
llama_model_loader: - type f32: 365 tensors
llama_model_loader: - type f16: 3 tensors
llama_model_loader: - type q8_0: 192 tensors
llama_model_loader: - type q2_K: 120 tensors
llama_model_loader: - type q3_K: 182 tensors
llama_model_loader: - type q4_K: 223 tensors
llama_model_loader: - type q6_K: 11 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q2_K - Medium
print_info: file size = 349.01 GiB (2.92 BPW)
load: 0 unused tokens
load: printing all EOG tokens:
load: - 163586 ('<|im_end|>')
load: special tokens cache size = 256
load: token to piece cache size = 1.0606 MB
print_info: arch = deepseek2
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 262144
print_info: n_embd = 7168
print_info: n_embd_inp = 7168
print_info: n_layer = 61
print_info: n_head = 64
print_info: n_head_kv = 1
print_info: n_rot = 64
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 576
print_info: n_embd_head_v = 512
print_info: n_gqa = 64
print_info: n_embd_k_gqa = 576
print_info: n_embd_v_gqa = 512
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-05
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 18432
print_info: n_expert = 384
print_info: n_expert_used = 8
print_info: n_expert_groups = 1
print_info: n_group_used = 1
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 0
print_info: rope scaling = yarn
print_info: freq_base_train = 50000.0
print_info: freq_scale_train = 0.015625
print_info: n_ctx_orig_yarn = 4096
print_info: rope_yarn_log_mul = 1.0000
print_info: rope_finetuned = unknown
print_info: model type = 671B
print_info: model params = 1.03 T
print_info: general.name = Kimi-K2.5
print_info: n_layer_dense_lead = 1
print_info: n_lora_q = 1536
print_info: n_lora_kv = 512
print_info: n_embd_head_k_mla = 192
print_info: n_embd_head_v_mla = 128
print_info: n_ff_exp = 2048
print_info: n_expert_shared = 1
print_info: expert_weights_scale = 2.8
print_info: expert_weights_norm = 1
print_info: expert_gating_func = sigmoid
print_info: vocab type = BPE
print_info: n_vocab = 163840
print_info: n_merges = 163328
print_info: BOS token = 163584 '[BOS]'
print_info: EOS token = 163586 '<|im_end|>'
print_info: EOT token = 163586 '<|im_end|>'
print_info: PAD token = 163839 '[PAD]'
print_info: LF token = 198 'Ċ'
print_info: EOG token = 163586 '<|im_end|>'
print_info: max token length = 512
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = true)
load_tensors: offloading output layer to GPU
load_tensors: offloading 60 repeating layers to GPU
load_tensors: offloaded 62/62 layers to GPU
load_tensors: CPU model buffer size = 630.00 MiB
load_tensors: CUDA0 model buffer size = 13310.75 MiB
load_tensors: CUDA1 model buffer size = 14847.42 MiB
load_tensors: CUDA2 model buffer size = 14806.50 MiB
load_tensors: CUDA3 model buffer size = 12948.39 MiB
load_tensors: CUDA4 model buffer size = 14797.50 MiB
load_tensors: CUDA5 model buffer size = 14792.86 MiB
load_tensors: CUDA6 model buffer size = 14792.86 MiB
load_tensors: CUDA7 model buffer size = 12022.70 MiB
load_tensors: CUDA_Host model buffer size = 244440.00 MiB
....................................................................................................
common_init_result: added <|im_end|> logit bias = -inf
llama_context: constructing llama_context
llama_context: setting new yarn_attn_factor = 1.0000 (mscale == 1.0, mscale_all_dim = 1.0)
llama_context: n_seq_max = 1
llama_context: n_ctx = 262144
llama_context: n_ctx_seq = 262144
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = enabled
llama_context: kv_unified = true
llama_context: freq_base = 50000.0
llama_context: freq_scale = 0.015625
llama_context: CUDA_Host output buffer size = 0.62 MiB
llama_kv_cache: CUDA0 KV buffer size = 2304.00 MiB
llama_kv_cache: CUDA1 KV buffer size = 2304.00 MiB
llama_kv_cache: CUDA2 KV buffer size = 2304.00 MiB
llama_kv_cache: CUDA3 KV buffer size = 2016.00 MiB
llama_kv_cache: CUDA4 KV buffer size = 2304.00 MiB
llama_kv_cache: CUDA5 KV buffer size = 2304.00 MiB
llama_kv_cache: CUDA6 KV buffer size = 2304.00 MiB
llama_kv_cache: CUDA7 KV buffer size = 1728.00 MiB
llama_kv_cache: size = 17568.00 MiB (262144 cells, 61 layers, 1/1 seqs), K (f16): 17568.00 MiB, V (f16): 0.00 MiB
sched_reserve: reserving ...
sched_reserve: CUDA0 compute buffer size = 4088.75 MiB
sched_reserve: CUDA1 compute buffer size = 458.01 MiB
sched_reserve: CUDA2 compute buffer size = 458.01 MiB
sched_reserve: CUDA3 compute buffer size = 458.01 MiB
sched_reserve: CUDA4 compute buffer size = 458.01 MiB
sched_reserve: CUDA5 compute buffer size = 458.01 MiB
sched_reserve: CUDA6 compute buffer size = 458.01 MiB
sched_reserve: CUDA7 compute buffer size = 458.01 MiB
sched_reserve: CUDA_Host compute buffer size = 540.01 MiB
sched_reserve: graph nodes = 4791
sched_reserve: graph splits = 182 (with bs=512), 129 (with bs=1)
sched_reserve: reserve took 489.55 ms, sched copies = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv load_model: initializing slots, n_slots = 1
slot load_model: id 0 | task -1 | new slot, n_ctx = 262144
srv load_model: prompt cache is enabled, size limit: 8192 MiB
srv load_model: use
--cache-ram 0to disable the prompt cachesrv load_model: for more info see https://github.com//pull/16391
init: chat template, example_format: '<|im_system|>system<|im_middle|>You are a helpful assistant<|im_end|><|im_user|>user<|im_middle|>Hello<|im_end|><|im_assistant|>assistant<|im_middle|>Hi there<|im_end|><|im_user|>user<|im_middle|>How are you?<|im_end|><|im_assistant|>assistant<|im_middle|>'
srv init: init: chat template, thinking = 0
main: model loaded
main: server is listening on http://192.168.1.69:8088
main: starting the main loop...
srv update_slots: all slots are idle
srv log_server_r: done request: GET / 192.168.1.100 200
srv params_from_: Chat format: Kimi K2
slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> ?temp-ext -> dist
slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
slot update_slots: id 0 | task 0 | new prompt, n_ctx_slot = 262144, n_keep = 0, task.n_tokens = 27
slot update_slots: id 0 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_tokens = 27, batch.n_tokens = 27, progress = 1.000000
slot update_slots: id 0 | task 0 | prompt done, n_tokens = 27, batch.n_tokens = 27
slot init_sampler: id 0 | task 0 | init sampler, took 0.01 ms, tokens: text = 27, total = 27
slot print_timing: id 0 | task 0 |
prompt eval time = 2529.58 ms / 27 tokens ( 93.69 ms per token, 10.67 tokens per second)
eval time = 20520.64 ms / 120 tokens ( 171.01 ms per token, 5.85 tokens per second)
total time = 23050.22 ms / 147 tokens
slot release: id 0 | task 0 | stop processing: n_tokens = 146, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: done request: POST /v1/chat/completions 192.168.1.100 200
srv params_from_: Chat format: Kimi K2
slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, sim_best = 0.159 (> 0.100 thold), f_keep = 0.185
srv get_availabl: updating prompt cache
srv prompt_save: - saving prompt with length 146, total state size = 9.787 MiB
srv load: - looking for better prompt, base f_keep = 0.185, sim = 0.159
srv update: - cache state: 1 prompts, 9.787 MiB (limits: 8192.000 MiB, 262144 tokens, 262144 est)
srv update: - prompt 0x721fd57d75d0: 146 tokens, checkpoints: 0, 9.787 MiB
srv get_availabl: prompt cache update took 19.24 ms
slot launch_slot_: id 0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> ?temp-ext -> dist
slot launch_slot_: id 0 | task 121 | processing task, is_child = 0
slot update_slots: id 0 | task 121 | new prompt, n_ctx_slot = 262144, n_keep = 0, task.n_tokens = 170
slot update_slots: id 0 | task 121 | n_tokens = 27, memory_seq_rm [27, end)
slot update_slots: id 0 | task 121 | prompt processing progress, n_tokens = 170, batch.n_tokens = 143, progress = 1.000000
slot update_slots: id 0 | task 121 | prompt done, n_tokens = 170, batch.n_tokens = 143
slot init_sampler: id 0 | task 121 | init sampler, took 0.05 ms, tokens: text = 170, total = 170
slot print_timing: id 0 | task 121 |
prompt eval time = 19121.00 ms / 143 tokens ( 133.71 ms per token, 7.48 tokens per second)
eval time = 898702.69 ms / 4912 tokens ( 182.96 ms per token, 5.47 tokens per second)
total time = 917823.69 ms / 5055 tokens
slot release: id 0 | task 121 | stop processing: n_tokens = 5081, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: done request: POST /v1/chat/completions 192.168.1.100 200
^Csrv operator(): operator(): cleaning up before exit...
llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |
llama_memory_breakdown_print: | - CUDA0 (P40) | 24445 = 4015 + ( 19703 = 13310 + 2304 + 4088) + 726 |
llama_memory_breakdown_print: | - CUDA1 (P40) | 24445 = 6145 + ( 17609 = 14847 + 2304 + 458) + 690 |
llama_memory_breakdown_print: | - CUDA2 (P40) | 24445 = 6185 + ( 17568 = 14806 + 2304 + 458) + 691 |
llama_memory_breakdown_print: | - CUDA3 (P40) | 24445 = 8331 + ( 15422 = 12948 + 2016 + 458) + 691 |
llama_memory_breakdown_print: | - CUDA4 (P40) | 24445 = 6195 + ( 17559 = 14797 + 2304 + 458) + 690 |
llama_memory_breakdown_print: | - CUDA5 (P40) | 24445 = 6199 + ( 17554 = 14792 + 2304 + 458) + 691 |
llama_memory_breakdown_print: | - CUDA6 (P40) | 24445 = 6199 + ( 17554 = 14792 + 2304 + 458) + 691 |
llama_memory_breakdown_print: | - CUDA7 (P40) | 24445 = 9545 + ( 14208 = 12022 + 1728 + 458) + 691 |
llama_memory_breakdown_print: | - Host | 245610 = 245070 + 0 + 540 |
garfieldd@G250-G51-ZB:~/llama.cpp/build/bin$
All reactions