Speed benchmarks of various LLMs #1544
magikRUKKOLA
started this conversation in
Show and tell
Replies: 6 comments 10 replies
-
|
I have a simple question, eypc 9654 96 core seems cheaper than 10*3090... I guess, the speed will be close or surpass the 'full offload layer' layer one??? |
Beta Was this translation helpful? Give feedback.
4 replies
-
DDR5-4800 MT/s ? Well, lets compare. https://huggingface.co/ubergarm/Qwen3.5-397B-A17B-GGUF/discussions/4
That's 56C QYFS and 8 channel DDR5-4800 MT/s and a single GPU (5090?). Suppose EPYC have x 1.5 performance. Then we will have about 500 tps prefill and 42 tps decode. I doubt it will beat 1300 tps prefill and 60 tps decode. That said, the maintenance of the 12 sticks of DDR5 is much easier lol. The tower of RTX 3090 with Noctua Redux'es from each side of every GPU does look pretty funny. |
Beta Was this translation helpful? Give feedback.
-
|
no clue but seems nice only if the ddr5 price drop back to last year otherwise 😂 |
Beta Was this translation helpful? Give feedback.
-
Yeah, last time I bought DDR5-4800 MT/s ECC price was about 250 EUR per 32GB stick. Its either going to be double QYFS or double EPYC (not sure which one). |
Beta Was this translation helpful? Give feedback.
-
|
I was the guy think 2*48g is enough at that time 🤷🏿 what a waste... |
Beta Was this translation helpful? Give feedback.
-
|
GLM5 smol-IQ2_KL 10 x RTX 3090 (the first GPU is x16); split mode: layer;
log Details
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Qwen3.5-27B-GGUF/IQ5_KS Two RTX 3090 EVGA x16; The dynamic TDP is used to keep the GPUs lower than 80C (hence the zig-zags).
logs Detailsgraph:
layer:
|
Beta Was this translation helpful? Give feedback.
1 reply
-
|
That drops so quickly... |
Beta Was this translation helpful? Give feedback.
-
|
AesSedai/Kimi-K2.5/Q4_X *its basically for the illustrative purposes. The previous benchmark ( https://huggingface.co/AesSedai/Kimi-K2.5-GGUF/discussions/7#69c9120d934491df89a986b1 ) with 8x3090 partial offload came with the result: That seems to be very slow. So the below shows that running a hybrid inference just with a head offloaded works much better. So here is how becomes actually more-or-less usable: hardware: 2xEPYC 7B13; NPS=0; DDR4-2933 ECC (2666 overclocked; timings are auto; command rate=1T); single RTX 3090 EVGA
command: DetailsGGML_CUDA_NO_PINNED=1 numactl --interleave=all /opt/ik_llama.cpp/ik_llama.cpp/build/bin/llama-sweep-bench \
--warmup-batch \
-f /opt/ik_llama.cpp/wiki.test.raw \
--model /opt/AesSedai/Kimi-K2.5/Q4_X/Kimi-K2.5-Q4_X-00001-of-00014.gguf \
--alias AesSedai/Kimi-K2.5-GGUF \
-b $((1 * 1024)) -ub $((1 * 1024)) \
--ctx-size $((128 * 1024)) \
--mlock \
--temp 0.0 --top-k 0 --top-p 1.0 \
-ctk f16 \
-ctv f16 \
-khad -vhad \
-amb 256 \
-muge \
--merge-qkv \
--split-mode layer \
--cpu-moe \
--graph-reduce-type f16 \
--threads 128 \
--gpu-layers 99 \
--host 0.0.0.0 \
--port 8080 \
--log-enable \
--logdir /var/log/ \
--jinja \
--chat-template-file /opt/AesSedai/Kimi-K2.5/Q4_X/chat_template.jinja \
--special \
--verbose-prompt --verbosity 2 \
--prompt-cache "$HOME/.cache/ik_llama.cpp/prompt-cache.bin" --prompt-cache-all \
--slot-save-path "$HOME/.cache/ik_llama.cpp/slot.bin" \
--lookup-cache-dynamic "$HOME/.cache/ik_llama.cpp/slot.bin" \
--keep -1 \
--slot-prompt-similarity 0.35 \
--metrics \
-cuda fusion=1 |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
bartowski Qwen3.6-27b Q8_0 hardware: 2 x RTX 3090. 
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Kimi-K2.6-THIREUS-1.825bpw recipe: Detailsppl: note: 13 GPUs, 200W TDP limited, overclocked; 1kb batch for the graph with 96k ctx and 4kb batch for the layer with 256k ctx 
logs: DetailsFile: /root/utils/bench-kimi-k26-1.825bpw-200w-q5_0-kv-graph-1kb.log File: /root/utils/bench-kimi-k26-200w-q5_0-kv.1.825bpw-layer-4kb.log FYI above is the comparison of Kimi-K2.6 1.825bpw quant graph vs layer split mode. |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
Do you have all of the settings used? Did you run out of vram for the longer context with graph? |
Beta Was this translation helpful? Give feedback.
-
Yes, I do have all the settings etc. Which ones do you need? run command: export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7,8,9,10,11,12"
/opt/ik_llama.cpp/ik_llama.cpp/build/bin/llama-server \
-f /opt/ik_llama.cpp/wiki.test.raw \
--model /opt/THIREUS/Kimi-K2.6/smol-IQ1-KT-mist.bin/Kimi-K2.6-THIREUS-mist.bin-SPECIAL_TENSOR-00001-of-01097.gguf \
--alias THIREUS/Kimi-K2.6-smol-IQ1-KT-mist.bin \
--gpu-fit-margin 12,1600 \
-b $((2 * 512)) -ub $((2 * 512)) \
--ctx-size $((96 * 1024)) \
--fit \
--temp 0.0 --top-k 0 --top-p 1.0 \
-ctk q5_0 \
-ctv q8_0 \
-amb 16 \
-mea 16 \
-wgt 1 \
-muge \
--mlock \
--merge-qkv \
--split-mode graph \
--graph-reduce-type f16 \
--threads 1 \
-sas \
--gpu-layers 99 \
--host 0.0.0.0 \
--port 8080 \
--log-enable \
--logdir /var/log/ \
--jinja \
--special \
--prompt-cache "$HOME/.cache/ik_llama.cpp/prompt-cache.bin" --prompt-cache-all \
--slot-save-path "$HOME/.cache/ik_llama.cpp/slot.bin" \
--lookup-cache-dynamic "$HOME/.cache/ik_llama.cpp/slot.bin" \
--keep -1 \
--slot-prompt-similarity 0.35 \
--metrics \
-cuda fusion=1
Well, yes. 13 GPUs can't hold the full context. 96k ctx is the largest ctx with So its about 2.3GB + 1.3GB per GPU. So 3.6GB per GPU goes for the KV ( |
Beta Was this translation helpful? Give feedback.
-
|
hardware: two 6-slimSAS slot PEX 88096 -- one is working in x8x8 mode at PCIe 4.0 x16 slot of the mobo and provides six x8 GPUs. Another one, contrary, provides for the three x16 GPUs. Also got two risers and one bifurcation x16 to two x8 slimSAS slots/x8 GPUs. DetailsWhy this way and not another? Well, its ... a complicated story which should start with a fact that the heatsink at one of the PEX 88096 became so hot it desoldered itself into two parts. So I had to glue the Noctua coolers on top of the turbo-fan of the PEX 88096 boards. That altered the way I can connect the other GPUs. etc. |
Beta Was this translation helpful? Give feedback.
-
|
Do you want to check it out yourself? I can provide you the remote SSH access so you can use the machine for the test if the machine is free (that is, if Iwan doesn't work there). |
Beta Was this translation helpful? Give feedback.
-
|
So basically ... the full 256k ctx [EDIT]:
|
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
// I went through the repos of
vllm,sglangetc. (not the first time) and I was unable to find any info related to the speed benchmarks. So creating the separate discussion to keep the interesting performance data here.Qwen3.5 397B IQ4_KSS
hardware: RTX 3090 FE x10 TDP
350W420W, DDR4 2666 MT/s ECC, AMD THREADRIPPER PRO 3995WX; ASROCK WRX80 Creator 2.0UPDATE: Moved the GPUs to the GIGABYTE MC62-G40. That allowed to get the config with one x16 gpu and the rest with the x8. That improved the prefill (see the green graph below). So the GIGABYTE motherboards are recommended, not ASROCK.
UPDATE2: Installed the
PLX 88096switch to have more x16 GPUs.hybrid inference (single GPU, layer) / full offload (layer vs graph)
llama-sweep-bench run command
Details
/usr/share/doc/nvidia-cuda-toolkit/examples/bin/x86_64/linux/release/p2pBandwidthLatencyTest --p2p_read --sm_copyDetails
updated for x16 and x8 rest:
Details
updated for five x16 and x8 rest via
PLX 88096switch:Details
File: /root/p2p-11gpu.log
logs
Details
File: qwen3.5-397b-iq4_kss-full-offload-graph.log
File: qwen3.5-397b-iq4_kss-full-offload-layer.log
File: qwen3.5-397b-iq4_kss-hybrid-layer.log
File: /root/utils/bench-10gpu-mist-3k_b-5x(x16)-5x(x8).log
generate_svgs.sh (a dodgy script to generate svg out of
llama-sweep-bench.sh)Details
Beta Was this translation helpful? Give feedback.
All reactions