Repository navigation
Initial Release - v0.1.0
Pre-release===============================================================================
llama.cpp - sm-tensor-4xv100
Tensor-parallel split and MoE expert offload for 4x Tesla V100
Hardware: 4x Tesla V100-SXM2-32GB (sm_70), 128 GiB VRAM, capped at 150 W.
Merged through upstream b10731. 34 commits ahead, ~2850 lines, 25 files.
- TENSOR PARALLEL SPLIT (-sm tensor)
Splits weights by column across all four GPUs instead of assigning whole layers
to each device. Layer assignment leaves three GPUs idle for most of a token;
column-parallel keeps all four busy on every operation.
Implemented as a meta backend that presents one logical device and fans each op
out to the four real ones.
- Tensors that cannot be cut four ways are mirrored instead: the MLA latent
KV, a lone KV head under MQA, PLE caches, and lm_head. Mirroring lm_head
makes logits come out replicated, so every sampler reduction works
unchanged. - Expert splits take their boundary from ffn_down's quantization block, so
gate/up/down agree. Mismatched boundaries would have each device's
down-projection consume a slice it never produced. - A one-shot P2P all-reduce handles small messages alongside NCCL.
Worth +25% decode at 4 GPUs.
Required at 4 GPUs: GGML_CUDA_ALLREDUCE=nccl
- EXPERT OFFLOAD TO SYSTEM RAM (-ncmoe N)
Runs MoE models substantially larger than 128 GiB of VRAM by keeping the first
N layers' experts in host RAM and streaming them per token.
- Only the experts a batch actually routes to are copied
(GGML_META_PARTIAL_COPY=1). It self-limits to batches of 32 or fewer, so
decode benefits and prefill is left alone. - Decode cost per offloaded layer is set by bytes, not architecture:
2.24 ms/token/layer for qwen4exp against 3.56 for deepseek4, a ratio of
1.59 where their expert bytes per layer differ by 1.60. - Use --load-mode none whenever experts are host-resident. mmap costs 2.8x
prefill. - Staging slots buy nothing today, because decode reads expert ids back to
the host once per layer and that sync caps pipeline depth at 1. Four slots
measured identical to twelve, so spend the VRAM on resident layers instead.
(A consequence of a current limitation, not a permanent property.)
scripts/estimate-ncmoe.py reads real per-layer expert sizes from a GGUF and
suggests a starting -ncmoe. It does not model staging slots or the true compute
buffer, so treat its answer as a floor to test, not a value to trust.
- V100 PERFORMANCE
A 156 GiB model on 128 GiB of VRAM went from 15.7 to 42.2 tokens/s decode.
Measured on build 10784, -c 16384, 2000-token prompt, 1000-token generation,
even tensor split:
model -ncmoe prefill t/s decode t/s TTFT VRAM/GPU
qwen4exp Q6_K 2 993.0 41.64 2.03s 30.6 GiB
qwen4exp Q6_K 8 775.3 27.31 2.59s 27.9 GiB
qwen4exp Q6_K 16 607.7 17.90 3.30s 23.2 GiB
qwen4exp Q6_K 99 328.3 7.86 6.11s 6.5 GiB
deepseek4 Q4_K 16 410.7 11.33 4.90s 30.9 GiB
deepseek4 Q4_K 20 410.2 9.76 4.90s 27.5 GiB
deepseek4 Q4_K 26 360.9 8.00 5.57s 22.5 GiB
deepseek4 Q4_K 99 268.1 5.32 7.47s 8.2 GiB
Prefill degrades far more gently than decode - offloading spends decode first.
Where the gains came from:
MoE prefill MMQ fallback 12x prefill. 256-expert mul_mat_id was taking
the cuBLAS per-expert loop on Volta.
Meta backend offload_op +39-47% prefill. -sm tensor never streamed
offloaded experts at all.
Graph decomposition cache +18-35% across five MoE architectures. Keyed on
split-graph uid plus a mirror generation.
P2P all-reduce +25% decode at 4 GPUs for small messages.
FA register spill (hs 256) -27.5% kernel, +5.2% prefill. 8 warps with Q in
shared memory.
Used-experts-only copy +31% decode, batch-gated so prefill is untouched.
CORRECTNESS
Greedy text diverges by design between a split and a single GPU, so matching
output is the wrong test. Validated by perplexity instead: agrees to 0.013%.
Two corruption bugs fixed this cycle, both in quantized MoE paths:
- MMQ read past the end of each staged expert range. Safe only when a slot
happened to hold the same quant kind, otherwise F16 scales decoded to
infinity and output collapsed into one repeated character. Fixed by padding
each staged range by one chunk. - An unbounded used-experts scan in ggml_backend_sched_compute_splits walked
off the end of its bitset when an MoE layer had no routed tokens. This one
is upstream's, introduced by 5682a37 (#15346) and still present at b10731.
A report is drafted on the upstream-reports branch.
Note what the architecture sweep does NOT prove: it is F16-only, so a green
sweep says nothing about MMQ, quantized padding or expert geometry. Both bugs
above lived in paths no test reaches.
QUICK START
set CUDA_VISIBLE_DEVICES=0,1,2,3
set GGML_CUDA_ALLREDUCE=nccl
set GGML_META_PARTIAL_COPY=1
set GGML_META_STAGE_SLOTS=4
llama-server -m model.gguf -sm tensor -ngl 999 -ncmoe 2 ^
--load-mode none --fit off --flash-attn on ^
-c 16384 -b 4096 -ub 4096 -np 1
Push -ncmoe as low as VRAM allows; it is the single biggest lever. Raise it if
startup runs out of memory. Full detail in README.md.