Skip to content

Releases

Initial Release - v0.1.0

Pre-release

Choose a tag to compare

@WyvernTKC WyvernTKC released this 02 Sep 01:04

===============================================================================
llama.cpp - sm-tensor-4xv100
Tensor-parallel split and MoE expert offload for 4x Tesla V100

Hardware: 4x Tesla V100-SXM2-32GB (sm_70), 128 GiB VRAM, capped at 150 W.
Merged through upstream b10731. 34 commits ahead, ~2850 lines, 25 files.

  1. TENSOR PARALLEL SPLIT (-sm tensor)

Splits weights by column across all four GPUs instead of assigning whole layers
to each device. Layer assignment leaves three GPUs idle for most of a token;
column-parallel keeps all four busy on every operation.
Implemented as a meta backend that presents one logical device and fans each op
out to the four real ones.

  • Tensors that cannot be cut four ways are mirrored instead: the MLA latent
    KV, a lone KV head under MQA, PLE caches, and lm_head. Mirroring lm_head
    makes logits come out replicated, so every sampler reduction works
    unchanged.
  • Expert splits take their boundary from ffn_down's quantization block, so
    gate/up/down agree. Mismatched boundaries would have each device's
    down-projection consume a slice it never produced.
  • A one-shot P2P all-reduce handles small messages alongside NCCL.
    Worth +25% decode at 4 GPUs.

Required at 4 GPUs: GGML_CUDA_ALLREDUCE=nccl

  1. EXPERT OFFLOAD TO SYSTEM RAM (-ncmoe N)

Runs MoE models substantially larger than 128 GiB of VRAM by keeping the first
N layers' experts in host RAM and streaming them per token.

  • Only the experts a batch actually routes to are copied
    (GGML_META_PARTIAL_COPY=1). It self-limits to batches of 32 or fewer, so
    decode benefits and prefill is left alone.
  • Decode cost per offloaded layer is set by bytes, not architecture:
    2.24 ms/token/layer for qwen4exp against 3.56 for deepseek4, a ratio of
    1.59 where their expert bytes per layer differ by 1.60.
  • Use --load-mode none whenever experts are host-resident. mmap costs 2.8x
    prefill.
  • Staging slots buy nothing today, because decode reads expert ids back to
    the host once per layer and that sync caps pipeline depth at 1. Four slots
    measured identical to twelve, so spend the VRAM on resident layers instead.
    (A consequence of a current limitation, not a permanent property.)
    scripts/estimate-ncmoe.py reads real per-layer expert sizes from a GGUF and
    suggests a starting -ncmoe. It does not model staging slots or the true compute
    buffer, so treat its answer as a floor to test, not a value to trust.

  1. V100 PERFORMANCE

A 156 GiB model on 128 GiB of VRAM went from 15.7 to 42.2 tokens/s decode.

Measured on build 10784, -c 16384, 2000-token prompt, 1000-token generation,
even tensor split:
model -ncmoe prefill t/s decode t/s TTFT VRAM/GPU

qwen4exp Q6_K 2 993.0 41.64 2.03s 30.6 GiB
qwen4exp Q6_K 8 775.3 27.31 2.59s 27.9 GiB
qwen4exp Q6_K 16 607.7 17.90 3.30s 23.2 GiB
qwen4exp Q6_K 99 328.3 7.86 6.11s 6.5 GiB
deepseek4 Q4_K 16 410.7 11.33 4.90s 30.9 GiB
deepseek4 Q4_K 20 410.2 9.76 4.90s 27.5 GiB
deepseek4 Q4_K 26 360.9 8.00 5.57s 22.5 GiB
deepseek4 Q4_K 99 268.1 5.32 7.47s 8.2 GiB
Prefill degrades far more gently than decode - offloading spends decode first.
Where the gains came from:
MoE prefill MMQ fallback 12x prefill. 256-expert mul_mat_id was taking
the cuBLAS per-expert loop on Volta.
Meta backend offload_op +39-47% prefill. -sm tensor never streamed
offloaded experts at all.
Graph decomposition cache +18-35% across five MoE architectures. Keyed on
split-graph uid plus a mirror generation.
P2P all-reduce +25% decode at 4 GPUs for small messages.
FA register spill (hs 256) -27.5% kernel, +5.2% prefill. 8 warps with Q in
shared memory.
Used-experts-only copy +31% decode, batch-gated so prefill is untouched.

CORRECTNESS

Greedy text diverges by design between a split and a single GPU, so matching
output is the wrong test. Validated by perplexity instead: agrees to 0.013%.
Two corruption bugs fixed this cycle, both in quantized MoE paths:

  • MMQ read past the end of each staged expert range. Safe only when a slot
    happened to hold the same quant kind, otherwise F16 scales decoded to
    infinity and output collapsed into one repeated character. Fixed by padding
    each staged range by one chunk.
  • An unbounded used-experts scan in ggml_backend_sched_compute_splits walked
    off the end of its bitset when an MoE layer had no routed tokens. This one
    is upstream's, introduced by 5682a37 (#15346) and still present at b10731.
    A report is drafted on the upstream-reports branch.
    Note what the architecture sweep does NOT prove: it is F16-only, so a green
    sweep says nothing about MMQ, quantized padding or expert geometry. Both bugs
    above lived in paths no test reaches.

QUICK START

set CUDA_VISIBLE_DEVICES=0,1,2,3
set GGML_CUDA_ALLREDUCE=nccl
set GGML_META_PARTIAL_COPY=1
set GGML_META_STAGE_SLOTS=4
llama-server -m model.gguf -sm tensor -ngl 999 -ncmoe 2 ^
--load-mode none --fit off --flash-attn on ^
-c 16384 -b 4096 -ub 4096 -np 1
Push -ncmoe as low as VRAM allows; it is the single biggest lever. Raise it if
startup runs out of memory. Full detail in README.md.