Skip to content

p100-series-v5.41

Choose a tag to compare

View all tags
@wispoffates wispoffates released this 08 Nov 08:30
· 341 commits to p100-series since this release
p100-series-v5.41
c6f10c7

llama.cpp tuned for NVIDIA Tesla P100 (Pascal, sm_60): v5.40 rebased onto upstream
cf659f240 (+48 upstream commits). Outputs identical to v5.40 on every
production model, same kernels launched; serving speed unchanged. Other GPUs keep upstream's code paths.

What changed since v5.40

  • Rebase onto upstream cf659f240. Notable upstream changes for this fork: ggml-org#31047 lays out MoE MMQ batches per expert and sizes
    the tile by tokens per expert on every GPU; ggml-org#30137 makes context checkpoints shared immutable buffers; ggml-org#31683 / ggml-org#31733 / ggml-org#31693
    refactor the model graphs (the qwen35 gated-delta-net fusions still fire); ggml-org#31005 adds a NaN guard to the MoE top-k kernel.
  • Kane's per-expert MMQ tile dropped. It is upstream now (ggml-org#31047), so the series carries only the GGML_CUDA_MMQ_MOE_TPE=0
    switch. Our tile GEMM for MoE prefill builds its own expert bounds instead of reusing MMQ's.
  • Checkpoint pool on shared checkpoints. The pre-faulted checkpoint pool now holds ggml-org#30137's common_state_data buffers, and the
    current checkpoint a slot keeps is claimed on that slot.

Gate (2x Tesla P100 16 GB, vs v5.40)

  • test-backend-ops passes (35 op groups; MUL_MAT_ID 1036/1036, also with GGML_CUDA_MMQ_MOE_TPE=0).
  • KL divergence against full-F32 references (prefill, decode, 4/6/8-token verify, long sets): identical on all six production models,
    nemotron included despite ggml-org#31047; opt-outs 7/7 identical.
  • Long-context block 6/6 pass, allocator audit 12/12 pass, --parallel 2 servers OK.
  • Kernel census: no new, missing or recounted kernels on any of the 10 production entries.
  • Serving on 10 entries: all 27 outputs, vision answers, chat turns and draft acceptance identical; Flash-Next outputs identical
    (4 mirrored loads), MTP acceptance equal.
  • Speed: prefill within +-0.3 % (nemotron +0.8 %, ggml-org#31047). llama-bench decode reads lower on gemma4 (-0.8 %) and qwen36 (-0.6 %),
    confirmed with 4 mirrored runs per build; it is not host work (perf shows slightly less), not graph reuse and not a register change,
    and the cause is not yet identified. Serving decode with production settings shows no loss on any entry.
  • Soaks clean: gemma4 781 requests, Swift 130, Flash-Next 258, no failures.

Performance vs stock llama.cpp

Serving decode with production settings, as measured for v5.28-v5.32: gemma4 1.68x, Swift 27B 1.68x, qwen36 / ornith 1.52x,
glimmer 1.65x, nemotron 1.33x, Qwen 3.8 Flash-Next 1.71x; v5.33 added ~8 % to batched gemma4 decode, v5.35 2-5 % to
qwen36 / ornith, v5.37 28 % to nemotron prefill, v5.38 7-10 % to qwen36 / ornith and ~3 % to Swift decode. Step-by-step numbers:
docs/p100/PERFORMANCE.md.

Build

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=60 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Developed with CUDA 12.4. Two cards: -sm tensor and GGML_CUDA_P2P=1. See P100-SERIES.md.

Credits

227 code patches: wispoffates (180, with Claude Code), Kaden / Kmic-68 (33), drrros (5), Keoian (4), Kane (2), lich-77 (2), wrj97 (1);
qr0n, buun and Keoian credited as co-authors where their code was ported. Kane's per-expert MMQ tile (v5.37-v5.40) is now upstream's
ggml-org#31047. Details: CREDITS.md.
Contributions, including AI-written PRs, are welcome: CONTRIBUTING.md.

Assets 2
Loading