Skip to content

p100-series-v5.43

Latest

Choose a tag to compare

@wispoffates wispoffates released this 15 Nov 22:23

llama.cpp tuned for NVIDIA Tesla P100 (Pascal, sm_60): v5.42 rebased onto upstream
32bdc3a89 (+102 upstream commits, ggml 1.0.0). Outputs, draft acceptance
and the kernels launched are identical to v5.42 on every production model; speed equal. Other GPUs keep upstream's code paths.

What changed since v5.42

  • Rebase onto upstream 32bdc3a89. Notable upstream changes for this fork: ggml-org#29801 prunes unused ggml ops; ggml-org#32078 removes the
    unfused GDN path (the series already ran the fused one, so nothing changes at runtime); ggml-org#30565 removes the context's warmup flag;
    ggml-org#32016 renames the MTP state-source call; ggml-org#26048 renames the fused matmul op tests; ggml-org#28413 adds FP8 matmul and flash-attention paths
    (not used by the production models); ggml-org#32102 merges the qkvz tensors for newly converted GGUFs.
  • RMS_NORM stays F32 on the series' norm kernels. Upstream's BF16 RMS_NORM (ggml-org#32135) is not carried over to them; the CUDA backend
    reports RMS_NORM and its fusions as F32-only, so BF16 graphs fall back instead of reaching a kernel that can't run them.
  • The expert cache no longer checks the removed warmup flag (ggml-org#30565); Flash-Next first-turn outputs and acceptance are unchanged.
  • ggml-org#31927 (MTP draft rows on the device) stays off. A port of the series' draft position pairing into that path was tried and
    parked: under -sm tensor the rows land on the meta device, which has no events, so the path never enables; the gain on P100 would
    be about 0.1 %.

Gate (2x Tesla P100 16 GB, vs v5.42)

  • test-backend-ops passes (35 op groups; MUL_MAT 3090, MUL_MAT_ID 1145/1145 also without the activation cache and with
    GGML_CUDA_MMQ_MOE_TPE=0, MUL_MAT_FUSION 1643, FLASH_ATTN_EXT 4322).
  • KL divergence against full-F32 references (prefill, decode, 4/6/8-token verify, long sets): 30/30 identical on all six production
    models; opt-outs 7/7 identical.
  • Long-context block 6/6 pass, allocator audit 12/12 pass, --parallel 2 KLD identical and servers OK.
  • Kernel census: no new, missing or recounted kernels on all 10 production entries, and equal launch totals.
  • Serving on 10 entries: all 27 outputs, 8 vision answers, chat turns and draft acceptance identical; Flash-Next outputs identical
    (4 mirrored loads), MTP acceptance and decode speed equal.
  • Speed: llama-bench within 0.4 % on all rows but two single-run outliers (qwen36 decode at 16K depth -0.8 %, ornith prefill -0.6 %);
    serving speed equal.
  • Soaks clean: gemma4 786 requests, Swift 127, Flash-Next 265, no failures.

Performance vs stock llama.cpp

Serving decode with production settings, as measured for v5.28-v5.32: gemma4 1.68x, Swift 27B 1.68x, qwen36 / ornith 1.52x,
glimmer 1.65x, nemotron 1.33x, Qwen 3.8 Flash-Next 1.71x; v5.33 added ~8 % to batched gemma4 decode, v5.35 2-5 % to
qwen36 / ornith, v5.37 28 % to nemotron prefill, v5.38 7-10 % to qwen36 / ornith and ~3 % to Swift decode. Step-by-step numbers:
docs/p100/PERFORMANCE.md.

Build

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=60 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Developed with CUDA 12.4. Two cards: -sm tensor and GGML_CUDA_P2P=1. See P100-SERIES.md.

Credits

231 code patches: wispoffates (184, with Claude Code), Kaden / Kmic-68 (33), drrros (5), Keoian (4), Kane (2), lich-77 (2), wrj97 (1);
qr0n, buun and Keoian credited as co-authors where their code was ported. Details:
CREDITS.md.
Contributions, including AI-written PRs, are welcome: CONTRIBUTING.md.