Repository navigation
llama.cpp tuned for NVIDIA Tesla P100 (Pascal, sm_60): v5.42 rebased onto upstream
32bdc3a89 (+102 upstream commits, ggml 1.0.0). Outputs, draft acceptance
and the kernels launched are identical to v5.42 on every production model; speed equal. Other GPUs keep upstream's code paths.
What changed since v5.42
- Rebase onto upstream
32bdc3a89. Notable upstream changes for this fork: ggml-org#29801 prunes unused ggml ops; ggml-org#32078 removes the
unfused GDN path (the series already ran the fused one, so nothing changes at runtime); ggml-org#30565 removes the context's warmup flag;
ggml-org#32016 renames the MTP state-source call; ggml-org#26048 renames the fused matmul op tests; ggml-org#28413 adds FP8 matmul and flash-attention paths
(not used by the production models); ggml-org#32102 merges the qkvz tensors for newly converted GGUFs. - RMS_NORM stays F32 on the series' norm kernels. Upstream's BF16 RMS_NORM (ggml-org#32135) is not carried over to them; the CUDA backend
reports RMS_NORM and its fusions as F32-only, so BF16 graphs fall back instead of reaching a kernel that can't run them. - The expert cache no longer checks the removed warmup flag (ggml-org#30565); Flash-Next first-turn outputs and acceptance are unchanged.
- ggml-org#31927 (MTP draft rows on the device) stays off. A port of the series' draft position pairing into that path was tried and
parked: under-sm tensorthe rows land on the meta device, which has no events, so the path never enables; the gain on P100 would
be about 0.1 %.
Gate (2x Tesla P100 16 GB, vs v5.42)
test-backend-opspasses (35 op groups;MUL_MAT3090,MUL_MAT_ID1145/1145 also without the activation cache and with
GGML_CUDA_MMQ_MOE_TPE=0,MUL_MAT_FUSION1643,FLASH_ATTN_EXT4322).- KL divergence against full-F32 references (prefill, decode, 4/6/8-token verify, long sets): 30/30 identical on all six production
models; opt-outs 7/7 identical. - Long-context block 6/6 pass, allocator audit 12/12 pass,
--parallel 2KLD identical and servers OK. - Kernel census: no new, missing or recounted kernels on all 10 production entries, and equal launch totals.
- Serving on 10 entries: all 27 outputs, 8 vision answers, chat turns and draft acceptance identical; Flash-Next outputs identical
(4 mirrored loads), MTP acceptance and decode speed equal. - Speed: llama-bench within 0.4 % on all rows but two single-run outliers (qwen36 decode at 16K depth -0.8 %, ornith prefill -0.6 %);
serving speed equal. - Soaks clean: gemma4 786 requests, Swift 127, Flash-Next 265, no failures.
Performance vs stock llama.cpp
Serving decode with production settings, as measured for v5.28-v5.32: gemma4 1.68x, Swift 27B 1.68x, qwen36 / ornith 1.52x,
glimmer 1.65x, nemotron 1.33x, Qwen 3.8 Flash-Next 1.71x; v5.33 added ~8 % to batched gemma4 decode, v5.35 2-5 % to
qwen36 / ornith, v5.37 28 % to nemotron prefill, v5.38 7-10 % to qwen36 / ornith and ~3 % to Swift decode. Step-by-step numbers:
docs/p100/PERFORMANCE.md.
Build
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=60 -DCMAKE_BUILD_TYPE=Release
cmake --build build -jDeveloped with CUDA 12.4. Two cards: -sm tensor and GGML_CUDA_P2P=1. See P100-SERIES.md.
Credits
231 code patches: wispoffates (184, with Claude Code), Kaden / Kmic-68 (33), drrros (5), Keoian (4), Kane (2), lich-77 (2), wrj97 (1);
qr0n, buun and Keoian credited as co-authors where their code was ported. Details:
CREDITS.md.
Contributions, including AI-written PRs, are welcome: CONTRIBUTING.md.