Repository navigation
llama.cpp tuned for NVIDIA Tesla P100 (Pascal, sm_60): v5.43 rebased onto upstream
41d8297fe (+166 upstream commits), plus faster norms and fewer conversion
kernels on the P100s. Serving decode is 6-8 % faster on qwen36 / ornith and 3-4 % on gemma4. Every new piece has an opt-out
switch; all of them together reproduce v5.43 bit for bit. Other GPUs keep upstream's code paths.
What changed since v5.43
Rounding changes:
- Faster RMSNorm. Upstream ggml-org#29720 (float4 loads) and ggml-org#32500 are ported into the series' single-pass kernels.
- ggml-org#32500 is the q/k RMSNorm + multi-section RoPE fusion with tuned blocks, for the qwen35 family.
- The narrow blocks apply only from 256 rows per launch; on sm_60 they lose below that.
- Norm launches sized to the row.
- Norm blocks are fitted to the row width.
- The GDN gated norm runs one warp per row (prompt batches -83 %).
- The decode norm that feeds the fp16 matvecs reads float4.
Bit-identical:
- Shared-expert gate fusion (qwen36, ornith, Qwen 3.8 Flash-Next): sigmoid gate × shared expert + MoE output + residual + the next
RMSNorm now run in one launch. - Fewer f32 ↔ fp16 conversions on prompts.
- The cuBLAS fp16 input conversion is reused across matmuls of one activation.
- Gate / up results feed SWIGLU in fp16.
- The tile GEMM's prepared activation and MoE routing are reused for gate and up.
- Vision LayerNorm fusion. Upstream ggml-org#32454 (LayerNorm × w + b fusion) runs as a separate kernel, so the plain LayerNorm keeps its
v5.43 code.
Opt-outs: docs/p100/SETTINGS.md.
Gate (2x Tesla P100 16 GB, vs v5.43)
-
Op tests:
test-backend-opspasses, 55 op groups including the new fusion tests, with the new features on and off (MUL_MAT3094,
MUL_MAT_ID1165). -
Upstream merge: v5.44 with every new feature off is identical to v5.43 on all 27 KL-divergence sets, 7 opt-out checks and
Flash-Next's outputs. -
Accuracy: KL divergence against full-F32 references is equal within noise.
- Better at long gemma4 prompts (-35 % at 8K), on ornith and on nemotron prefill.
- qwen36 decode varies ±12 % with any rounding change (MoE routing sensitivity).
-
Standing blocks: long-context 6/6 pass, allocator audit 12/12 pass,
--parallel 2servers OK. -
Kernel census: every new or missing kernel is explained; up to 11 % fewer launches on qwen36 / ornith.
-
Serving decode with production settings:
Model Change qwen36 +6..8 % ornith +7..8 % gemma4 +3..4 % nemotron +1..2 % Swift / glimmer flat Batched gemma4 is +0.5 / +5 / +3 % at 1 / 2 / 3 sequences.
-
Flash-Next:
- MTP draft acceptance 0.61 -> 0.57 (rounding), with decode still about +2 %.
- About 150 MiB more VRAM per GPU for the reuse caches (
GGML_CUDA_TILE_PREP_CACHE=0returns it).
-
Soaks: gemma4 815 requests, Swift 130, Flash-Next 272, no failures.
- One Swift soak's after-load reference reply differed once and never again in 14 further soaks (5 exact replicas) and 14
co-batching probes.
- One Swift soak's after-load reference reply differed once and never again in 14 further soaks (5 exact replicas) and 14
Performance vs stock llama.cpp
Serving decode with production settings, as measured for v5.28-v5.32: gemma4 1.68x, Swift 27B 1.68x, qwen36 / ornith 1.52x,
glimmer 1.65x, nemotron 1.33x, Qwen 3.8 Flash-Next 1.71x. Since then:
| Release | Gain |
|---|---|
| v5.33 | ~8 % on batched gemma4 decode |
| v5.35 | 2-5 % on qwen36 / ornith |
| v5.37 | 28 % on nemotron prefill |
| v5.38 | 7-10 % on qwen36 / ornith, ~3 % on Swift decode |
| v5.44 | 6-8 % on qwen36 / ornith, 3-4 % on gemma4 serving decode |
Step-by-step numbers: docs/p100/PERFORMANCE.md.
Build
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=60 -DCMAKE_BUILD_TYPE=Release
cmake --build build -jDeveloped with CUDA 12.4. Two cards: -sm tensor and GGML_CUDA_P2P=1. See P100-SERIES.md.
Credits
247 code patches: wispoffates (200, with Claude Code), Kaden / Kmic-68 (33), drrros (5), Keoian (4), Kane (2), lich-77 (2), wrj97 (1);
qr0n, buun and Keoian credited as co-authors where their code was ported; the upstream norm work (ggml-org#29720, ggml-org#32500, ggml-org#32454) is ported
into the series' kernels. Details: CREDITS.md.
Contributions, including AI-written PRs, are welcome: CONTRIBUTING.md.