Skip to content

p100-series-v5.44

Latest

Choose a tag to compare

@wispoffates wispoffates released this 20 Nov 00:10
· 1 commit to p100-series since this release

llama.cpp tuned for NVIDIA Tesla P100 (Pascal, sm_60): v5.43 rebased onto upstream
41d8297fe (+166 upstream commits), plus faster norms and fewer conversion
kernels on the P100s. Serving decode is 6-8 % faster on qwen36 / ornith and 3-4 % on gemma4. Every new piece has an opt-out
switch; all of them together reproduce v5.43 bit for bit. Other GPUs keep upstream's code paths.

What changed since v5.43

Rounding changes:

  • Faster RMSNorm. Upstream ggml-org#29720 (float4 loads) and ggml-org#32500 are ported into the series' single-pass kernels.
    • ggml-org#32500 is the q/k RMSNorm + multi-section RoPE fusion with tuned blocks, for the qwen35 family.
    • The narrow blocks apply only from 256 rows per launch; on sm_60 they lose below that.
  • Norm launches sized to the row.
    • Norm blocks are fitted to the row width.
    • The GDN gated norm runs one warp per row (prompt batches -83 %).
    • The decode norm that feeds the fp16 matvecs reads float4.

Bit-identical:

  • Shared-expert gate fusion (qwen36, ornith, Qwen 3.8 Flash-Next): sigmoid gate × shared expert + MoE output + residual + the next
    RMSNorm now run in one launch.
  • Fewer f32 ↔ fp16 conversions on prompts.
    • The cuBLAS fp16 input conversion is reused across matmuls of one activation.
    • Gate / up results feed SWIGLU in fp16.
    • The tile GEMM's prepared activation and MoE routing are reused for gate and up.
  • Vision LayerNorm fusion. Upstream ggml-org#32454 (LayerNorm × w + b fusion) runs as a separate kernel, so the plain LayerNorm keeps its
    v5.43 code.

Opt-outs: docs/p100/SETTINGS.md.

Gate (2x Tesla P100 16 GB, vs v5.43)

  • Op tests: test-backend-ops passes, 55 op groups including the new fusion tests, with the new features on and off (MUL_MAT 3094,
    MUL_MAT_ID 1165).

  • Upstream merge: v5.44 with every new feature off is identical to v5.43 on all 27 KL-divergence sets, 7 opt-out checks and
    Flash-Next's outputs.

  • Accuracy: KL divergence against full-F32 references is equal within noise.

    • Better at long gemma4 prompts (-35 % at 8K), on ornith and on nemotron prefill.
    • qwen36 decode varies ±12 % with any rounding change (MoE routing sensitivity).
  • Standing blocks: long-context 6/6 pass, allocator audit 12/12 pass, --parallel 2 servers OK.

  • Kernel census: every new or missing kernel is explained; up to 11 % fewer launches on qwen36 / ornith.

  • Serving decode with production settings:

    Model Change
    qwen36 +6..8 %
    ornith +7..8 %
    gemma4 +3..4 %
    nemotron +1..2 %
    Swift / glimmer flat

    Batched gemma4 is +0.5 / +5 / +3 % at 1 / 2 / 3 sequences.

  • Flash-Next:

    • MTP draft acceptance 0.61 -> 0.57 (rounding), with decode still about +2 %.
    • About 150 MiB more VRAM per GPU for the reuse caches (GGML_CUDA_TILE_PREP_CACHE=0 returns it).
  • Soaks: gemma4 815 requests, Swift 130, Flash-Next 272, no failures.

    • One Swift soak's after-load reference reply differed once and never again in 14 further soaks (5 exact replicas) and 14
      co-batching probes.

Performance vs stock llama.cpp

Serving decode with production settings, as measured for v5.28-v5.32: gemma4 1.68x, Swift 27B 1.68x, qwen36 / ornith 1.52x,
glimmer 1.65x, nemotron 1.33x, Qwen 3.8 Flash-Next 1.71x. Since then:

Release Gain
v5.33 ~8 % on batched gemma4 decode
v5.35 2-5 % on qwen36 / ornith
v5.37 28 % on nemotron prefill
v5.38 7-10 % on qwen36 / ornith, ~3 % on Swift decode
v5.44 6-8 % on qwen36 / ornith, 3-4 % on gemma4 serving decode

Step-by-step numbers: docs/p100/PERFORMANCE.md.

Build

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=60 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Developed with CUDA 12.4. Two cards: -sm tensor and GGML_CUDA_P2P=1. See P100-SERIES.md.

Credits

247 code patches: wispoffates (200, with Claude Code), Kaden / Kmic-68 (33), drrros (5), Keoian (4), Kane (2), lich-77 (2), wrj97 (1);
qr0n, buun and Keoian credited as co-authors where their code was ported; the upstream norm work (ggml-org#29720, ggml-org#32500, ggml-org#32454) is ported
into the series' kernels. Details: CREDITS.md.
Contributions, including AI-written PRs, are welcome: CONTRIBUTING.md.