Skip to content

p100-series-v5.42

Latest
Latest

Choose a tag to compare

View all tags
@wispoffates wispoffates released this 11 Nov 08:54
p100-series-v5.42
cca2fca

llama.cpp tuned for NVIDIA Tesla P100 (Pascal, sm_60): v5.41 rebased onto upstream
cd4ed902e (+64 upstream commits). Outputs and draft acceptance identical
to v5.41 on every production model; speed equal or slightly up. Other GPUs keep upstream's code paths.

What changed since v5.41

  • Rebase onto upstream cd4ed902e. Notable upstream changes for this fork: ggml-org#31084 makes the CUDA radix top-k pick tied values
    deterministically (one kernel fewer per sampled step on the Swift MTP entries); ggml-org#31419 stops uploading graph inputs that no computed
    node reads; ggml-org#32001 / ggml-org#32018 rework CUDA kernel launch setup and the MMVQ quantize call; ggml-org#31854 keeps F16 matmul inputs in F32 on the CPU
    backend; ggml-org#31865 consolidates clip into mtmd; ggml-org#31185 adds external per-layer embeddings (--ple).
  • ggml-org#31927 (MTP draft rows on the device) is switched off in the series for now. That path skips the series' draft position pairing,
    so Swift and Flash-Next drafting run exactly as in v5.41. Porting the pairing into it is planned for a later release.
  • The PDL check returns early below Hopper, before the mutex and hash lookup that every CUDA kernel launch would otherwise pay on
    the P100s; the cached q8_1 activation is invalidated in the CUDA context constructor (both adapted to ggml-org#32001).
  • ggml-org#31419 also applies to the user-input pre-upload (ggml_backend_sched_upload_user_inputs), as it does in compute_splits.

Gate (2x Tesla P100 16 GB, vs v5.41)

  • test-backend-ops passes (35 op groups; MUL_MAT_ID 1072/1072, also without the activation cache and with GGML_CUDA_MMQ_MOE_TPE=0).
  • KL divergence against full-F32 references (prefill, decode, 4/6/8-token verify, long sets): 27/27 identical on all six production
    models; opt-outs 7/7 identical.
  • Long-context block 6/6 pass, allocator audit 12/12 pass, --parallel 2 KLD identical and servers OK.
  • Kernel census: no new, missing or recounted kernels on 8 of the 10 production entries; on the two Swift MTP entries ggml-org#31084's top-k
    gather replaces the old gather and its counter reset.
  • Serving on 10 entries: all 27 outputs, chat turns and draft acceptance identical; Flash-Next outputs identical (4 mirrored loads),
    MTP acceptance equal. Vision answers identical where the encoder runs on the GPU; the two entries that keep it on the CPU
    (--no-mmproj-offload) answer slightly differently because of ggml-org#31854.
  • Speed: llama-bench -0.04 % to +1.65 % (gemma4 decode at 16K depth); the v5.41 decode dip did not recur. Batched gemma4 decode +0.3-0.5 %.
  • Soaks clean: gemma4 792 requests, Swift 132, Flash-Next 262, no failures.

Performance vs stock llama.cpp

Serving decode with production settings, as measured for v5.28-v5.32: gemma4 1.68x, Swift 27B 1.68x, qwen36 / ornith 1.52x,
glimmer 1.65x, nemotron 1.33x, Qwen 3.8 Flash-Next 1.71x; v5.33 added ~8 % to batched gemma4 decode, v5.35 2-5 % to
qwen36 / ornith, v5.37 28 % to nemotron prefill, v5.38 7-10 % to qwen36 / ornith and ~3 % to Swift decode. Step-by-step numbers:
docs/p100/PERFORMANCE.md.

Build

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=60 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Developed with CUDA 12.4. Two cards: -sm tensor and GGML_CUDA_P2P=1. See P100-SERIES.md.

Credits

228 code patches: wispoffates (181, with Claude Code), Kaden / Kmic-68 (33), drrros (5), Keoian (4), Kane (2), lich-77 (2), wrj97 (1);
qr0n, buun and Keoian credited as co-authors where their code was ported. Details:
CREDITS.md.
Contributions, including AI-written PRs, are welcome: CONTRIBUTING.md.

Assets 2
Loading