Skip to content

p100-series-v5.40

Latest
Latest

Choose a tag to compare

View all tags
@wispoffates wispoffates released this 06 Nov 11:10
p100-series-v5.40
482ff03

llama.cpp tuned for NVIDIA Tesla P100 (Pascal, sm_60): v5.39 rebased onto upstream
9c9074e14 (+37 upstream commits). Outputs identical to v5.39 on every
production model; speed unchanged. Other GPUs keep upstream's code paths.

What changed since v5.39

  • Rebase onto upstream 9c9074e14. Notable upstream change for this fork: ggml-org#31660 moves the conv-state snapshot store of every
    drafting architecture (qwen35 / qwen35moe and qwen4exp) into one shared builder whose slot order is computed in the graph
    (arange -> scale -> cast -> view) instead of read from a host input.
  • Conv-state fusion keeps firing. Without a fix, ggml-org#31660's form stopped the CUDA conv-state fusion on every drafting server (Swift
    MTP / n-gram, Flash-Next MTP): +15.5 % kernel launches on Swift. The fusion now recognises the in-graph slot order (exact op
    parameters, nodes used only by the store) and skips those three nodes on decode, so decode runs the same kernels as v5.39.
    Prefill batches run the three small index kernels (+0.7 % launches on Swift); Flash-Next launches are unchanged.
    GGML_CUDA_GDN_FUSE=0 gives the same outputs bit for bit.

Gate (2x Tesla P100 16 GB, vs v5.39)

  • test-backend-ops passes (34 op groups, including new conv-state fusion cases for the in-graph form and ARANGE).
  • KL divergence against full-F32 references (prefill, decode, 4/6/8-token verify, long sets): identical on all six production models;
    opt-outs (GGML_CUDA_GDN_FUSE=0 and others) 7/7 identical.
  • Long-context block pass, allocator audit 12/12 pass, --parallel 2 servers OK.
  • Kernel census: unchanged on gemma4, qwen36, ornith, nemotron and glimmer; Swift gains the three index kernels on prefill batches only;
    Flash-Next replaces half its per-slot copies with one set_rows, total launches equal.
  • llama-bench ABBA: every model within +-0.5 % (one noisy gemma4 decode arm aside; kernel census identical). Serving on 9 models: all
    outputs, vision answers, chat turns and draft acceptance identical; Flash-Next outputs identical (4 mirrored loads), acceptance and
    speed equal.
  • Soaks clean: gemma4 751 requests, Swift 127, Flash-Next 240, no failures.

Performance vs stock llama.cpp

Serving decode with production settings, as measured for v5.28-v5.32: gemma4 1.68x, Swift 27B 1.68x, qwen36 / ornith 1.52x,
glimmer 1.65x, nemotron 1.33x, Qwen 3.8 Flash-Next 1.71x; v5.33 added ~8 % to batched gemma4 decode, v5.35 2-5 % to
qwen36 / ornith, v5.37 28 % to nemotron prefill, v5.38 7-10 % to qwen36 / ornith and ~3 % to Swift decode. Step-by-step numbers:
docs/p100/PERFORMANCE.md.

Build

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=60 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Developed with CUDA 12.4. Two cards: -sm tensor and GGML_CUDA_P2P=1. See P100-SERIES.md.

Credits

225 code patches: wispoffates (177, with Claude Code), Kaden / Kmic-68 (33), drrros (5), Keoian (4), Kane (3), lich-77 (2), wrj97 (1);
qr0n, buun and Keoian credited as co-authors where their code was ported. Details: CREDITS.md.
Contributions, including AI-written PRs, are welcome: CONTRIBUTING.md.

Assets 2
Loading