Skip to content

engine b9e1b78 for sm_120 (glibc 2.35+)

Choose a tag to compare

View all tags
@marcospaulo marcospaulo released this 27 Sep 04:04
engine-b9e1b78
b9e1b78

A build of this fork at b9e1b78ec for NVIDIA Blackwell GeForce and workstation cards (sm_120: RTX 5080, 5070 Ti, 5090, RTX PRO 6000). It holds llama-server, llama-bench, llama-kv-mean-center and their libraries, with RUNPATH $ORIGIN. It runs on glibc 2.35 or newer: Ubuntu 22.04+, Debian 12+, Fedora 36+.

b9e1b78ec is the train engine-10 (#70), 2 commits past engine-87a3596:

  • at most 8 CUDA graphs per context, the least recently used evicted (d0f8bae): the graphs one per shape were bounded only by a 10 s eviction, so the memory they held followed the traffic. GGML_CUDA_GRAPH_MAX sets the cap (0: none); the cap is logged at load, and the most held and the largest instance as they grow;
  • test-recurrent-rollback-rounding (b9e1b78): a recurrent state rolled back holds to the rounding of its batch shapes, in the state's own type (f32, f16, q8_0), and one token early or late fails it.

It is what rig installs when engine/engine.toml pins this release. rig checks the file against the sha256 below. rig also fetches NVIDIA's CUDA 13.3 runtime from NVIDIA's own server (cuda_cudart 13.3.29, libcublas 13.5.1.27, each pinned by sha256) and unpacks it beside the binaries. A machine needs only an NVIDIA driver that supports CUDA 13; no toolkit or compiler:

curl -fsSL https://github.com/torad-labs/rig/releases/latest/download/install.sh | sh
rig up bonsai-2-27b

Using it without rig: unpack the tarball. Put libcudart.so.13, libcublas.so.13 and libcublasLt.so.13 from a CUDA 13.x install (or NVIDIA's redistributable archives) beside the binaries or on LD_LIBRARY_PATH.

How it was built: rig build --portable in nvidia/cuda:13.3.0-devel-ubuntu22.04 (glibc 2.35, gcc 11.4.0, CUDA 13.3.33), with the two steps of rig's scripts/prebuilt/Dockerfile on top (CMake 3.31.6 by sha256, ninja). It ran on a rented RTX 5090.

  • CMAKE_CUDA_ARCHITECTURES=120a
  • the AVX2 + FMA + F16C CPU baseline, not -march=native
  • FlashAttention on, CUDA graphs on
  • -ffile-prefix-map, so no build path is recorded
  • no OpenSSL: llama-server's HTTPS downloads and TLS serving are off; serve local files on loopback
  • the compiler's libgomp.so.1 bundled (GCC runtime library exception)

The highest symbol versions across the tarball's ELF files are GLIBC_2.34, GLIBCXX_3.4.30 and CXXABI_1.3.13.

Measurements: this build against engine-87a3596's on a rented RTX 5090 (driver 615.71.09), each with NVIDIA's pinned runtime (checked with ldd before any leg), Ternary Bonsai 2 27B as rig serves it (q4_0 K/V with its mean-center, f16 recurrent state), one slot at 262,144, on a 245,752-token prompt's four questions, greedy, 384 tokens each with top-5 log-probabilities:

  • bit-exact, plain and with the MTP draft: the same 1,536 tokens, log-probabilities, top-5 and draft counts;
  • without n_probs, where the top-k prefilter runs, the same tokens and drafting, plain and drafted;
  • a 245K conversation swapped out of its slot and back at 4 × 294,912 returns its first token in 1.03 s with the answer of the conversation never swapped out, token for token (|dlogprob| p99 0.027, max 0.041).

The cap's cost, on an RTX 5080 at 4 × 294,912 under four concurrent requests: peak 15,180 MiB capped against 15,228 uncapped; one request at a time, the same text at 214.6 tok/s against 216.3, the five requests after the first within 0.3 %. scripts/e2e-driver-only.sh with this tarball on an RTX 5070 Ti (Ubuntu 22.04, driver only): install, ldd -r, decode pass.

sha256 8e1de8e13e68d7d66997baa9c4edba950533de5e7e06031d4b32e0ad4ef37af2

Assets 3
Loading