Skip to content

FLE v1.1.0: portable CUDA builds for Linux and Windows

Latest
Latest

Choose a tag to compare

View all tags
@sirfyyn sirfyyn released this 10 Oct 22:31
v1.1.0
bbedbbe

Portable binary packages of the FLE engine (the llama.cpp fork with the expert pager), built from source by GitHub Actions (workflow fle-binaries.yml, run 38052011997). They were built from commit 4f420c33, which is part of this release and has exactly the same tree as the tagged commit.

Package Platform CUDA GPU code
fle-4f420c33-bin-ubuntu-cuda-12.8-x64.tar.gz Linux x86_64 (glibc >= 2.35: Ubuntu 22.04+, Debian 12+) 12.8 machine code sm_86, sm_89, sm_120a; PTX for sm_50 to sm_90
fle-4f420c33-bin-win-cuda-13.4-x64.zip Windows 10+ x86_64 13.4 machine code sm_86, sm_89, sm_120a, sm_121a; PTX for sm_75 to sm_90

Not Blackwell-only: Ampere (sm_86), Ada (sm_89) and Blackwell (sm_120a) have native machine code; other GPUs get PTX that the driver compiles at first use. The CPU backend is built as one module per x86-64 feature level and the best one is picked at start, so the packages also run without a GPU (the FLE CPU kernels need AVX2).

Use

  1. Unpack the package.
  2. The CUDA runtime libraries are not part of the packages. Unpack the matching upstream llama.cpp package into the same directory (or install the CUDA 12 runtime on Linux):
  3. Start the server (an NVIDIA driver is needed for the GPU):
    • Linux: ./llama-server -m model.gguf --host 127.0.0.1 --port 8080 -ngl 99
    • Windows: llama-server.exe -m model.gguf --host 127.0.0.1 --port 8080 -ngl 99

Model files (GGUF): https://huggingface.co/sirfyyn/Qwen3.8-Flash-Next-FLE-GGUF. The FLE options are described in README-FLE.md and docs/fle/ENV.md. Each package contains a README.txt and a fle-build-info.json (commit, toolchain, requested and contained CUDA architectures).

What was checked

For each package the workflow checks the libraries and imports of every executable and library (Linux: RPATH, glibc and libstdc++ baseline; Windows: imports and exports of every exe and dll) and runs a smoke test on a small GGUF without a GPU (server start, /health, 16 tokens through /completion). It then downloads the package again onto a plain runner, adds the upstream cudart package and repeats both checks. These runs have no GPU: the CUDA code was compiled and linked (on Linux the CUDA module is also loaded against the stub libcuda, which proves that every symbol resolves), but no CI run computed on a GPU. Speed is not covered by this release.

Changes in the source

The source differs from v1.0.1 only where the engine had to be made portable: the CUDA backend and the expert pager can be built as loadable modules (GGML_BACKEND_DL), and POSIX-only code compiles with MSVC. The linked build used so far is unchanged. The workflow itself builds workflow artifacts only and is started by hand.

SHA-256

4b5bd41b343aad7ece729306367f399f4c72afc03ae0c1d1a42eda43f1f0b9cd  fle-4f420c33-bin-ubuntu-cuda-12.8-x64.tar.gz
4a72a71e7b900baa6f2a78d1d6a2d8baf26b5b92b37de7663c7829703f4ea50c  fle-4f420c33-bin-win-cuda-13.4-x64.zip
Assets 4
Loading