llama.cpp on NVIDIA Tesla K80

A practical how-to for building and running llama.cpp with CUDA acceleration on the NVIDIA Tesla K80 (compute capability 3.7).

The K80 was once the world’s most popular datacenter GPU. It is not a consumer chat accelerator by modern standards — generation is measured in single-digit to low double-digit tokens per second on mid-size models — but if you are willing to wait for the answer, it remains highly effective in a modern LLM workflow. That profile fits lights-out and batched work: overnight summarization, offline coding agents, bulk classification, RAG pipelines that queue jobs, and multi-GPU fleets of small models running in parallel.

This repository is documentation and a reproducible build recipe, not a fork of llama.cpp.

Document	Contents
This README	How to build and run (start here)
docs/benchmarks.md	Measured llama-bench results and VRAM notes
docs/dell-r730.md	Lab notes: Dell PowerEdge R730 with 4× K80
.gitlab-ci.yml	Optional automated build (CUDA 11.4 / arch 37)

---

Requirements

Component	Value used in this guide
GPU	NVIDIA Tesla K80 (CC 3.7), ~11.3 GiB usable VRAM per GPU
OS	Ubuntu 24.04 LTS (close releases should work)
NVIDIA driver	470-series (e.g. nvidia-driver-470-server)
CUDA toolkit	11.4 (matches driver era; newer toolkits drop Kepler)
CMake	≥ 3.18 (this guide installs 3.28)
Build tools	build-essential, Ninja, git, GCC/G++
llama.cpp tag	b9781 (pin for reproducibility; bump when ready)
Host CPU flag	-march=haswell (x86_64-v3 baseline; change if needed)

Benchmarks in this repo were taken with llama.cpp build 51eae8c and Q4_K_M GGUF models, full GPU offload.

---

Why a custom build?

Default llama.cpp / CUDA builds often target architectures and toolkit versions that do not include Kepler (CC 3.7). On a K80 you typically need:

1. CUDA 11.4 toolkit and a 470-series driver
2. CMAKE_CUDA_ARCHITECTURES=37 so kernels are compiled for the K80
3. A CUDA stub library symlink (libcuda.so.1) when linking in containers or on machines without a live GPU
4. NCCL available next to the binaries at runtime

Once those are set, llama.cpp runs normally: full layer offload, Flash Attention (when enabled), OpenAI-compatible llama-server, and multi-model workflows.

---

Build

1. Install packages

sudo apt-get update
sudo apt-get install -y --no-install-recommends \
    wget \
    build-essential \
    ninja-build \
    git \
    libgomp1

2. Install CMake 3.28

Ubuntu’s packaged CMake may be too old for current llama.cpp:

wget -qO- "https://github.com/Kitware/CMake/releases/download/v3.28.0/cmake-3.28.0-linux-x86_64.tar.gz" \
  | sudo tar --strip-components=1 -xz -C /usr/local

cmake --version

3. Prepare CUDA stub libraries (for linking)

The CUDA image/toolkit often ships libcuda.so but not libcuda.so.1. llama.cpp’s CUDA backend expects the versioned name:

export LD_LIBRARY_PATH="/usr/local/cuda/lib64/stubs:${LD_LIBRARY_PATH}"
export LIBRARY_PATH="/usr/local/cuda/lib64/stubs:${LIBRARY_PATH}"

sudo ln -sfn /usr/local/cuda/lib64/stubs/libcuda.so \
             /usr/local/cuda/lib64/stubs/libcuda.so.1

4. Clone llama.cpp

git clone https://github.com/ggml-org/llama.cpp.git --branch b9781 --depth 1
cd llama.cpp

Use a later release tag when you intentionally upgrade; re-validate flags and performance after upgrades.

5. Configure (CUDA 11.4, arch 37)

mkdir -p build && cd build

cmake .. \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES="37" \
  -DCMAKE_SYSTEM_NAME=Linux \
  -DCMAKE_CROSSCOMPILING=ON \
  -DGGML_CUDA=ON \
  -DGGML_NATIVE=OFF \
  -DLLAMA_BUILD_TESTS=OFF \
  -DCMAKE_C_COMPILER=gcc \
  -DCMAKE_CXX_COMPILER=g++ \
  -DCMAKE_C_FLAGS="-march=haswell" \
  -DCMAKE_CXX_FLAGS="-march=haswell" \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
  -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs" \
  -DCMAKE_SHARED_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs" \
  -DCMAKE_INSTALL_RPATH="\$ORIGIN/lib" \
  -G Ninja

Notes:

• CUDA_ARCHITECTURES=37 → Kepler / K80
• -march=haswell → portable x86_64-v3 host binary (adjust for your CPUs)
• GGML_CUDA=ON → CUDA backend
• Stub library paths allow linking without a GPU present (CI / build hosts)

6. Compile

cmake --build . --config Release -j "$(nproc)"
ls -lh bin/

7. Copy NCCL into the binary directory

cp -a /usr/lib/x86_64-linux-gnu/libnccl.so.2* bin/

(If your distro places NCCL elsewhere, copy from that path instead.)

Expected layout

llama.cpp/
  bin/
    llama-cli
    llama-server
    llama-bench
    libnccl.so.2*
    …

The same flags and steps are encoded in .gitlab-ci.yml for automated builds.

---

Host setup (driver + CUDA runtime)

On the machine that will run inference:

NVIDIA driver (470 series)

sudo apt update
sudo apt install -y nvidia-utils-470-server nvidia-driver-470-server
sudo apt install -y "linux-headers-$(uname -r)"
sudo depmod -a
nvidia-smi   # reboot once if the driver does not load

CUDA 11.4 toolkit (toolkit only — keep the 470 driver)

mkdir -p ~/cuda && cd ~/cuda
wget https://developer.download.nvidia.com/compute/cuda/11.4.4/local_installers/cuda_11.4.4_470.82.01_linux.run
chmod +x cuda_11.4.4_470.82.01_linux.run

sudo sh ./cuda_11.4.4_470.82.01_linux.run \
  --silent \
  --toolkit \
  --no-opengl-files \
  --override

sudo ldconfig

Shared libraries

If binaries fail with missing libllama-*.so (or similar), add the bin directory to the dynamic linker:

echo "$HOME/llama.cpp/bin" | sudo tee /etc/ld.so.conf.d/llama.conf
sudo ldconfig

Use the actual path where you installed the build.

Verify

nvidia-smi
~/llama.cpp/bin/llama-cli --version
ldconfig -p | grep libcudart

Expect Tesla K80, driver 470.x, and a working CUDA 11.4 runtime.

---

Run

Quick CLI smoke test

export CUDA_VISIBLE_DEVICES=0

./bin/llama-cli \
  -m /path/to/model-Q4_K_M.gguf \
  -ngl 99 \
  -p "Explain Kepler GPUs in one paragraph." \
  -n 128

llama-server (recommended baseline)

export CUDA_VISIBLE_DEVICES=0

./bin/llama-server \
  -m /path/to/model-Q4_K_M.gguf \
  --n-gpu-layers 99 \
  --ctx-size 8192 \
  --batch-size 512 \
  --ubatch-size 256 \
  --parallel 2 \
  --flash-attn on \
  --host 0.0.0.0 \
  --port 8080

OpenAI-compatible HTTP API: see the upstream llama-server README.

Always set max_tokens (or equivalent) in API clients so generations cannot run away.

Multi-GPU pattern

On multi-K80 hosts, prefer one llama-server process per GPU (CUDA_VISIBLE_DEVICES=N and a distinct port) rather than aggressive layer-splitting for concurrency. Layer split can help fit larger models or improve long prefill; it does not meaningfully speed up token generation on this hardware. Details: docs/dell-r730.md.

---

Recommended runtime settings (K80)

Parameter	Practical default	Notes
GPU layers (-ngl)	99 / -1	Full offload
Context (--ctx-size)	8k–16k chat; up to ~62k auto-fit on small models	KV cache dominates VRAM (~1 GiB per ~8k tokens is a useful rule of thumb)
Batch (--batch-size)	512–768	Raise for offline throughput
Microbatch (--ubatch-size)	128–256 interactive; 256–512 batch	Strongly affects latency
Parallel slots (--parallel)	1–2	Higher concurrency often increases latency on bandwidth-limited Kepler
Flash Attention	on	Helps modestly on K80
KV cache	f16 default; q8_0 to reclaim VRAM	Trade a little quality for context headroom

Workload profiles

Profile	Context	Batch	Microbatch	Parallel	Best for
Snappy / agents	8k–16k	256–512	128–256	1–2	Low-latency tools and chat
Balanced	16k–24k	512–768	256	2	General use
Heavy / offline	24k–32k	512–1024	256–512	1–2	Summarization, batch jobs
Max context	~48k–62k	256–512	128–256	1	Extreme context; expect slower decode

Keep roughly ≥1 GiB free VRAM after load so long prompts do not OOM.

---

What works well on K80

• Q4_K_M (and similar) quantizations with full GPU offload
• Models roughly ≤4B for interactive feel; 7–9B for usable batch work; 14B+ if you accept ~4 t/s generation
• Queued / overnight jobs and multi-card fleets of independent servers
• Measured numbers: docs/benchmarks.md

Kepler notes you will see in logs:

• CUDA graphs are typically disabled for this architecture
• Memory bandwidth, not raw TFLOPS, dominates decode speed
• PCIe host↔device transfers are relatively expensive — keep work on-GPU

---

Further reading

1. Benchmarks — prompt processing, token generation, VRAM by model
2. Dell R730 lab notes — four-card deployment experience and gotchas
3. Upstream — llama.cpp, server docs

---

License

This project’s documentation and build configuration are released under the MIT License. llama.cpp remains under its own license.