Skip to content

llama-server with per-device memory reporting (/metrics + GET /memory)

Latest
Latest

Choose a tag to compare

View all tags
@tobocop2 tobocop2 released this 27 Jul 03:42
build-memory-metrics-release-b10139-b11c3a9
b11c3a9

Prebuilt llama.cpp binaries for all platforms, with the per-device memory reporting from my upstream PR: ggml-org#26130 (issue: ggml-org#26129).

What you get

llama-server can now tell you how many bytes it allocated on each GPU and in host RAM - split into model weights, context/KV cache, and compute buffers - plus each device's total and free memory and the model's layer count. With a multimodal projector loaded, its memory is reported too.

Two surfaces over the same data:

  • --metrics exposes it on /metrics as device-labelled Prometheus gauges - graph VRAM per card in Grafana, alert on headroom
  • a new --memory flag enables GET /memory, the same numbers as JSON for scripts and orchestrators

Until now these numbers were only printed in the server log, so reading them from another program meant regex-parsing log lines.

What's included

Asset Platform
llama-*-bin-ubuntu-x64.tar.gz Linux x64, CPU
llama-*-bin-ubuntu-arm64.tar.gz Linux arm64, CPU
llama-*-bin-ubuntu-vulkan-x64.tar.gz Linux x64, Vulkan
llama-*-bin-ubuntu-vulkan-arm64.tar.gz Linux arm64, Vulkan
llama-*-bin-ubuntu-rocm-7.2-x64.tar.gz Linux x64, ROCm (AMD)
llama-*-bin-ubuntu-openvino-2026.2.1-x64.tar.gz Linux x64, OpenVINO
llama-*-bin-win-cpu-x64.zip / -arm64.zip Windows, CPU
llama-*-bin-win-cuda-12.4-x64.zip / -13.3-x64.zip Windows x64, NVIDIA CUDA
cudart-llama-bin-win-cuda-*.zip CUDA runtime DLLs, drop next to the CUDA build
llama-*-bin-win-hip-radeon-x64.zip Windows x64, HIP (AMD Radeon)
llama-*-bin-win-vulkan-x64.zip Windows x64, Vulkan
llama-*-bin-win-opencl-adreno-arm64.zip Windows arm64, OpenCL (Adreno)
llama-*-bin-win-openvino-2026.2.1-x64.zip Windows x64, OpenVINO
llama-*-bin-macos-arm64.tar.gz macOS Apple Silicon (requires macOS 26, same as upstream releases)
llama-*-bin-macos-arm64-macos14.tar.gz macOS Apple Silicon, macOS 14+ (extra build, see below)
llama-*-bin-macos-x64.tar.gz macOS Intel (13.3+)
llama-*-bin-android-arm64.tar.gz Android arm64
llama-*-xcframework.zip iOS/macOS XCFramework
llama-*-ui.tar.gz Web UI bundle

The Windows zips are self-contained: the CPU backend is bundled into each backend zip, matching the official release packaging.

How it was built

By the repo's own release.yml workflow running on this fork, from the PR's commit plus one workaround commit for an android build break that exists on upstream master (unrelated to the PR). Not included: SYCL variants (failing on upstream master as well) and s390x (GitHub provides no s390x runners to forks).

macOS note

The standard macos-arm64 asset is built on GitHub's macOS 26 runner and requires macOS 26 - upstream's releases have the same constraint. The extra macos-arm64-macos14 asset is built from the same source with a macOS 14 deployment target for Apple Silicon machines on current macOS.

Verified

Each row is a machine I ran a published asset on, serving tinygemma3 with its multimodal projector straight from the tarball:

Machine OS Backend / device Asset Checked
MacBook Pro (M1 Pro) macOS 14 Metal (MTL0) macos-arm64-macos14 GET /memory, /metrics, mmproj rows
HP desktop Manjaro (glibc 2.40) CPU ubuntu-x64 GET /memory, /metrics series
HP desktop Manjaro (glibc 2.40) Vulkan, GTX 1070 Ti (Vulkan0) ubuntu-vulkan-x64 GET /memory with total/free, completion
ThinkPad Manjaro (glibc 2.38) Vulkan, Intel iGPU (Vulkan0) ubuntu-vulkan-x64 GET /memory with the iGPU's shared-memory budget

Real /metrics output from that run (trimmed):

llamacpp:model_n_layer 8
llamacpp:memory_model_bytes{device="MTL0"} 40707072
llamacpp:memory_model_bytes{device="Host"} 35660288
llamacpp:memory_context_bytes{device="MTL0"} 33554432
llamacpp:memory_compute_bytes{device="MTL0"} 17682464
llamacpp:memory_mmproj_bytes{device="MTL0"} 2181120
llamacpp:memory_mmproj_bytes{device="Host"} 12288
llamacpp:memory_total_bytes{device="MTL0"} 22906503168
llamacpp:memory_free_bytes{device="MTL0"} 22811934720

And GET /memory from the same run:

{
  "n_layer": 8,
  "data": [
    {
      "name": "MTL0",
      "model": 40707072,
      "context": 33554432,
      "compute": 17682464,
      "mmproj": 2181120,
      "total": 22906503168,
      "free": 22811934720
    },
    {
      "name": "Host",
      "model": 35660288,
      "context": 4195328,
      "compute": 4739104,
      "mmproj": 12288
    }
  ]
}

GET /memory from the Vulkan run on the 1070 Ti:

{
  "n_layer": 8,
  "data": [
    {
      "name": "Vulkan0",
      "model": 40707072,
      "context": 33554432,
      "compute": 11028496,
      "mmproj": 1951744,
      "total": 8847884288,
      "free": 8405647360
    },
    {
      "name": "Host",
      "model": 35660288,
      "context": 4195360,
      "compute": 4739168,
      "mmproj": 12288
    }
  ]
}
Assets 24
Loading