Detect whatever GPU hardware you actually have — new or used, one vendor or mixed — figure out the biggest open-weight model that fits in VRAM, and launch inference. No vendor lock-in, no reinventing drivers.
NVIDIA/CUDA still runs the large majority of AI compute. Open compute stacks (tinygrad, ROCm, Triton, Mesa/NVK) are closing the gap but none has displaced it, and none of them solve a much smaller, much more tractable problem: if you already own a used mining-era NVIDIA card, a consumer AMD GPU, and maybe an old datacenter card, there's no tool that looks at that mismatched pile of VRAM and tells you what you can actually run on it.
cheapstack is that tool. It's a thin, opinionated planning + launch layer on top of llama.cpp, which already has solid CUDA, HIP/ROCm, and Vulkan backends — Vulkan in particular is the only one of the three that can drive NVIDIA + AMD + Intel GPUs in the same process, which is what makes heterogeneous "cheap GPU cluster" setups tractable without hand-rolled multi-process sharding.
pip install -e .Requires Python 3.9+. No runtime dependencies — hardware probing shells out
to nvidia-smi / amd-smi / rocm-smi / vulkaninfo, whichever exist on
your system.
# What GPUs does this machine actually have, across every vendor?
cheapstack detect
# Given that hardware, what's the biggest open model I can run?
cheapstack plan --context 8192
# Plan + launch llama-server with the right --tensor-split and backend flags
cheapstack serve --model-path ./llama-3.1-8b-q6_k.gguf --context 8192
# Benchmark and translate raw tok/s into $/1K tokens for a given hardware cost
cheapstack bench --model-path ./model.gguf --hardware-cost 700 --power-watts 350plan and serve need a llama.cpp binary on PATH (llama-server,
llama-bench) built with the backend cheapstack recommends. cheapstack plan prints the exact cmake invocation for your detected hardware.
- Model/quant sizing (
quant.py) is computed from bits-per-weight per GGUF quant level, not a hardcoded size table — so any parameter count in the catalog (models.py) can be sized for any quant on demand. - VRAM budget (
planner.py) reserves a headroom fraction (default 10%) plus a small fixed per-GPU overhead, then walks the catalog largest-model- first, best-quant-first, returning the first config that fits. This encodes the common local-inference preference: more parameters over more precision. - Tensor split is VRAM-proportional across however many GPUs you have,
mismatched or not — feeds directly into llama.cpp's
--tensor-split. - Backend choice (
backend.py): single-vendor fleet gets that vendor's native backend (CUDA/HIP); mixed-vendor fleet gets Vulkan, the only cross-vendor option of the three.
KV-cache sizing is a rule-of-thumb estimate (documented in quant.py) — real
figures depend on model architecture (layer/head count) that isn't derivable
from parameter count alone. Treat plan output as a starting point, not an
exact allocation.
# Rockchip RK3588-class NPU: generate an RKLLM conversion script (a genuinely
# different pipeline from the GPU path above — see cheapstack/npu.py)
cheapstack npu-convert --model meta-llama/Llama-3.2-3B-Instruct --target rk3588
# Autotuned Triton matmul, correctness-checked against torch.matmul and
# benchmarked — needs `pip install cheapstack[kernels]` (torch + triton) and
# a CUDA device. This is the harness the KernelLLM research gap needs to be
# measured against.
cheapstack autotune-matmul --m 4096 --n 4096 --k 4096
# Read-only PCI enumeration (vendor/device IDs, BAR sizes, link speed) —
# see DRIVER_NOTES.md for why this is the actual scope of "driver work"
# without published register documentation.
cheapstack pci-infoEarly. The catalog in models.py covers common open releases (Llama,
Mistral, Qwen, Mixtral, Gemma, GLM, Phi, DeepSeek-distill, Command R) — add
a line to extend it. Hardware detection is best-effort across nvidia-smi,
amd-smi/rocm-smi, a Rockchip RK3xxx NPU probe, and a vulkaninfo
fallback for anything those miss; PRs adding detection for other tools
(Intel xpu-smi, etc.) are the most useful next contribution.
See DRIVER_NOTES.md before assuming a from-scratch driver is the natural
next step — on NVIDIA hardware specifically, it isn't, and the reasoning is
worth reading before writing code under that name.
MIT