VRAM & KV-Cache Estimator for GGUF Export & Post-Fine-Tuning Serving (Zero-Dependency CLI & Interactive WebApp)

[DangerousANEN]

After training or fine-tuning models with Unsloth and exporting them (e.g. via model.save_pretrained_gguf("...", tokenizer, quantization_method="q4_k_m") or 16-bit LoRA merges for vLLM / llama.cpp), the immediate next hurdle is deployment budgeting: will the exported model fit into target GPU VRAM during inference across extended context lengths?

A very common pitfall when transitioning from fine-tuning to inference is looking solely at the quantized model weight file size ($M_{\text{weights}}$) and neglecting the dynamic growth of the KV-cache ($M_{\text{kv}}$) and CUDA runtime overhead. For instance, while an 8B model in Q4_K_M occupies only ~5.5 GB of weights, running it at 16k–32k context in FP16 KV-cache adds another 2–4 GB, causing unexpected CUDA Out of Memory (OOM) crashes on 8 GB / 12 GB consumer cards (RTX 3060, 4060, 4070).

To make memory planning seamless for local serving, GGUF offloading, and benchmark validation, I built llm-vram-calculator — an open-source utility with both an interactive WebApp and a standalone zero-dependency Python CLI:

• Interactive WebApp: https://dangerousanen.github.io/llm-vram-calculator/
• GitHub Repository: https://github.com/DangerousANEN/llm-vram-calculator
• Zero Dependencies: Pure Python standard library (vram_calc.py), works cross-platform (Linux, macOS, Windows).
• Technical Deep-Dive & Math: https://telegra.ph/Lokalnye-LLM-v-2026-rantajmy-raschet-VRAM-i-lovushka-KV-kehsha-10-28

---

Core Features

1. Complete VRAM Allocation Breakdown:
   $$\text{Total VRAM} = M_{\text{weights}} + M_{\text{kv-cache}} + \text{CUDA Overhead} + \text{Peak Activations}$$

   • Quantization profiles for popular export formats: Q4_K_M, Q5_K_M, Q8_0, FP16, BF16, FP8, NVFP4.
   • KV-cache precision scaling: fp16, fp8, q8_0, q4_0 (crucial for evaluating quantized KV savings in llama.cpp / vLLM).
   • Support for Multi-Head Latent Attention (MLA) for DeepSeek V3/R1 architectures.

2. GPU Layer Offloading Planning (-ngl):

   • Calculates per-layer memory consumption to determine exact layer offload distribution between GPU VRAM and host RAM for hybrid local setups.

3. Inference Benchmarking:

   • Standalone bench subcommand to measure Time to First Token (TTFT) and Tokens Per Second (TPS) across local endpoints (vLLM, llama-server, Ollama) with streaming latency breakdowns.

Feedback, suggestions, and edge case benchmarks from the Unsloth community are warmly welcome!