VRAM & KV-Cache Sizing Calculator for High-Throughput Serving & RadixAttention Planning (Zero-Dependency CLI & WebApp)

[DangerousANEN]

When deploying high-throughput model serving engines like SGLang (python3 -m sglang.launch_server), one of the key tuning parameters is memory budgeting (--mem-fraction-static) and sizing the KV-cache pool to support long context windows and concurrent request streams without triggering CUDA OOMs or cache thrashing.

With long sequence lengths (16k–128k) and high concurrency, the dynamic KV-cache footprint frequently outgrows the static model weights (e.g. for a 70B model in FP8 at 64k tokens, KV-cache alone requires dozens of gigabytes). Understanding exact memory allocation boundaries is essential before deploying on cluster hardware (A100, H100, L40S, RTX 4090/6000 Ada).

To help engineer and size GPU memory budgets for serving environments, I created llm-vram-calculator — an open-source utility featuring both an interactive browser WebApp and a standalone zero-dependency Python CLI:

• Interactive WebApp: https://dangerousanen.github.io/llm-vram-calculator/
• GitHub Repository: https://github.com/DangerousANEN/llm-vram-calculator
• Zero External Dependencies: Pure Python standard library (vram_calc.py), works seamlessly on Linux, macOS, and Windows.
• In-Depth Technical Analysis & Math: https://telegra.ph/Lokalnye-LLM-v-2026-rantajmy-raschet-VRAM-i-lovushka-KV-kehsha-10-28

---

Key Capabilities for Serving & Inference Planning

1. Complete Memory Footprint Decomposition:
   $$\text{Total VRAM} = M_{\text{weights}} + M_{\text{kv-cache}} + \text{CUDA Overhead} + \text{Peak Activations}$$

   • Quantization profiles for serving: FP8, NVFP4, AWQ, GPTQ, BF16/FP16, and GGUF (Q4_K_M, Q5_K_M, Q8_0).
   • KV-cache precision scaling: fp16, fp8, q8_0, q4_0 (essential when sizing quantized KV cache pools).
   • Support for Multi-Head Latent Attention (MLA) architecture parameters (DeepSeek V3 / R1) for accurate compressed KV calculation.

2. Concurrency & Context Scaling:

   • Evaluates memory demand across varying sequence lengths and batch concurrency levels to determine optimal --mem-fraction-static margins.

3. Lightweight Benchmarking CLI:

   • Built-in bench subcommand to measure Time to First Token (TTFT), Time Per Output Token (TPOT), and Tokens Per Second (TPS) directly against SGLang OpenAI-compatible endpoints (/v1/chat/completions) with streaming latency breakdowns.

Feedback, suggestions, and feature requests from the SGLang community are very welcome!