gpu-memory-profiler

Author: João Felipe De Souza

---

Overview

Detailed VRAM breakdown profiler for transformer inference.

This project separates GPU memory into:

• weights
• KV-cache
• runtime / activation / framework overhead
• per-layer memory
• activation footprints
• forward-pass memory timeline
• practical VRAM budgeting

For architecture and methodology, see DESIGN.md.

---

Why This Matters

Analytical KV-cache formulas are useful, but they often underpredict measured VRAM.

This project answers:

Where does the rest of the GPU memory go?

And then goes one step further:

Can we fit a practical memory model that predicts VRAM from model size and KV-cache?

---

Models and Hardware

Models

• GPT-2 (117M)
• GPT-2-medium (345M)

Hardware

• NVIDIA RTX 2070
• CUDA 13.0

Runtime sweep

• prompt lengths: 128, 512, 960
• batch sizes: 1, 2, 4, 8, 16

Activation profiling sweep

• prompt lengths: 128, 512
• batch sizes: 1, 4

---

Key Findings

Finding 1 — Feed-forward layers dominate static VRAM

GPT-2-medium weight breakdown

Category	Memory
FFN	384.2 MB
Attention	192.2 MB
Embedding	100.2 MB
LayerNorm	0.2 MB

The largest share of static memory is in feed-forward layers, not attention.

Finding 2 — Per-layer weight cost is perfectly uniform

Model	Mean per-layer memory	Std
GPT-2	13.52 MB	0.0000
GPT-2-medium	24.03 MB	0.0000

This means static transformer memory scales almost perfectly linearly with layer count.

Finding 3 — KV-cache explains only part of runtime VRAM

Average over the runtime sweep:

Model	Runtime peak	KV-cache estimate	Runtime overhead
GPT-2	691.6 MB	116.5 MB	324.1 MB
GPT-2-medium	1322.3 MB	310.6 MB	326.8 MB

A large fraction of runtime memory is not KV-cache.

Finding 4 — Activation sums are much larger than peak runtime

Example:

GPT-2-medium, prompt=512, batch=4

• Sum of hooked activations: 1841.3 MB
• Peak runtime memory: 200.3 MB

This shows that summing all intermediate activations grossly overestimates actual peak VRAM, because most tensors do not coexist simultaneously.

Finding 5 — Activation volume scales almost perfectly with prompt × batch

GPT-2

Prompt	Batch	Activation total
128	1	51.3 MB
128	4	204.5 MB
512	1	205.1 MB
512	4	818.1 MB

GPT-2-medium

Prompt	Batch	Activation total
128	1	115.3 MB
128	4	460.3 MB
512	1	461.1 MB
512	4	1841.3 MB

This is the expected linear scaling law.

Finding 6 — A simple linear VRAM model predicts memory extremely well

Fitted formulas:

GPT-2

VRAM = 237.4 + 3.770 × KV_est + 15.2   MB

GPT-2-medium

VRAM = 676.8 + 2.040 × KV_est + 11.8   MB

Goodness of fit:

• GPT-2: R² = 0.999970
• GPT-2-medium: R² = 0.999954

Finding 7 — Prediction error is below 1.2%

Model	Mean error	Max error
GPT-2	0.29%	0.82%
GPT-2-medium	0.45%	1.11%

This makes the profiler practically useful as a memory budget calculator.

Finding 8 — Practical VRAM budgeting becomes possible

GPT-2

Prompt	Batch	Predicted VRAM
128	8	0.38 GB
512	16	1.31 GB
960	16	2.24 GB

GPT-2-medium

Prompt	Batch	Predicted VRAM
128	8	0.86 GB
512	16	2.21 GB
960	16	3.54 GB

---

Main Conclusion

Transformer inference memory is not just:

• model weights
• plus KV-cache

There is a third major term:

• runtime / activation / framework overhead

The practical formula is:

VRAM = weights + alpha × KV-cache + beta

This project explains why simple KV-cache formulas often underpredict real VRAM by 2–4×, and provides a practical memory model that predicts usage with <1.2% error.

---

Results Files

File	Description
results/weight_detail.csv	Per-parameter memory breakdown
results/weight_summary.csv	Weights grouped by category
results/weight_totals.csv	Total static memory
results/runtime_memory.csv	Runtime memory sweep
results/runtime_summary.csv	Aggregated runtime summary
results/per_layer_weights.csv	Per-layer weight memory
results/activation_summary.csv	Activation profiling summary
results/memory_timeline.csv	Forward-pass memory timeline
results/memory_model_fit.csv	Linear VRAM model coefficients
results/memory_predictions.csv	Predicted vs measured VRAM
results/memory_budget_table.csv	Practical budget table
results/metadata.json	Benchmark configuration

---

Plots

File	Description
plots/gpt2_weight_breakdown.png	GPT-2 static weight categories
plots/gpt2-medium_weight_breakdown.png	GPT-2-medium static weight categories
plots/gpt2_peak_memory_vs_batch.png	GPT-2 peak memory scaling
plots/gpt2-medium_peak_memory_vs_batch.png	GPT-2-medium peak memory scaling
plots/runtime_vs_kv_estimate.png	Measured runtime vs KV estimate
plots/gpt2_overhead_ratio.png	GPT-2 overhead / KV ratio
plots/gpt2-medium_overhead_ratio.png	GPT-2-medium overhead / KV ratio
plots/gpt2_runtime_breakdown.png	GPT-2 runtime memory breakdown
plots/gpt2-medium_runtime_breakdown.png	GPT-2-medium runtime breakdown
plots/gpt2_per_layer_weights.png	GPT-2 per-layer memory
plots/gpt2-medium_per_layer_weights.png	GPT-2-medium per-layer memory
plots/activation_vs_prompt_len.png	Activation scaling
plots/gpt2_memory_timeline.png	GPT-2 forward-pass memory timeline
plots/gpt2-medium_memory_timeline.png	GPT-2-medium memory timeline
plots/predicted_vs_measured.png	Linear model fit
plots/gpt2_budget_heatmap.png	GPT-2 VRAM budget heatmap
plots/gpt2-medium_budget_heatmap.png	GPT-2-medium VRAM budget heatmap
plots/max_batch_8gb.png	Max batch that fits in 8 GB

---

Repository Structure

gpu-memory-profiler/
├── gpu_memory_profiler.py
├── advanced_memory_profiler.py
├── memory_budget_calculator.py
├── plot_gpu_memory.py
├── README.md
├── DESIGN.md
├── LICENSE
├── requirements.txt
├── results/
└── plots/

---

How to Run

1. Setup

python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

2. Base profiler

python3 gpu_memory_profiler.py

3. Advanced profiler

python3 advanced_memory_profiler.py

4. Memory budget calculator

python3 memory_budget_calculator.py

5. Generate plots

python3 plot_gpu_memory.py

---

Limitations

• Only GPT-2 family tested
• RTX 2070 only
• Runtime overhead is inferred, not directly instrumented inside PyTorch internals
• Activation memory is measured by hooks and summed, which overestimates simultaneous residency
• The fitted memory model is empirical and validated only within the tested range

---

References

• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)
• Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (2022)