Skip to content

Performance Optimization

Jump to bottom
Richard Huang edited this page Aug 3, 2026 · 1 revision

Performance Optimization

Performance optimization is the discipline of finding out why a system is slower than it should be, and fixing the thing that actually matters. It spans every layer — Model Frontend through Runtime — because the bottleneck can live anywhere and is rarely where intuition suggests.

The defining skill is not making things fast. It is correctly identifying what to make fast.

Objectives

  • Find the real bottleneck, not a plausible one
  • Quantify the ceiling — know what performance is theoretically available
  • Prioritize by impact — fix what dominates
  • Verify improvement — measure before and after, on the same workload
  • Prevent regression — performance that isn't tracked degrades

Where It Fits

flowchart LR
    A[Runtime execution] --> B[Measure]
    B --> C[Analyze<br/>find the limit]
    C --> D[Fix]
    D --> A
    C -.->|architectural limit| E[Architecture feedback]
    style C fill:#2d6a9f,color:#fff
Loading

The dashed path matters: sometimes the honest conclusion is that software cannot fix it, and the finding belongs to Architecture.

The Roofline Model

Every kernel is limited by either compute or memory bandwidth. Which one is determined by arithmetic intensity:

Arithmetic intensity = FLOPs / bytes moved

Machine balance = peak FLOPs/s / peak bytes/s

If intensity < machine balance  →  memory bound
If intensity > machine balance  →  compute bound

This single comparison should be the first step in any performance investigation, because it tells you which optimizations are even capable of helping.

Situation Useful Useless
Memory bound Quantization, fusion, better layout, batching Faster arithmetic, more compute units
Compute bound Better tiling, higher utilization, narrower types Reducing memory traffic

For LLM decode, arithmetic intensity is very low and nearly everything is memory bound. Teams that skip this analysis routinely spend weeks optimizing arithmetic in a bandwidth-limited system and gain nothing.

Establishing the Ceiling

Before optimizing, compute what the hardware could do in principle.

For decode, a useful bound:

Minimum time per token ≈ (bytes that must be read) / (peak memory bandwidth)

Bytes that must be read = model weights + KV cache for the batch. If measured time is close to this bound, the software is doing well and further gains require reading fewer bytes — quantization, sparsity, or architectural change. If measured time is far above it, there is software work to do.

This calculation takes minutes and reframes entire projects. It converts "is this fast?" from an opinion into an arithmetic question.

Measurement Discipline

Principle Why
Measure the real workload Synthetic benchmarks mislead; real request mixes have different shapes
Warm up first Compilation, caching, and allocation distort early iterations
Report distributions Averages hide the tail, and the tail is what users experience
Change one thing Multiple simultaneous changes make attribution impossible
Control the environment Frequency scaling, other tenants, and thermal state all shift results
Repeat A single run is an anecdote

Report P50, P95, and P99 — not just the mean. A change that improves the mean while worsening P99 has usually made the system worse.

Profiling Layers

Bottlenecks hide at different levels, and each needs its own tool:

Layer Question Look for
System Is the device even busy? Utilization, idle gaps between steps
Step Where does a step go? Dispatch, compute, sample, update breakdown
Kernel Which kernels dominate? Time per kernel, achieved bandwidth
Instruction Why is this kernel slow? Stalls, occupancy, memory patterns

Work top-down. Starting at the instruction level is the classic beginner error — it produces a beautifully optimized kernel that accounts for 3% of runtime while the device sits idle 40% of the time between steps.

Common Bottleneck Patterns

Symptom Likely cause Where to look
Device idle between steps Host overhead dominating Runtime — dispatch, allocation, sync
Low bandwidth utilization Poor access patterns Kernel Library — layout, coalescing
Low compute utilization, high bandwidth Memory bound, working as expected Quantization, Graph Compiler fusion
Small batch sizes Cache capacity limiting concurrency Memory Management
Latency spikes Long prefills, preemption, or allocation Batching and Scheduling
Fast at one shape, slow at another Missing tuned variant Kernel Library
Excessive memory traffic Unfused operations Graph Compiler

The Optimization Levers, Ranked

For memory-bound LLM decode, in rough order of impact:

  1. Reduce bytes read — quantization of weights and KV cache
  2. Increase batch size — amortize weight reads across requests
  3. Fuse operations — eliminate intermediate round trips
  4. Reduce host overhead — graph capture, no allocation in the loop
  5. Improve layout — make every read a full-width, aligned access
  6. Tune tiles — maximize reuse within capacity
  7. Optimize arithmetic — last, and usually smallest

That ordering surprises people. It follows directly from the roofline: when memory bound, anything that reduces bytes moved beats anything that speeds up arithmetic.

Metrics

Metric Definition
Tokens/second System throughput
TTFT / TPOT Latency as users experience it
Goodput Throughput within latency SLOs
Model bandwidth utilization Achieved bytes/s vs. peak — the key decode efficiency metric
Compute utilization Achieved FLOPs vs. peak
Device idle fraction Time the hardware did nothing
Energy per token The metric that matters at deployment scale

For decode, bandwidth utilization is the honest efficiency number. A system at 85% of peak memory bandwidth is running well even at a few percent of peak FLOPs, and reporting the FLOPs number instead would suggest a problem that doesn't exist.

Common Problems

Problem Root cause Fix
Optimization gave no speedup Wrong bottleneck targeted Roofline analysis first
Benchmark improves, production doesn't Unrepresentative benchmark Measure real traffic
Results vary run to run Uncontrolled environment Pin frequency; isolate; repeat
Mean improves, users complain Tail regression Track P99
Gains disappear over time No regression tracking Continuous performance testing
Micro-optimizing a small kernel Bottom-up profiling Work top-down
Can't reproduce a report Undocumented conditions Record config, versions, workload with every result

Best Practices

  • Compute the theoretical bound first. It takes minutes and prevents weeks of wasted effort.
  • Profile top-down. System, then step, then kernel, then instruction.
  • Fix the biggest thing. A 50% improvement on 5% of runtime is 2.5%.
  • Change one variable at a time.
  • Track performance continuously, like tests. Regressions found months later are expensive.
  • Report the distribution, not the mean.
  • Know when to stop. At 90% of the bandwidth bound, further software work has little left to give — the next gain is architectural.
  • Feed findings upstream. Performance analysis is how software tells Architecture what the next chip should change.

See also: Graph Compiler · Kernel Library · Memory Management · Batching and Scheduling · Runtime · Inference Software

Clone this wiki locally