-
Notifications
You must be signed in to change notification settings - Fork 0
Graph Compiler
The graph compiler turns a normalized IR from the Model Frontend into an execution plan for the hardware. It decides what gets fused, how work is tiled, where tensors live, and in what order everything runs.
This is where most of the achievable performance is won or lost. A kernel can only be as fast as its inputs allow; the compiler decides what those inputs are.
- Minimize memory traffic — the dominant cost in inference
- Maximize hardware utilization — keep the compute units fed
- Respect capacity limits — on-chip memory is small and fixed
- Preserve numerics — optimization must not change results beyond tolerance
- Compile in reasonable time — a compiler nobody waits for is a compiler nobody uses
flowchart LR
A[Model Frontend<br/>IR] --> B[Graph Compiler]
B --> C[Kernel Library]
B --> D[Memory Management]
C --> E[Runtime]
D --> E
style B fill:#2d6a9f,color:#fff
Every optimization in this stage traces back to one ratio:
Arithmetic intensity = FLOPs performed / bytes moved
Hardware has a fixed ratio of compute throughput to memory bandwidth. If a kernel's arithmetic intensity is below that ratio, it is memory bound — the compute units idle waiting for data, and adding more of them changes nothing. Above it, the kernel is compute bound.
This is the roofline model, and it explains the entire shape of an inference compiler:
| Operation | Intensity | Implication |
|---|---|---|
| Matrix-matrix (prefill) | High | Compute bound; tiling and reuse matter |
| Matrix-vector (decode) | Very low | Memory bound; every byte read is precious |
| Elementwise (activation, add) | Near zero | Never worth a separate memory round trip |
| Normalization | Low | Must be fused into a neighbor |
The single most valuable thing a compiler does for decode-phase inference is stop moving data. That's what fusion is.
Fusion merges adjacent operations so intermediates stay in fast local memory instead of round-tripping through main memory.
Consider a normalization followed by a matrix multiply followed by an activation. Unfused, each writes its output to memory and the next reads it back. Fused, the intermediate never leaves the chip.
| Fusion type | Example | Benefit |
|---|---|---|
| Elementwise chains | add → scale → activation | Trivially profitable; always do it |
| Producer-consumer | matmul → bias → activation | Removes a full write/read of the result |
| Reduction fusion | matmul → row sum | Avoids materializing the full matrix |
| Attention fusion | Q·Kᵀ → softmax → ·V without materializing scores | Removes an O(n²) intermediate entirely |
That last row is the big one for transformers. Materializing the full attention score matrix costs memory proportional to sequence length squared. Fusing the whole attention computation so scores are consumed tile-by-tile as they're produced eliminates that intermediate — turning attention from memory bound to something much closer to compute bound.
Fusion isn't free:
- Register and local memory pressure — a fused region needs room for all live intermediates
- Reduced reuse — a fused producer may be recomputed for multiple consumers
- Numerical differences — keeping values in higher internal precision changes results, usually for the better, but it changes them
- Combinatorial search space — deciding what to fuse is a real optimization problem
Tiling breaks a large computation into blocks sized to fit on-chip memory, so each block of data is loaded once and reused many times.
For a matrix multiply, tile sizes determine:
Data reuse ≈ tile dimension
Bytes moved ≈ (problem size) / (tile dimension)
Bigger tiles mean more reuse and less traffic — until they no longer fit in local memory, at which point performance falls off a cliff. Tile selection is therefore a constrained optimization against real capacity limits, and it is one of the highest-leverage decisions the compiler makes.
Related transformations:
| Transformation | Purpose |
|---|---|
| Loop tiling | Fit working set in local memory |
| Loop fusion | Combine loops over the same data |
| Loop reordering | Improve access locality |
| Unrolling | Reduce loop overhead, expose parallelism |
| Software pipelining | Overlap load, compute, and store stages |
| Double buffering | Load the next tile while computing the current one |
Double buffering deserves emphasis: it converts memory latency from a stall into overlapped work. Without it, an accelerator with excellent peak throughput can sit idle most of the time.
How a tensor is arranged in memory determines whether hardware can read it efficiently.
- Contiguity — the fastest-varying dimension should match the access pattern
- Tiling / blocking — storing data pre-blocked avoids gather work at load time
- Padding and alignment — aligning to the memory interface width avoids split accesses
- Weight pre-packing — reordering weights offline into the exact layout the kernel wants costs nothing at runtime
Layout conflicts between consecutive operations force transpose operations, which are pure overhead. A good compiler propagates layout constraints globally and picks an assignment that minimizes conversions, rather than choosing greedily per operation.
The compiler decides execution order, subject to data dependencies. Order affects:
- Peak memory — the maximum number of simultaneously live tensors
- Overlap — whether data movement hides behind compute
- Parallelism — which independent operations can run concurrently
A common technique is to schedule for minimum peak memory when capacity constrained, and for maximum overlap when bandwidth constrained. These goals conflict, and which one wins depends on the workload phase.
| Metric | Why it matters |
|---|---|
| Fusion rate | Fraction of ops fused into larger regions |
| Memory traffic | Bytes moved per token; the number that predicts decode speed |
| Arithmetic intensity | Per fused region; tells you which side of the roofline you're on |
| Peak live memory | Determines whether the model fits |
| Compute utilization | Achieved FLOPs / peak FLOPs |
| Compile time | Developer iteration cost |
| Layout conversion count | Transposes inserted; ideally zero |
| Problem | Root cause | Fix |
|---|---|---|
| Low utilization despite fast kernels | Memory bound — kernels are starved | Fuse more; improve reuse; check intensity |
| Fused region spills | Tile too large for local memory | Reduce tile size; split the fused region |
| Transposes everywhere | Greedy per-op layout choice | Solve layout globally |
| Out of memory at peak | Schedule keeps too many tensors live | Reorder for lower peak; recompute instead of storing |
| Numerics differ after fusion | Different internal accumulation precision | Define and test the tolerance explicitly |
| Compile time explodes | Unbounded fusion search | Bound the search; cache compiled artifacts |
| Fast on one shape, slow on another | Tiles tuned for a single case | Tune per shape bucket |
- Measure intensity before optimizing. Fusing compute-bound regions buys little; fusing memory-bound ones buys everything.
- Fuse aggressively in the memory-bound phase, conservatively where capacity is tight.
- Solve layout globally, not per operation.
- Make the schedule inspectable. Being able to dump the plan — tiles, layouts, buffer assignments — is essential for debugging performance.
- Cache compilation. Recompiling an unchanged model wastes developer time daily.
- Validate numerics after every optimization pass, not just at the end.
- Keep a reference path. An unoptimized execution mode that's known correct makes every performance bug bisectable.
See also: Model Frontend · Kernel Library · Memory Management · Performance Optimization · Inference Software