On the M4 GPU through PyTorch MPS, the first call at a tensor shape it has never seen costs tens of times its steady per-call time. That cost is one-time and cached, it is keyed on the shape rather than the buffer size, and it appears for matmul, softmax, and elementwise alike. The size and behavior are the signature of the Metal pipeline (the MPSGraph / compute pipeline state) being specialized and compiled for that exact shape. It matters for inference: variable-length LLM serving presents a different sequence length on almost every request, so the backend recompiles per length. Padding or bucketing sequence lengths to a small fixed set removes most of that cost.
For a never-seen shape, time the first (cold) call against the steady per-call time, both bracketed with
torch.mps.synchronize() in wall time. The oracle is categorical, not a single measurement: a genuine
compile-then-cache shows first >> steady, a re-run of the same shape back near the steady rate (no second
compile), and every new distinct shape paying again; a mere per-call overhead would make first and steady
similar. Controls: the seen-shape re-run, a matched-output-size different-shape test on all three op types,
and a bucketing sweep. Op types: matmul, softmax, elementwise. Brief GPU bursts only, no sustained load.
The cost is in the compilation band, not the allocation band. Steady per-call time is 0.03 to 0.13 ms and a warm cold call (a shape already seen) is only 1.9 to 5.5x steady, so first-touch buffer allocation, stream or context init, and dispatch launch overhead together live at or below a millisecond here. The first-call tax is 1 to 96 ms, one to two orders of magnitude above that floor, in the range where shader / pipeline-state construction lives and nowhere near where allocation or launch does. This is what separates compilation from generic warmup. The study infers compilation from this behavioral fingerprint (per-new-shape, cached, survives full warmup, survives matched size, millisecond magnitude); it does not read a shader-cache counter.
Median first-call vs steady per-call time, over several never-seen shapes each (torch 2.9.1):
| op | first / steady (median) | first (ms) | steady (ms) |
|---|---|---|---|
| matmul | 20x | 2.2 | 0.126 |
| softmax | 39x | 1.2 | 0.030 |
| elementwise | 34x | 1.4 | 0.040 |
Every op type pays a large first-call tax at a new shape. The median is the robust quantity; individual shapes range wider (matmul 11 to 45x, softmax 32 to 132x, elementwise 23 to 1466x, the high ends being single odd shapes with the heaviest specialization). The absolute compile time also grows with the shape's complexity: a separate sweep of larger matmuls up to 811x809x823 showed cold first calls of 24 to 96 ms against sub-millisecond steady, a 37x to 467x ratio. The ratio and the cached / shape-keyed structure are the load-bearing claims, not a precise millisecond figure.
It is cached and one-time. Re-running a shape that was already seen returns to near the steady rate rather than paying the compile again: the seen-shape re-run is 0.13x of the first-call time (median), the residual gap over steady being ordinary single-call launch overhead, not recompilation. So the cost is paid once per distinct shape and cached thereafter.
It is keyed on shape, not size, on all three op types. For each op, a pair of shapes with the identical output element count (262144) but different shape both pay the full tax, so it is not the buffer size that is being cached:
| op | shape A | shape B | A pays | B pays |
|---|---|---|---|---|
| softmax | 512x512 | 256x1024 | 66x | 48x |
| matmul | 512x512x512 | 256x512x1024 | 25x | 595x |
| elementwise | 512x512 | 256x1024 | 107x | 92x |
A new shape of the same total size is not free, and this holds for matmul, softmax, and elementwise alike. For the reductions (softmax over the last dim) the specialization key includes the reduced width, which is exactly the kind of shape parameter a compiled pipeline bakes in.
The mechanism gives the fix directly: presenting K distinct sequence lengths compiles K pipelines, so
collapsing them to B distinct bucket lengths compiles B. Running a [seq, d] x [d, d] matmul at 20 distinct
sequence lengths compiles 20 pipelines; padding those requests up to 4 fixed bucket lengths compiles 4. The
wall-clock here (29 ms for the 20 distinct compiles vs 6 ms for the 4 bucket compiles, 5.3x) uses larger
bucket shapes and one call each, so it is illustrative of the count collapse rather than a matched
request-for-request measurement; in a real server the 4 bucket compiles amortize across every request, so the
saving is larger still.
| prediction | outcome | |
|---|---|---|
| P1 | first-call >> steady (> 10x) at new shapes | holds. 20x / 39x / 34x median (matmul / softmax / elementwise) |
| P2 | cost is one-time and cached (seen re-run cheap) | holds. seen re-run is 0.13x of first-call, back near steady |
| P3 | triggered by shape not size, across op types | holds. matched-output-size pair both pay on all three ops (25-595x); all three op types pay |
| P4 | bucketing amortizes to about B compilations | holds. 20 distinct shapes to 4 buckets = 4 compiles (5.3x wall-clock here) |
If you serve variable-length sequences on the M4 GPU through MPS, you are recompiling a Metal pipeline on the first request at each new length, a tens-of-milliseconds tax hidden inside that request's latency. Bucket or pad sequence lengths to a small fixed set so the pool of shapes, and therefore compilations, stays small; warm the buckets once at startup. This is a property of the MPS / MPSGraph path this torch build dispatches; other Metal stacks (MLX, llama.cpp Metal) may cache pipelines differently and are out of scope.
M4 GPU via PyTorch MPS (torch 2.9.1), matmul / softmax / elementwise, fp32, single process. The categorical first-vs-steady gap is orders of magnitude, far above timing noise; absolute compile times vary run to run and with shape complexity, so the ratio and the cached / shape-keyed structure are the load-bearing claims. The mechanism is inferred from the behavioral signature and the millisecond magnitude, not from a direct compilation counter.
python run.py # first-vs-steady, cache control, shape-not-size (all 3 ops), bucketing -> results/main.jsonMIT.