pgw#989: the dynamo mint's warmup_forward hour gets a breakdown, and inductor_compile: 0.0 stops lying

[PaulFidika]

The number

Every published JIT cell reports this shape. sdxl w8a8-lora64, L40S, gen-worker 0.93.1, pod wz4g1ya8a4khne, cell ck5-a53e02a7885f9312fb1fa7df…, 2026-08-06 — read off the hub's own worker_activity_events, not a log:

{'load': 13.87, 'warmup_forward': 4416.91, 'inductor_compile': 0.0,
 'seal_publish': 96.809, 'finalize': 0.155}

97.6 % of the mint is one row, and the row named after the work is zero.

The zero is not a broken clock. mint_child opened the inductor_compile frame around _drain_router, and a fleet mint arms COLD with no router (gw#587) — so that phase measures an empty queue, while every compile runs INLINE inside the warm forwards. warmup_forward is therefore a bucket holding compile AND execution with no way to tell them apart, which is exactly the shape you cannot optimise or regression-test: a compile-bound mint and a warm plan running too many forwards look identical.

What changes

• the drain frame is router_drain — what it is. PHASE_INDUCTOR_COMPILE keeps its real meaning on the AOT path (aot_mint) and in the executor.
• gen_worker.warm_spans — a ledger around the warm loop. Per job and in total: warm_compile_s vs warm_execute_s (an honest residual), the inductor partition inside the compile half, and compile_other_s so a newly-introduced phase shows up as the residual growing rather than as time vanishing (pgw#830's rule, applied to the JIT path).
• MintReport.mint_phases, documented as "empty for the dynamo recipe, which has no per-graph-class breakdown", now carries it. The parent re-emits under jit_compile / warm:totals, warm:<member>, warm_job:<name> — same kind, so an AOT-vs-JIT comparison stays ONE grouped query.

Why a second key set and not the AOT one

MEASURED on the pin (torch 2.13.0+cu130), one torch.compile call, 5.06 s wall:

ledger	attributed	note
AOT partition (aot_compile_spans)	1.104 s (22 %)	host_compile_s = 0.000 — AotCodeCompiler.compile is AOT-only
_compile.compile_inner	5.054 s (99.9 %)	the JIT total
↳ PyCodeCache.load_by_key_path	3.203 s (63 %)	where a JIT compile's time actually is
↳ ↳ async_compile.wait	2.886 s	overlay, never summed in

Reusing the AOT key set would have priced this mint at a fifth of its cost and called the rest residual. That containment is asserted as a test row, so a pin bump that moves it fails loudly.

Identity

No cell is re-keyed. mint_child, mint_delegate, activity and the new warm_spans are all outside compile_cache.static_code_closure (verified against the live 147-file closure), and nothing here touches the traced graph, the seal, or codegen. Telemetry only: every path is exception-swallowing and cannot fail a mint.

Tests

tests/test_warm_spans_pgw989.py — 9 rows at the real seams, no card and no compile required, over a frozen REAL metric delta:

• the partition reconciles to _compile.compile_inner with a named residual;
• the AOT key set is shown to miss 78 % of the same compile;
• overlays are reported and never summed into the partition (parallel_kernel_cpu_s legitimately exceeds the wall — it prices the async workers' CPU);
• a failing warm job keeps its seconds (pgw#825's rule: a mint that died in the warm plan is exactly the one whose spent minutes must reach the hub);
• an unmeasurable wall OMITS compile_fraction rather than reporting "compiled 0 %";
• the drain phase no longer calls itself a compile, with one definition re-exported where the vocabulary lives;
• the parent emits the rows, and stays silent when the recipe produced no ledger.

On origin/master: warm_spans does not exist, mint_child carries 2 phase="inductor_compile" frames, mint_delegate has no _emit_warm_ledger, activity has no PHASE_ROUTER_DRAIN.

Verification

• tests/ 3302 passed, 37 skipped, 1 xfailed (-n 4 --dist loadfile)
• mypy clean (232 files), ruff clean, HTTP-timeout guard clean, unreached-surface guard unchanged (19 pre-existing, none new)

Note for the release lane

This is master-only. master carries the wire-v1 protocol cut and cannot talk to the deployed pre-v1 hub, so do not fork a wheel off this — if the attribution is wanted on a pod before the next minor, it should ride the attempt-24 lane's 0.93.2 patch off v0.93.1 rather than a second patch line.

[PaulFidika]

Rebased onto origin/master (#512/#513/#514 landed). One conflict, in mint_child: pgw#984 factored the warm loop into _drive_warm_plan, so the ledger moved inside it rather than being re-applied at the call site. That is the better home — the AOT recipe's pgw#984 proof forward now gets measured too, and an unmeasured cost is exactly how this one hid.

Suite re-run after the rebase: 3318 passed, 37 skipped, 1 xfailed; mypy clean (232 files), ruff clean.

This also rides release/0.93.3 (cut from the v0.93.2 tag, pre-v1 verified) alongside pgw#992, so the next pod mint reports honest phase numbers.