## Summary
- Add `trainer.warmup: bool = True` to the RL trainer. After the startup
weight broadcast, the trainer does its one-time forward and backward
work while the orchestrator generates the first batch: `torch.compile`,
kernel JIT and NCCL communicator setup.
- Move the step's forward and backward into a `forward_backward`
function. The training step and the warmup both call it.
- The warmup runs `forward_backward` on three synthetic micro batches.
Their loss mask is empty, so they train nothing, like a dropped micro
batch. They carry the inputs that the config adds to real micro batches,
such as routed experts for router replay and score-centering supports.
The three micro batches are:
- one at full `seq_len`, rounded up to the micro batch multiple, which
compiles the static graph;
- one at the longest length below full, which compiles the dynamic
graph. Its 32-bit indexing guards come from the first length that the
graph sees, so every shorter real length passes them;
- one at the same length with a document of half its length, because a
long document changes the graphs of some layers.
- The warmup does not step the optimizer. Its gradients stay in their
buffers, and `optimizer.zero_grad()` discards them, so the weights and
the optimizer state stay unchanged. Optimizer state setup and
compilation still happen in step 1.
- The warmup logs `time/warmup` and its peak memory. The peak is a worst
case: a full-length micro batch with a long document can use more memory
than the first real batches. An OOM then shows in the first minutes
instead of at the first long batch.
## Verification
All runs used H200 nodes. "Cold" means fresh Inductor and Triton cache
directories.
> [!NOTE]
> The DeepSeek-V4-Flash runs below also took a dry optimizer step at the
end of the warmup, which this PR does not do. The "optimizer" column
shows that step's share of the warmup time. Without it, the optimizer's
first-step cost moves to step 1. Step 1 then ends later by that cost,
but at most by the time the warmup finished before the first batch
arrived.
**Unit tests:** the CPU tests in `tests/unit/train/rl/test_warmup.py`
pass in CI.
**Debug model, 1 node:** after the warmup, step 1 forward/backward took
6.2 s against 4.6 s steady. No compiles ran after the warmup.
**DeepSeek-V4-Flash, trainer only, 1 node, fake data, `seq_len`
131072.** One run per trainer setup:
| setup | warmup | of which optimizer | step-1 fwd/bwd | step-2 fwd/bwd
|
|---|---|---|---|---|
| base (CP 4, EP 8, optimizer offload) | 4m37s | 34 s | 6.3 s | 4.7 s |
| `model.offload = "full"` | 2m37s | — | 7.5 s | 6.4 s |
| `model.offload = "none"` | 2m24s | 7 s | 5.2 s | 3.5 s |
| HSDP (`dp_replicate` 2, EP 4) | 3m18s | 28 s | 9.4 s | 6.1 s |
In each setup, no recompile and no cuDNN JIT ran after the warmup. With
`offload = "full"`, step 1 logged the same entropy and mismatch KL as
the base run, so the warmup left the weights unchanged. The warmup also
completed with router replay, with score centering and with CP 1. The
fake data loader cannot train these setups, so no real step ran after
their warmup. These runs share one compile cache, so they show which
compiles the warmup covers, not cold compile times.
**DeepSeek-V4-Flash, trainer only, 4 nodes, CP 4, FP8, warmup on:**
| | warmup | of which optimizer | step-1 fwd/bwd | step-2 fwd/bwd |
|---|---|---|---|---|
| cold | 10m48s | 3m20s | 21.9 s | 19.3 s |
| warm cache | 4m38s | 59 s | — | 19.1 s |
**DeepSeek-V4-Flash RL, 5 nodes (1 inference, 4 trainer), cold, same
nodes, run back to back.** Times are from the startup weight broadcast:
| | warmup off | warmup on |
|---|---|---|
| first batch ready | 6m41s | 6m44s |
| warmup (of which optimizer) | — | 11m31s (3m26s) |
| trainer idle, waiting for the batch | 6m41s | 0 s |
| step-1 / step-2 fwd/bwd | 637 s / 209 s | 61 s / 28 s |
| **step 1 done** | **18m42s** | **13m57s** |
| **step 2 done** | **22m47s** | **15m01s** |
The warmup does about the same compile work as the cold first steps, but
it overlaps it with the first batch's generation. Here the warmup
finished 4m47s after the first batch arrived, which is longer than its
optimizer step. Without the optimizer step, step 1 would end at about
the same time. With a warm compile cache, the warmup fits inside the
batch generation.
Peak memory in the RL run: the warmup peaked at 132.0 GiB, and real
steps peaked at 117–123 GiB, against 141 GiB on an H200. The extra
memory comes from the warmup's forward/backward, most likely the
long-document micro batch.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches the critical training startup path and distributed
forward/backward, but synthetic batches have zero loss tokens so
optimizer updates should not change weights; warmup can still raise peak
memory or OOM before the first real step.
>
> **Overview**
> Adds **`trainer.warmup`** (default **on**) so the RL trainer can run
one-time compile/JIT/communicator work on **synthetic micro batches**
while the first real batch is still being generated, instead of paying
that cost on step 1.
>
> The training loop’s forward/backward path is **extracted into a
reusable `forward_backward` helper** (with knobs for optimizer overlap
and trace annotations). On the **first step**, after startup weight
broadcast, warmup calls that same helper on **three shaped batches**
(full padded length, near-full, and near-full with a long document) with
**empty loss masks** so weights and optimizer state stay unchanged;
warmup time and peak memory are logged as **`time/warmup`**.
>
> New **`prime_rl.trainer.rl.warmup`** builds those batches (including
optional router-replay and score-centering fields). Unit tests cover
micro-batch length planning and synthetic batch layout.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
1e97b95583e338ee48ac549f93cbf14b0d158d2b. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>