Skip to content

Carry rollout MoE routing from the vLLM sampler to the maxtext trainer - #2046

Merged
copybara-service[bot] merged 1 commit into
mainfrom
mohit/router-replay-plumbing
Sep 1, 2026
Merged

Carry rollout MoE routing from the vLLM sampler to the maxtext trainer#2046
copybara-service[bot] merged 1 commit into
mainfrom
mohit/router-replay-plumbing

Conversation

@khatwanimohit

@khatwanimohit khatwanimohit commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Lets an RL run replay the MoE expert routing a rollout actually used instead of re-running the router during training. Both ends already existed -- vLLM and tpu-inference capture routing (enable_return_routed_experts -> CompletionOutput.routed_experts, [length, num_layers, top_k]) and MaxText consumes it (forced_routed_experts) -- so this is the middle.

Sampler side:

  • VllmConfig.return_routed_experts sets vLLM's engine arg; VllmSampler.detokenize collects the arrays and SamplerOutput carries them.
  • The in-process adapter populates SamplingResponse.routed_experts, and warns when a request asks for routing the engine was not built to capture, since capture is engine-level and the alternative is silently handing the trainer None -- indistinguishable from a dense model.

Pipeline side:

  • routed_experts on TokenSegment, TrajectoryItem and RLTrainerPayload, plus UNSET_ROUTED_EXPERT = -1 for slots the trainer must not replay.
  • The GRPO adapter pads short captures with the sentinel rather than expert 0.
  • The padded assembler lays routing out over [prompt | completion] the same way the token ids are padded: prompts right-aligned keeping the tail (as _left_pad does), completions left-aligned. Getting this wrong detaches every replayed expert from its token -- silent, not a crash.

Demo side (math_gsm8k_dist), all opt-in:

  • trainer node: --trainer_engine=maxtext swaps PeftTrainer for MaxText's engine wired for replay, importing MaxText lazily and skipping the tunix actor model, which only knows the demo's dense architectures. --maxtext_load_parameters_path points it at the checkpoint; without one it warns, since random trainer weights make replayed routing meaningless.
  • rollout node: --return_routed_experts, and --load_rollout_weights to load the real checkpoint instead of vLLM's dummy weights. With both sides on the same weights a first step is meaningful with no weight transfer at all.
  • launcher.sh and k8s_launcher.sh: ROUTER_REPLAY=1 sets the above; slices and meshes are now configurable rather than hardcoded.
  • Dockerfile: opt-in INSTALL_MAXTEXT build arg, which also drops triton (GPU-only, and its native module aborts the process when MaxText imports on a TPU host).

Tests pin layout rather than mere presence, since the realistic failure is misalignment. The loss assertions are differential -- two different replays must disagree -- because a "loss is finite" check passes just as happily when the routing is dropped. Each was verified to fail against a mutation that disables what it covers.

Checklist

  • I have added all the necessary unit tests for my change.
  • I have verified that my change does not break existing code and all unit tests pass.
  • I have added all appropriate doc-strings/documentation.
  • My PR is based on the latest changes of the main branch (if unsure, rebase the code).
  • I have signed the Contributor License Agreement.
  • I have followed Contribution Guidelines.

Comment thread tests/experimental/worker/router_replay_trainer_worker_test.py Outdated
Comment thread tunix/experimental/common/datatypes.py Outdated
Comment thread tunix/experimental/examples/math_gsm8k_dist/k8s_launcher.sh Outdated
Comment thread tunix/experimental/examples/math_gsm8k_dist/launcher.sh Outdated
Comment thread tunix/experimental/examples/math_gsm8k_dist/launcher.sh Outdated
Comment thread tunix/experimental/examples/math_gsm8k_dist/run_trainer_node.py Outdated
Comment thread tunix/experimental/examples/math_gsm8k_dist/run_trainer_node.py Outdated
Comment thread tunix/experimental/orchestrator/batch_assembly.py
Comment thread tunix/experimental/orchestrator/batch_assembly.py Outdated

@tianshub tianshub left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the update, looks much cleaner.

Comment thread tunix/rl/common.py Outdated
@khatwanimohit
khatwanimohit force-pushed the mohit/router-replay-plumbing branch from 2d57fc3 to cc68986 Compare September 1, 2026 04:35
@khatwanimohit
khatwanimohit force-pushed the mohit/router-replay-plumbing branch from cc68986 to 000fc96 Compare September 1, 2026 15:47
@khatwanimohit
khatwanimohit force-pushed the mohit/router-replay-plumbing branch from 000fc96 to 6527bb1 Compare September 1, 2026 17:45
Lets an RL run replay the MoE expert routing a rollout actually used, instead of
re-running the router during the training forward pass. Both ends already
existed -- vLLM/tpu-inference capture routing
(`enable_return_routed_experts` -> `CompletionOutput.routed_experts`, shaped
`[length, num_layers, top_k]`) and MaxText consumes it
(`forced_routed_experts`, AI-Hypercomputer/maxtext#4826) -- so this is the
middle, and it covers both the `tunix/rl` and `tunix/experimental` stacks.

Opt-in end to end, and a no-op on dense models and on any model whose
`__call__` does not accept the kwarg.

Capture (shared by both stacks):
- `VllmConfig.return_routed_experts` sets vLLM's engine arg;
  `VllmSampler.detokenize` collects the arrays into `SamplerOutput`.
- `RolloutConfig.return_routed_experts` turns it on from either stack's config.
  The CLI passes rollout config through by dataclass field name, so this needs
  no CLI change.

`tunix/experimental`:
- `routed_experts` on `TokenSegment`, `TrajectoryItem` and `RLTrainerPayload`,
  plus `UNSET_ROUTED_EXPERT = -1` for slots the trainer must leave to the
  router -- `-1` and not `0`, because expert 0 is a real expert.
- The in-process adapter fills `SamplingResponse.routed_experts`, and warns when
  a request asks for routing the engine was not built to capture, since capture
  is engine-level and silently returning None is indistinguishable from a dense
  model.
- The GRPO adapter pads short captures with the sentinel; the padded assembler
  lays routing out over `[prompt | completion]` the same way the token ids are
  padded.

`tunix/rl`:
- `RolloutOutput.routed_experts`, `TrainExample.routed_experts`, and
  `common.align_routed_experts` for the same padding rules. The prompt /
  completion split comes from the unpadded completion length, since both sides
  are padded and the widths alone cannot recover it.
- `compute_per_token_logps` forwards the routing only to models that advertise
  the kwarg, reusing the `model_call_contains` gate already used for
  `segment_ids`. This is the single point where replay reaches the model, and
  both stacks share it: the experimental orchestrator sets
  `algo_core.grpo_loss_fn` as its loss, which calls straight into it.

Replay is all-or-nothing per batch, so a batch cannot mix replayed and freshly
routed rows.

Tests pin the padding layout rather than merely that a field is non-None, since
the realistic failure is misalignment rather than absence, and the loss
assertions are differential -- two different replays must disagree, because a
"loss is finite" check passes just as happily when the routing is dropped. The
differential runs against a real MaxText MoE (Qwen3.5 shrunk to CPU size,
random init, no checkpoint) and was verified to fail when the threading is
removed.

Two existing GRPO loss tests stub `train_example` with a bare MagicMock and set
each optional field to None explicitly; `routed_experts` is set the same way,
matching how `segment_ids` and `sampler_is_weights` are handled there.
@khatwanimohit
khatwanimohit force-pushed the mohit/router-replay-plumbing branch from 6527bb1 to 2b8603b Compare September 1, 2026 18:13
@copybara-service
copybara-service Bot merged commit a265c06 into main Sep 1, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants