Carry rollout MoE routing from the vLLM sampler to the maxtext trainer - #2046
Merged
Conversation
khatwanimohit
requested review from
abheesht17,
hgao327,
jiangyangmu,
lc5211,
s-noghabi,
sizhit2,
tianshub and
wang2yn84
as code owners
August 31, 2026 17:31
tianshub
requested changes
Aug 31, 2026
khatwanimohit
force-pushed
the
mohit/router-replay-plumbing
branch
from
August 31, 2026 22:11
9316f27 to
4e8baca
Compare
khatwanimohit
force-pushed
the
mohit/router-replay-plumbing
branch
from
August 31, 2026 22:20
4e8baca to
94a5a69
Compare
tianshub
approved these changes
Aug 31, 2026
tianshub
left a comment
Collaborator
There was a problem hiding this comment.
Thank you for the update, looks much cleaner.
khatwanimohit
force-pushed
the
mohit/router-replay-plumbing
branch
from
August 31, 2026 23:44
94a5a69 to
2d57fc3
Compare
khatwanimohit
force-pushed
the
mohit/router-replay-plumbing
branch
from
September 1, 2026 04:35
2d57fc3 to
cc68986
Compare
khatwanimohit
temporarily deployed
to
testing
September 1, 2026 04:35 — with
GitHub Actions
Inactive
khatwanimohit
temporarily deployed
to
testing
September 1, 2026 04:35 — with
GitHub Actions
Inactive
khatwanimohit
temporarily deployed
to
testing
September 1, 2026 04:35 — with
GitHub Actions
Inactive
khatwanimohit
temporarily deployed
to
testing
September 1, 2026 04:35 — with
GitHub Actions
Inactive
khatwanimohit
force-pushed
the
mohit/router-replay-plumbing
branch
from
September 1, 2026 15:47
cc68986 to
000fc96
Compare
khatwanimohit
force-pushed
the
mohit/router-replay-plumbing
branch
from
September 1, 2026 17:45
000fc96 to
6527bb1
Compare
khatwanimohit
temporarily deployed
to
testing
September 1, 2026 17:46 — with
GitHub Actions
Inactive
khatwanimohit
temporarily deployed
to
testing
September 1, 2026 17:46 — with
GitHub Actions
Inactive
khatwanimohit
temporarily deployed
to
testing
September 1, 2026 17:46 — with
GitHub Actions
Inactive
khatwanimohit
temporarily deployed
to
testing
September 1, 2026 17:46 — with
GitHub Actions
Inactive
Lets an RL run replay the MoE expert routing a rollout actually used, instead of re-running the router during the training forward pass. Both ends already existed -- vLLM/tpu-inference capture routing (`enable_return_routed_experts` -> `CompletionOutput.routed_experts`, shaped `[length, num_layers, top_k]`) and MaxText consumes it (`forced_routed_experts`, AI-Hypercomputer/maxtext#4826) -- so this is the middle, and it covers both the `tunix/rl` and `tunix/experimental` stacks. Opt-in end to end, and a no-op on dense models and on any model whose `__call__` does not accept the kwarg. Capture (shared by both stacks): - `VllmConfig.return_routed_experts` sets vLLM's engine arg; `VllmSampler.detokenize` collects the arrays into `SamplerOutput`. - `RolloutConfig.return_routed_experts` turns it on from either stack's config. The CLI passes rollout config through by dataclass field name, so this needs no CLI change. `tunix/experimental`: - `routed_experts` on `TokenSegment`, `TrajectoryItem` and `RLTrainerPayload`, plus `UNSET_ROUTED_EXPERT = -1` for slots the trainer must leave to the router -- `-1` and not `0`, because expert 0 is a real expert. - The in-process adapter fills `SamplingResponse.routed_experts`, and warns when a request asks for routing the engine was not built to capture, since capture is engine-level and silently returning None is indistinguishable from a dense model. - The GRPO adapter pads short captures with the sentinel; the padded assembler lays routing out over `[prompt | completion]` the same way the token ids are padded. `tunix/rl`: - `RolloutOutput.routed_experts`, `TrainExample.routed_experts`, and `common.align_routed_experts` for the same padding rules. The prompt / completion split comes from the unpadded completion length, since both sides are padded and the widths alone cannot recover it. - `compute_per_token_logps` forwards the routing only to models that advertise the kwarg, reusing the `model_call_contains` gate already used for `segment_ids`. This is the single point where replay reaches the model, and both stacks share it: the experimental orchestrator sets `algo_core.grpo_loss_fn` as its loss, which calls straight into it. Replay is all-or-nothing per batch, so a batch cannot mix replayed and freshly routed rows. Tests pin the padding layout rather than merely that a field is non-None, since the realistic failure is misalignment rather than absence, and the loss assertions are differential -- two different replays must disagree, because a "loss is finite" check passes just as happily when the routing is dropped. The differential runs against a real MaxText MoE (Qwen3.5 shrunk to CPU size, random init, no checkpoint) and was verified to fail when the threading is removed. Two existing GRPO loss tests stub `train_example` with a bare MagicMock and set each optional field to None explicitly; `routed_experts` is set the same way, matching how `segment_ids` and `sampler_is_weights` are handled there.
khatwanimohit
force-pushed
the
mohit/router-replay-plumbing
branch
from
September 1, 2026 18:13
6527bb1 to
2b8603b
Compare
khatwanimohit
temporarily deployed
to
testing
September 1, 2026 18:13 — with
GitHub Actions
Inactive
khatwanimohit
temporarily deployed
to
testing
September 1, 2026 18:13 — with
GitHub Actions
Inactive
khatwanimohit
temporarily deployed
to
testing
September 1, 2026 18:13 — with
GitHub Actions
Inactive
khatwanimohit
temporarily deployed
to
testing
September 1, 2026 18:13 — with
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets an RL run replay the MoE expert routing a rollout actually used instead of re-running the router during training. Both ends already existed -- vLLM and tpu-inference capture routing (
enable_return_routed_experts->CompletionOutput.routed_experts,[length, num_layers, top_k]) and MaxText consumes it (forced_routed_experts) -- so this is the middle.Sampler side:
VllmConfig.return_routed_expertssets vLLM's engine arg;VllmSampler.detokenizecollects the arrays andSamplerOutputcarries them.SamplingResponse.routed_experts, and warns when a request asks for routing the engine was not built to capture, since capture is engine-level and the alternative is silently handing the trainer None -- indistinguishable from a dense model.Pipeline side:
routed_expertsonTokenSegment,TrajectoryItemandRLTrainerPayload, plusUNSET_ROUTED_EXPERT = -1for slots the trainer must not replay.[prompt | completion]the same way the token ids are padded: prompts right-aligned keeping the tail (as_left_paddoes), completions left-aligned. Getting this wrong detaches every replayed expert from its token -- silent, not a crash.Demo side (math_gsm8k_dist), all opt-in:
--trainer_engine=maxtextswaps PeftTrainer for MaxText's engine wired for replay, importing MaxText lazily and skipping the tunix actor model, which only knows the demo's dense architectures.--maxtext_load_parameters_pathpoints it at the checkpoint; without one it warns, since random trainer weights make replayed routing meaningless.--return_routed_experts, and--load_rollout_weightsto load the real checkpoint instead of vLLM's dummy weights. With both sides on the same weights a first step is meaningful with no weight transfer at all.launcher.shandk8s_launcher.sh:ROUTER_REPLAY=1sets the above; slices and meshes are now configurable rather than hardcoded.INSTALL_MAXTEXTbuild arg, which also drops triton (GPU-only, and its native module aborts the process when MaxText imports on a TPU host).Tests pin layout rather than mere presence, since the realistic failure is misalignment. The loss assertions are differential -- two different replays must disagree -- because a "loss is finite" check passes just as happily when the routing is dropped. Each was verified to fail against a mutation that disables what it covers.
Checklist