[RFC] Routing-aware replay recipe and diagnostics for MoE RL

[kaining-never-stop]

[RFC] Routing-aware replay recipe and diagnostics for MoE RL

Summary

I would like to propose a new verl-recipe entry named routing_aware_replay for experimenting with routing-aware replay policies in MoE RL post-training.

The first version is intended to be self-contained and non-invasive. It will provide CPU-testable utilities for Fisher-weighted replay masks, budget-matched replay controls, and routing replay diagnostics. The recipe is designed to complement existing router replay support in verl without changing core trainer behavior.

Motivation

MoE RL post-training often needs to balance two goals:

1. keeping routing behavior stable enough to avoid train-inference mismatch;
2. leaving enough adaptation space for experts that should continue changing during RL.

Existing router replay mechanisms are useful for stabilizing routing behavior, but a uniform replay policy does not always distinguish between experts that should be strongly preserved and experts that can be more freely adapted. For research and debugging, it is also useful to separate the effect of replay budget from the effect of replay selection policy.

This recipe focuses on that gap. It provides a small set of routing-aware replay utilities and diagnostics so users can compare different replay policies under matched budgets.

Proposed location

Repository:

verl-project/verl-recipe

Recipe directory:

routing_aware_replay/

The first version does not require changes to verl core. If maintainers find some interfaces generally useful, parts of the recipe can later be promoted into verl core as separate PRs.

Proposed contents

routing_aware_replay/
├── README.md
├── REQUIRED_VERL.txt
├── routing_aware_replay/
│   ├── fisher_mask.py
│   ├── replay_policy.py
│   ├── diagnostics.py
│   └── schema.py
├── examples/
│   └── synthetic_router_replay_demo.py
└── tests/
    ├── test_fisher_mask.py
    ├── test_budget_matched_replay.py
    └── test_diagnostics_schema.py

Scope of the first PR

The first PR will focus on a minimal, reviewable subset:

• Fisher-weighted expert replay mask computation.
• Budget-matched uniform replay baseline.
• Budget-matched random replay baseline.
• Lightweight routing replay diagnostics.
• Synthetic example that runs without GPUs or private datasets.
• CPU unit tests.
• README and REQUIRED_VERL.txt.

Non-goals

The first PR will not:

• modify verl core trainer behavior;
• require GPU training to verify correctness;
• introduce private datasets or internal platform dependencies;
• claim benchmark-leading results;
• provide a full paper reproduction package.

Relationship to existing router replay support

This proposal is not intended to replace existing router replay support. It is intended to provide a recipe-level space for comparing replay selection policies and collecting diagnostics around replay behavior.

In particular, the recipe will help distinguish:

• the effect of using replay at all;
• the effect of replay budget;
• the effect of choosing which experts/routes to preserve;
• whether the replay policy is too restrictive or too weak.

Testing plan

The initial PR should be testable with:

pytest routing_aware_replay/tests
python routing_aware_replay/examples/synthetic_router_replay_demo.py

The tests will validate:

• mask shape and dtype;
• deterministic behavior under fixed seeds;
• budget matching between Fisher-weighted, uniform, and random controls;
• diagnostic schema consistency.

Draft first PR

If the direction looks acceptable, I can start with a small PR titled:

[recipe] add routing-aware replay utilities for MoE RL

The PR would add only the self-contained recipe utilities, README, REQUIRED_VERL.txt, synthetic demo, and CPU tests.

Open questions

1. Should this live only in verl-recipe, or should a small diagnostics interface be added to verl core later?
2. Is routing_aware_replay an acceptable recipe name?
3. Should the first version include only CPU-testable utilities, or also include a small MoE training config?
4. What upstream router replay output schema should the recipe align with?

[kaining-never-stop]

I prepared a minimal draft implementation in verl-recipe for easier review: verl-project/verl-recipe#114

The draft only includes CPU-testable utilities, README, REQUIRED_VERL.txt, a synthetic demo, and unit tests. It does not modify verl core. Happy to rename the recipe or reduce the scope if maintainers prefer a different direction.