Release v0.1.3

What's Changed

Major Update

• feat(rollout): enable transparent distributed rollout with shared-memory Experiences, avoiding controller memory bottlenecks and improving scalability for large and multimodal rollouts by @hijkzzz in #22

Rollout Reliability

• fix rollout correctness after context compaction and chat-history rewrites by @fus3r in #16 and #20
• improve distributed rollout reliability with persistent warm-resume buffers and safer Ray resource allocation by @hijkzzz in bce1cc5 and c34d88f

Training and Performance

• fix evaluation sample limits and MoE auxiliary-loss scaling with context parallelism by @hijkzzz in e56cbf9 and 0ae27eb
• add token-weighted advantage monitoring by @k21993 in #23
• improve vLLM GPU placement and rollout-engine defaults for Omni3-30B and Qwen3-4B by @hijkzzz in e3d84a3 and 8ef1f9a

Packaging

• add PyPI publication for molt-rl, update AutoModel dependencies, and adopt Trusted Publishing by @hijkzzz in 48136c6, f9226e4, and 736bc61

Full Changelog: v0.1.2...v0.1.3

Release v0.1.2

What's Changed

This release mainly contains bug fixes.

Features

• feat(algo): add --no_whiten raw-returns advantage and independent critic knobs by @hijkzzz in e5f7647
• feat(recipe): make rl_qwen3_6_35b PPO-configurable by @hijkzzz in a016f4e
• feat(observability): log adaptive KL coefficient to status by @k21993 in #14

Fixes

• fix(critic): initialize value head with proper initializer_range to avoid vanishing gradients by @hijkzzz in 1796697
• fix(agents): isolate tokenize on a dedicated thread pool to prevent deadlocks by @hijkzzz in c2ac1aa
• Revert "fix(rollout): keep in-flight routing across refit (pause clear_cache=False)" to fix KV cache inconsistency and training drift by @hijkzzz in 546d9bc

Refactors

• refactor(loss): dispatch policy surrogates via a registry (mirrors advantage estimators) by @hijkzzz in #13

Tests

• test(gae): restore temporary sys.modules stubs to fix order-dependent collection by @k21993 in #6
• test(advantage): add coverage for group estimators (grpo, dr_grpo, reinforce_baseline, rloo) by @k21993 in #8

CI / Chore / Docs

• chore(ci): add Claude code review workflow by @svcnemo-autobot in #9
• chore(docker): bump vLLM from 0.24.0 to 0.25.1 by @hijkzzz in df44869
• docs(readme): improve installation flow and recommend Docker setup by @hijkzzz in 21c1b89

Other

• CISPO loss by @Athe-kunal in #12

New Contributors

• @k21993 made their first contribution in #6
• @svcnemo-autobot made their first contribution in #9
• @Athe-kunal made their first contribution in #12
• @hijkzzz made their first contribution in #13
• @fus3r made their first contribution in #15

Full Changelog: v0.1.1...v0.1.2

Release v0.1.1

This release mainly contains bug fixes.

🐛 Bug Fixes

• Make chat turns retry-safe — commit trajectory only after generate succeeds (f98fd1a) — @hijkzzz & @claude
• Token-exact TITO rollout with compaction segments (ab77ad3) — @hijkzzz & @claude
• Merge routed_experts when only some samples captured routing (#5) (a992877) — @khazic
• Correct GLM-5.2 recipe: no CP, EP256, 32 nodes (33e5ece) — @hijkzzz & @claude
• Drop dead --rollout.rescore_logprobs; fix async off-policy remedy text (a95608d) — @hijkzzz & @claude
• Reject half-written checkpoints, require extra_state.pt (2d12843) — @hijkzzz & @claude
• Back-fill turn-boundary token's routing (R3) (7709cba) — @hijkzzz & @claude
• Forward caller positional args in launchers (snapshot $@ before set --) (a8689ad) — @hijkzzz & @claude

Other Changes

• Refactor: tighten mixed-routing merge fill (aba01d4) — @hijkzzz & @claude
• Test: cover multi-turn VLM image carry + new-image-in-delta (7a0c330) — @hijkzzz & @claude
• Chore: bump version 0.1.0 -> 0.1.1 (433e98f) — @hijkzzz & @claude

Release v0.1.0

Molt v0.1.0 — first public release 🦋

Molt is an agentic-first, PyTorch-native RL post-training framework: the agent is the program you write, the trainer is a single trainable actor, and the reward is arbitrary Python. Three components carry the whole runtime — Ray (placement + async queue), vLLM (rollout), and NVIDIA AutoModel + FSDP2 (training) — in ~9K lines of core RL code, no Megatron, that still scale to hundred-billion-parameter MoE.

📄 Technical report (DOI): 10.13140/RG.2.2.23375.65447 · 📖 DeepWiki · License: Apache-2.0

---

✨ Highlights

• Agentic-first. Multi-turn tool use, multimodal (VLM) environments, and LLM-as-judge are first-class — the agent is a Gymnasium-style Env / ChatAgent, the reward is any Python you write.
• PyTorch-native, no Megatron. Ray + vLLM + NVIDIA AutoModel/FSDP2. ~9K lines you can read end to end.
• Scales. TP · EP · CP (· DP) — verified on Qwen3.5-397B, scaling toward GLM-5.2 753B, and DeepSeek-V3 at --fsdp.ep_size 256.
• Train/inference consistency built in. First-class vllm_kl mismatch metric + importance-sampling correction, plus MoE router replay (R3) and router freeze.
• Fully-async rollout with a vLLM router, partial rollout, and a one-flag switch back to strictly on-policy.

🧩 What's included

Training paradigms

• SFT and the full RL stack, one CLI each (molt.cli.train_sft, molt.cli.train_rl_ray).
• Estimators: reinforce, reinforce_baseline, rloo, grpo, dr_grpo, gae (PPO), on_policy_distill.
• Optional KL reference workers (the reference doubles as the distillation teacher).

Agent contract

• One Python module per run (--train.agent_path): choose Env (framework owns the LLM loop) or ChatAgent (you own it).
• Rewards, graders, tools, VLM environments, and LLM-as-judge — all plain Python; judge calls route back through the same vLLM engines.

Scale & parallelism

• FSDP2 with TP / EP / CP (· DP), comm/compute overlap, optimizer and full CPU offload.
• 1T-class MoE on vLLM (DeepSeek-V3 at EP=256).

MoE routing stability

• R3 (--train.routing_replay): replay vLLM's per-token expert selection in the training forward.
• Router freeze (--actor.freeze_moe_router).

Off-policy / async correctness

• vllm_kl metric for the rollout-vs-train logprob gap.
• IS correction: --algo.advantage.is_correction_level {off,token,seq,geo} × --algo.advantage.is_correction_mode {mask,clip,trunc} (covers TIS, IcePop, seq-mask-tis).
• Async rollout (--train.async_queue_size), partial rollout (--train.partial_rollout_enable).

Modalities & templates

• Text and VLM (image) prompts, multi-turn.
• Assistant-span / loss-mask derivation from the model's own chat template — verified on Qwen3.x, Nemotron Omni, Kimi, GLM, Gemma, and DeepSeek.

🚀 Getting started

Prebuilt image (full CUDA-13 stack — torch, vLLM, TransformerEngine, flash-attn, mamba, DeepEP, AutoModel):

docker pull hijkzzz/molt:latest   # or the pinned tag hijkzzz/molt:0.1

Minimal RL run:

python3 -m molt.cli.train_rl_ray \
  --actor.model_name_or_path /path/to/automodel \
  --data.prompt_dataset /path/to/prompts.jsonl \
  --train.agent_path examples/python/agents/math.py \
  --vllm.num_engines 2 --vllm.tensor_parallel_size 2 \
  --rollout.batch_size 128 --train.batch_size 128 \
  --algo.advantage.estimator reinforce \
  --fsdp.attn_implementation te \
  --ckpt.output_dir ./ckpt/rl

Ready recipes from Qwen3-4B to 397B/753B-scale live under examples/scripts/.

📚 Links

• Docs / full README: https://github.com/NVIDIA-NeMo/labs-molt
• Technical report (DOI): 10.13140/RG.2.2.23375.65447
• DeepWiki (auto-indexed Q&A): https://deepwiki.com/NVIDIA-NeMo/labs-molt

📝 Notes

• First public release; APIs may still change. Issues and PRs welcome (commits must be signed off, git commit -s, per the DCO).
• Qwen3.5-397B is verified end to end; GLM-5.2 753B is the upper end of ongoing scaling.

Full details: see the README. Apache-2.0.