Skip to content

NVIDIA NeMo-RL 0.8.0

Latest
Latest

Choose a tag to compare

View all tags
@nemo-automation-bot nemo-automation-bot released this 06 Nov 22:22
v0.8.0
5e012b4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

🚀 Release v0.8.0

✨ Highlights

Container

Both linux/amd64 and linux/arm64 Docker containers are available on NGC as nvcr.io/nvidia/nemo-rl:v0.8.0.

Here are the major software components included in the container:

Software Component Version
NeMo-RL 0.8.0
NeMo-Gym 0.7.0+1f8afc7
NeMo-Automodel 0.6.1+72dacef
Megatron-Bridge 0.7.0+e0b5630
Megatron-Core 0.20.0+a694348 (via Megatron-Bridge)
PyTorch 2.13.0 (2.11.0 in the SGLang and TensorRT-LLM environments)
vLLM 0.29.0
SGLang 0.5.17.dev30+g3003d70f6
TensorRT-LLM 1.3.0rc21
Ray 2.58.0
Transformers 5.12.1

The NeMo-RL container is built on top of nvcr.io/nvidia/cuda-dl-base:26.05-cuda13.2-devel-ubuntu24.04 and uses Python 3.13.14.

If you would like to build this container, or nightly containers, yourself, we provide the exact instructions we use at https://docs.nvidia.com/nemo/rl/latest/docker.html#release-image.

Single Controller (beta)

NeMo RL v0.8 adds the Single Controller, a new async RL runtime (#3219, #3220, #3266, #3267, #3427). One CPU-only Ray actor runs two loops at the same time: a rollout loop that generates one prompt group at a time, and a train loop that trains as soon as enough groups are ready. The two loops share the TransferQueue data plane. Finished groups go into a replay buffer, and a sampler picks which groups each step trains on, so rollout batching is no longer tied to the training batch.

Compared with the existing async GRPO path, the Single Controller keeps the GPUs busier in two ways:

  • Streaming: training starts as soon as async_rl.min_groups_for_streaming_train prompt groups are ready, instead of waiting for the full batch, so training and generation overlap more. Set it to num_prompts_per_step to keep full-batch steps.
  • Samplers: in_order matches the existing async GRPO path (each dispatched batch trains at its planned step). weight_fifo, ready_first, and windowed train on groups as they finish, so the trainer waits less; ready_first also uses late groups instead of throwing them away. The right sampler for your recipe depends on how much asynchrony (training on rollouts from older weights) your training can tolerate.

The Single Controller implements streaming as described in the StreamRL paper: instead of waiting for a full batch of rollouts (panel c), the trainer starts on groups as they finish (panel d). Figure 5 from StreamRL (Zhong et al., arXiv:2504.15930, CC BY-SA 4.0):

StreamRL Figure 5: how streaming reduces pipeline bubbles

async_rl:
  min_groups_for_streaming_train: 8   # below num_prompts_per_step turns on streaming
  sampler:
    name: ready_first                 # in_order | weight_fifo | ready_first | windowed | custom
    max_staleness_versions: 1

How each sampler picks groups from the same buffer:

image

What it supports in v0.8:

  • Megatron training backend only. Automodel training on the Single Controller is not supported yet and is coming in the next release.
  • GRPO and PPO (#3773), with five samplers: in_order, weight_fifo, ready_first (#3582), windowed, and custom. PPO supports only in_order without streaming.
  • vLLM or Megatron Inference for rollouts, colocated or not for Megatron Inference (#3727, #3730, #4009), plus NeMo Gym rollouts, Router Replay (R3) (#3378), VLM GRPO, and MOPD (#3768).
  • Keeps training when things fail (#3589, #3590, #3591, #3592, #3665). Failed rollouts are retried within a budget or dropped. If a generation shard dies, the run rebuilds its refit group over the shards that are left and keeps going. Dead shards can be restarted and rejoin at the next refit. All of this is off by default; see async_rl.rollout_failure, async_rl.generation_fleet_health, and async_rl.generation_router.
  • Checkpoint and resume, including rollouts (#3429, #3480, #3923, #3924, #3925, #3898). Checkpoints keep finished rollouts that were not trained on yet, and re-run prompt groups that were still in flight. Periodic rollout snapshots save progress between trainer checkpoints. Works with both the simple and mooncake_cpu data-plane backends.
  • Exact token capture (#3837, #4129, #4191). For NeMo Gym rollouts, the exact tokens and logprobs produced by inference are captured and used for training, with no retokenization. Works with vLLM and Megatron Inference; images and video are captured on vLLM.
uv run examples/run_grpo_single_controller.py --config examples/configs/grpo_math_1B_megatron_single_controller.yaml

The Single Controller needs data_plane.enabled: true and (for vLLM) async_engine: true with non-colocated generation. Validation, reward shaping, and dynamic sampling are not supported yet. Example recipes:

See the Single Controller guide for the required settings, the samplers, and the full list of supported features.

TensorRT-LLM Rollout Backend

NeMo RL v0.8 adds TensorRT-LLM as a rollout engine (#2420, #3130, #3418, #3537). With TensorRT-LLM and Dynamo (also new in v0.8), NeMo RL now has five rollout engines: vLLM, SGLang, Megatron Inference, TensorRT-LLM, and Dynamo. Set policy.generation.backend: trtllm. It works with both Megatron and Automodel training, colocated or non-colocated, and supports in-flight weight updates. An OpenAI-compatible HTTP server lets it drive NeMo Gym agentic rollouts such as SWE.

policy:
  generation:
    backend: trtllm
    trtllm_cfg:
      tensor_parallel_size: 1
      async_engine: true   # required
      gpu_memory_utilization: 0.6

TensorRT-LLM runs in its own trtllm uv environment. It was validated by matching vLLM convergence on math and SWE agentic RL; throughput tuning comes in a later release. Example recipes:

Faster Weight Refit: NCCL Reshard

Refit is the step that copies new weights from training to the rollout engine. NeMo RL v0.8 adds NCCL reshard refit (refit_transport: nccl_reshard) for non-colocated runs (#2971, #3477, #3739). Each training shard sends its large FFN and expert weights straight to the matching rollout shard, so no rank ever builds the full tensor. Small tensors still use the existing packed path. It works from Megatron to vLLM and from Megatron to Megatron Inference, including BF16 training with MXFP8 rollouts.

policy:
  generation:
    colocated:
      enabled: false
    refit_transport: nccl_reshard

Each weight slice crosses the network once, and generation GPUs in the same NVLink domain share it over NVLink. Refit latency with matched configurations:

Model GPU Broadcast (NCCL) NCCL reshard
Qwen3-235B-A22B H100 6.4 s 1.9 s
DeepSeek-V3 671B H100 14.9 s 2.6 s
DeepSeek-V3 671B GB200 6.1 s 2.3 s

In a full run of the DeepSeek-V3 async performance recipe on H100, refit time dropped from 15.3 s to 3.1 s and step time from 33.2 s to 21.0 s (#4870). Most async performance recipes now use it (#3865, #4869, #4870), for example grpo-deepseek-v3-64n8g-async-1off.yaml.

v0.8 also adds two more refit transports. Each transport is picked with the single policy.generation.refit_transport key, so only one can be active.

NIXL checkpoint-engine refit

NIXL refit sends full weights over a UCX/RDMA fabric (#2608). On DeepSeek-V3 it was 1.54× faster than NCCL refit (9.25 s vs 14.21 s). It needs vLLM generation and works with Automodel or Megatron training.

policy:
  generation:
    colocated:
      enabled: false
    refit_transport: nixl
    refit_cfg:
      nixl:
        update_weights_bucket_memory_ratio: 0.05
        device: cuda
        backend_name: UCX
        release_after_refit: false
        shard_expert_weights: true

Example config: grpo_math_8B_megatron_nixl.yaml. See the checkpoint-engine refit guide for UCX settings, sharded experts, and FP8.

nixl-refit

Sparse delta refit

Sparse delta refit sends only compressed weight deltas, for setups where the link between training and rollout is slow (#2444). vllm_zmq_sparse sends them through a ZeroMQ relay over TCP, and vllm_s3_sparse sends them through S3. It needs vLLM generation, Megatron training, and BF16/FP16 rollouts, and it works with GRPO only.

policy:
  generation:
    colocated:
      enabled: false
    refit_transport: vllm_zmq_sparse  # or vllm_s3_sparse
    refit_cfg:
      sparse:
        delta_compression:
          encoding: xor
        storage:
          s3_bucket: null  # required for vllm_s3_sparse

Example recipe: grpo-qwen3-30ba3b-4n8g-megatron-zmq-deltaweight-noncolocated.yaml. See the sparse delta refit design doc.

The new refit guide explains how to pick a transport. See also the NCCL reshard design doc.

sparse-delta-refit

Megatron Inference Rollouts

Megatron Inference (policy.generation.backend: megatron) is in active development, and v0.8 adds many features to it.

Colocated Megatron training and inference. Training and rollouts can share the same GPUs. New in v0.8:

  • Async GRPO with colocated Megatron Inference (#2884). Before, async GRPO needed non-colocated generation.
  • Colocated reshard: rollouts can use a different parallel layout from training on the same GPUs, for example training with TP4/CP2 and generating with TP4/EP4 (#3490).
  • Colocated Megatron Inference also works with the Single Controller (#3730) and NeMo Gym (#3406), and now builds CUDA graphs (#3718).
policy:
  generation:
    backend: megatron
    colocated:
      enabled: true
    refit_transport: mcore
    mcore_generation_config:
      refit_backend: nccl
      tensor_model_parallel_size: 4   # rollout layout, can differ from training
      expert_model_parallel_size: 4

Other additions:

  • MXFP8 rollouts from a BF16 trainer (#3731).
  • Multimodal generation for Nemotron Omni images and video, matching vLLM accuracy (#3920).
  • HTTP frontends on every model-parallel coordinator with prefix-cache-aware routing, for NeMo Gym (#3912), and the async scheduler on by default (#3762).
  • Logprobs after top-p/top-k processing, so top-p/top-k runs are correct (#3392).
  • NCCL refit in place of the nvshmem copy service: ~16–19 s per step and flat on a 30B-A3B MoE, where nvshmem grew from ~95 s to over 135 s (#3166).

Example recipe: grpo-nanov3-30BA3B-2n8g-megatron_generation-colocated-reshard-async-gym.yaml. See generation design doc.

Low-Precision Rollouts: MXFP8 and NVFP4

MXFP8 rollouts

Training stays in BF16, and vLLM converts the weights to MXFP8 at every refit, then rolls out in MXFP8 (#1887, #3652, #3477, #3478, #3630). It works with NCCL reshard refit, and batching the MoE weight shuffle cut the transfer time on Qwen3-30B-A3B from 5.9 s to 0.89 s (#3478). The shipped recipes keep the LM head, attention, and the first and last few layers in BF16, and use importance-sampling correction.

policy:
  generation:
    vllm_cfg:
      precision: "fp8"
      is_mx: true
      quantization_ignore_patterns:
        - model.layers.*.self_attn.*
        - model.layers.*.mlp.gate
      num_first_layers_in_bf16: 3
      num_last_layers_in_bf16: 18
    vllm_kwargs:
      moe_backend: flashinfer_trtllm

There are 10 MXFP8-rollout performance recipes for Qwen3-30B-A3B, Qwen3-32B, Qwen3-235B, DeepSeek-V3, and Nemotron 3 Super, sync and async, for example grpo-qwen3-235b-32n4g-async-1off-mxfp8-rollout.yaml. See the FP8 / MXFP8 guide.

End-to-end NVFP4 training with NVFP4 rollouts

On Blackwell, NeMo RL v0.8 can train routed-expert MLPs using NVFP4 (W4A4) forward computation with Transformer Engine and roll out in NVFP4 with matching per-token activation scaling (#3566, #3654). Persistent policy weights and refit payloads remain BF16, optimizer state remains FP32, and backward computation currently uses TE’s dequantized path. Attention, routers, shared experts, and selected boundary layers remain in BF16. On Qwen3-30B-A3B-Base over 900 steps with Router Replay, NVFP4 tracks BF16 on reward, validation accuracy, and response length, while:

  • rollout throughput is 1.40× higher and step time is 14.2% lower
  • end-to-end throughput is 1.15× higher
  • rollout weights are 68% smaller, with 2.62× the KV-cache capacity

Training throughput is lower (0.84×), and refit takes longer.

NVFP4 vs BF16, Qwen3-30B-A3B-Base, 900 steps

uv run examples/run_grpo.py --config examples/configs/recipes/llm/grpo-qwen3-30ba3b-base-8n4g-megatron-te-nvfp4-pertoken.yaml

See the NVFP4 per-token rollout design doc.

Quantization-aware RL with real NVFP4 rollouts for MoE and Mamba

Quantization-aware RL (QARL) with ModelOpt now supports real NVFP4 rollouts (W4A16 and W4A4) for MoE and hybrid Mamba models, including Qwen3-30B-A3B, Nemotron 3 Nano, and Nemotron 3 Super (#2983, #3308). On Nemotron 3 Super over 300 steps, W4A16 matches BF16 reward (~0.65), and W4A4 reaches ~0.58 while generating about 2.4× faster than BF16. An opt-in GPU-side export cuts step time by ~22–25% (#3308). The Nemotron 3 Super real-quant recipes currently fail on this release; see Known Issues.

QARL Nemotron 3 Super reward: BF16 vs W4A16 vs W4A4

See the Quantization-Aware RL guide.

PPO: Non-Colocated, Async, and Critic Warm Start

PPO, added in v0.7, grows several new modes in v0.8:

  • Non-colocated generation (#3262): vLLM runs on its own GPUs while the policy and critic share the training GPUs. Set policy.generation.colocated.enabled: false.
  • Async PPO (#3410): generation runs all the time while the policy and critic train, with staleness bounded by ppo.async_ppo.max_trajectory_age_steps. Sync and async PPO show no regression in reward, validation accuracy, or logprob error.
  • Critic warm start (#3821, #4213): ppo.warm_start_value_checkpoint starts the critic from a separately pretrained value checkpoint, so you don't pay for critic warmup in every run.
  • Separate critic epochs (#3829): ppo.critic_ppo_epochs.
  • Faster GAE (#4156): when gae_gamma and gae_lambda are both 1.0, GAE uses a vectorized path. On a 196K-token SWE PPO batch this saved about 20 s per step.
  • PPO on the Single Controller (#3773).

Sync (blue) vs async (yellow) Megatron PPO on Qwen2.5-1.5B GSM8K, from #3410:

Sync vs async PPO

Example recipes:

See the PPO guide.

Nemotron 3 Ultra Post-Training

In v0.7, the Nemotron 3 Ultra recipe lived on the separate ultra-v3 branch. NeMo RL v0.8 brings the Nemotron 3 Ultra (550B-A55B) post-training recipe to main (#3276, #3529, #3452). It includes seven stage configs (student RLVR 1 and 2, MOPD, and the reasoning, IFBench, RLHF, and SWE teachers) and a single launcher. With EXTERNAL_JUDGES=1, the GenRM and NL2Bash judges run as separate vLLM pools next to training instead of inside the training Ray cluster.

EXTERNAL_JUDGES=1 bash examples/nemo_gym/nemotron-3-ultra/ultra_launch.sh

The training data blends are on Hugging Face at nvidia/Nemotron-RL-Ultra-Training-Blends. See the Nemotron 3 Ultra guide.

Nemotron Omni: Multimodal Agentic RL

NeMo RL v0.8 adds training for the Nemotron Omni models on the Megatron backend (Nemotron 3 Nano Omni GRPO on the Automodel backend shipped in v0.7; see the Automodel section of the Nano Omni guide):

  • Nemotron Nano Omni image GRPO (#3290).
  • Nemotron Super Omni GRPO with MTP (#3494): MTP training and MTP speculative decoding with context parallelism. Over 50 steps on 16 nodes, mean reward went from 0.45 to 0.53, with about 75% draft acceptance.
  • Multi-turn, multi-image, tool-calling GRPO with NeMo Gym (#3414), sync and async.
  • Video in NeMo Gym, sync and async (#3648).
  • Super Omni image MOPD (#3430) and the MPO preference algorithm for Omni Nano and Super (#3345).
  • Multimodal payload deduplication (#3527): grpo.deduplicate_multimodal_data: true stores each image or video once instead of once per generation, which cut rollout-to-policy data by about 15× at 16 generations per prompt.

Multi-turn tool-calling GRPO on Nemotron Omni, from #3414:

Nemotron Omni multi-turn tool calling

Example recipes:

See the Nemotron 3 Nano Omni guide, the Super Omni MOPD guide, and the Omni MPO guide.

SFT v2 with Megatron-Energon

NeMo RL v0.8 adds SFT v2, a single-controller SFT loop with Megatron-Energon multimodal data loaders (#3917), plus Energon-side sequence packing (#4105). The existing SFT loop is unchanged (data.backend defaults to hf). SFT v2 currently supports the Megatron backend with data.backend: energon.

uv run --locked examples/run_sft_v2.py --config examples/configs/recipes/vlm/vlm_sft-qwen2.5-vl-3b-instruct-clevr-1n2g-megatrontp1-energon.v1.yaml

SFT v1 (blue) vs SFT v2 (orange) loss on Qwen2.5-VL-3B CLEVR, from #3917:

SFT v1 vs v2

See the SFT guide and its SFT v2 / Energon section.

Model Support

Nemotron

Additional Models

DeepSeek V4 Flash GLM-5.2 MiniMax-M3
DeepSeek V4 Flash GRPO GLM-5.2 GRPO MiniMax-M3 GRPO

For the full list, see the model support page and the per-family model guides (#2900).

⚡ Performance Optimizations

  • HybridEP for MoE on H100, B200, and GB200: MoE performance recipes now use HybridEP for expert-parallel communication, and the container builds it with multi-node support (#3436, #3438, #4118, #4037, #4038). On H100, the Nemotron 3 Super policy training step got 30% faster and the Qwen3-30B-A3B one 9% faster. On GB200, Qwen3-30B-A3B end-to-end steps got 11–13% faster. Automodel recipes use backend.dispatcher: hybridep.
  • vLLM CUDA graphs in release and nightly recipes: 10 recipes that ran vLLM in eager mode now use CUDA graphs, making generation 48–85% faster (#4568, #4569).
  • Single-forward sequence-logprob filtering: loss_fn.seq_logprob_error_in_loss: true applies the sequence-logprob filter inside the training forward pass, removing a separate forward pass. This cut mean step time by 17.8% on Qwen2.5-Math-1.5B (#4171).
  • FlashInfer TRT-LLM MoE for BF16 on Blackwell: BF16 MoE performance recipes on GB200 use moe_backend: flashinfer_trtllm, making Qwen3-235B steps 9–10% faster (#3461). H100 recipes stay on Triton (#4850).
  • Chunked linear fusion for GRPO: computes token logprobs from hidden states chunk by chunk, without building the full logits tensor, for GRPO and the other policy-gradient losses (#2833).
  • Lower logprob memory: no full-vocab softmax on the inference-only path (#3484), selected-token logprobs computed directly (#3930, #3124), and an option to run packed microbatches largest first (sequence_packing.microbatch_order: largest_first), which cut peak reserved memory from 158.5 GiB to 71.2 GiB on a Nemotron Nano run (#3476).
  • Megatron memory options: fine-grained activation offloading (#2279), offloading part of the optimizer state to CPU (#3628), and training CUDA graphs (#3483).
  • Async checkpointing for Megatron: checkpoint writes run in the background (#2828). This is now on in the base Megatron configs.
  • Faster startup and less per-step overhead at scale: Ray GCS tuning (#3089, #3158), batched worker port discovery (#2920), NeMo Gym startup that overlaps the generation model load (#3499, #3569), a GPU-free Ray head node option (DEDICATED_RAY_HEAD=1) (#3388), and less NeMo Gym logging and payload overhead (#3106, #3292, #3390, #3918).
  • Cross-tokenizer distillation: lower memory and faster loss for top-k KD and x-token P-KL (#3314, #3564).
  • Data plane: sync GRPO on the TransferQueue data plane with mooncake_cpu went from 21.5 s to 17.8 s per step, close to the 16.8 s without a data plane (#4580).

View the performance numbers from our published recipes in the performance summary.

🚫 Breaking Changes

DTensor v1 removed, and the DTensor backend is renamed to Automodel

In v0.4 we marked DTensorPolicy (v1) for deprecation. NeMo RL v0.8 removes it, and renames the remaining DTensor v2 backend to Automodel, after the NeMo Automodel library it is built on (#4198, #4270, #4273, #4276, #4282, #4283, #4277, #4701).

# v0.7                         # v0.8
policy:                        policy:
  dtensor_cfg:                   automodel_cfg:
    _v2: true    # delete          enabled: true
    enabled: true
  • Rename dtensor_cfg to automodel_cfg in policy, value, teacher, teachers[i], and env.reward_model, and delete _v2. The contents are unchanged. Old configs fail at startup with a message that names the fix.
  • DPO and RM users: check your backend. In v0.7, _v2 defaulted to false, and dpo.yaml and rm.yaml did not set it, so DPO and RM runs used DTensor v1 by default. In v0.8 they run on Automodel, which can change memory use, speed, numerics, and the checkpoint layout.
  • DTensor v1 checkpoints cannot be resumed in v0.8. Convert them to Hugging Face format with examples/converters/convert_dcp_to_hf.py from NeMo RL v0.7.x, then start from the HF weights.
  • Worker classes and modules are renamed (for example DTensorPolicyWorkerV2 → AutomodelPolicyWorker), the fsdp uv extra is removed, and recipe files with fsdp2 or dtensor in their names now use automodel (78 recipes; W&B run names change with them).

Other changes that need action

  • Automodel checkpoint keys moved (#3935): checkpointing.model_save_format and checkpointing.save_consolidated now live under policy.automodel_cfg.checkpoint. save_consolidated is now a string: "false", "final", or "every" (old true → "every"). Old keys raise an error at setup.
  • Fused-logprob keys renamed (#2833): policy.megatron_cfg.use_linear_ce_fusion_loss → use_fused_linear_logprobs, and linear_ce_fusion_chunk_size → fused_linear_logprobs_chunk_size. There is no startup check for the old names: a config that inherits sft.yaml or dpo.yaml silently runs with fusion off.
  • Megatron Inference (#3611, #3739, #3609): mcore_generation_config.async_engine is removed; Megatron Inference always uses the async engine, and setup fails if the key is set. Non-colocated Megatron Inference now refits through NeMo RL's packed collective by default. To use Megatron-Core's own refit, set refit_transport: mcore; the nvshmem refit backend is broken and warns.
  • AIME eval dataset names (#2928): aime2024, aime2025, aime2026 are now AIME2024, AIME2025, AIME2026. The old names fail with an unrelated file error.
  • Custom multi-reward environments (#2597) must return a dict of named rewards ({"reward/<name>": Tensor}) instead of a [B, K] tensor.
  • Multimodal prompt fields (#3803): vllm_images, vllm_audios, and vllm_videos are replaced by one vllm_multi_modal_data dict. This affects custom data processors and environments.
  • Data plane config (#2935): data_plane.storage_capacity and num_storage_units moved under data_plane.simple, and global_segment_size and local_buffer_size moved under data_plane.mooncake_cpu. Old flat keys are silently ignored. mooncake_cpu is now RDMA-only (no TCP mode), and its default memory sizes are smaller.
  • SGLang (#3612): the HTTP weight-transfer path is removed, and non-colocated Automodel + SGLang is rejected at setup. disable_outlines_disk_cache now defaults to true because of a CVE (#4797).
  • Python config objects (#2518, #2526, #2527): inner algorithm configs are now pydantic models. Custom code that reads master_config.grpo["key"] must use attribute access (master_config.grpo.key). YAML configs are unaffected.
  • Container: PyAV is no longer in the shipped image (#4184), joining opencv-python-headless, which was removed in v0.7. Run bash tools/install_audio_deps.sh if you need them.

Default changes

  • Megatron async checkpointing (megatron_cfg.checkpoint.async_save: true) and use_gloo_process_groups: false are now set in the base Megatron configs (#2828, #2746).
  • policy.offload_policy_before_refit defaults to false (#3300). Set it to true if you run out of memory at refit.
  • Async GRPO fails on the first generation-worker error (grpo.async_grpo.max_generation_failures: 0) instead of hanging (#2368).
  • logger.wandb.log_nemo_gym_full_result_tables defaults to false; v0.7 logged these tables by default (#3000).
  • Router Replay now skips MTP-layer routers; set NRL_ROUTER_REPLAY_EXCLUDE_MTP=0 for the old behavior (#3903).
  • PPO advantages now leave out overlong-filtered samples from GAE, matching GRPO (#3262).
  • Service ports moved below 9000, out of the kernel's ephemeral port range (#3103). Check firewall rules if your cluster restricts ports.

Notable Additions

  • Full-vocabulary on-policy distillation: on_policy_distillation.full uses the exact full-vocabulary reverse KL instead of the sampled-token gap, with multiple teachers, on the Single Controller (#3978, #4045). See the MOPD algorithm page.
  • Managed Dynamo generation backend (backend: dynamo) on Slurm (#3391). See the Dynamo guide.
  • SGLang: Megatron refit over NCCL for non-colocated SGLang rollouts, and opt-in engine fault tolerance (#3612, #3613).
  • External vLLM service pools in the same Slurm allocation, for NeMo Gym judges (#3386, #3587).
  • GDPO: per-reward weights (grpo.adv_estimator.reward_weights) and multi-reward NeMo Gym environments (#2680, #2597).
  • Validation controls for GRPO: grpo.val_start_at, validation-only sampling settings, grouped pass@k (grpo.val_num_generations_per_prompt), and early stop at a validation threshold (grpo.stop_at_validation_metric) (#3400, #3401, #3404).
  • Reward and advantage penalties for malformed NeMo Gym outputs (reward_penalties.*), and an option to stop masking environment-flagged samples (env.should_mask_flagged_samples) (#2885, #3734, #3402).
  • EAGLE-3 draft training: sequence packing (about 2× policy training throughput on Qwen3-30B-A3B), a separate draft optimizer and grad clipping, and a streaming soft-CE loss (#3463, #3323, #3701, #3703, #3707). See the EAGLE-3 guide.
  • LoRA: warm-start from a LoRA checkpoint (lora_cfg.restore_from, peft.restore_from) and per-expert adapters (peft.share_expert_adapters) (#3874, #4234). See the LoRA guide.
  • More low-precision options: simulated KV-cache quantization in QARL (#3012), a TE precision config file for per-module quant recipes (#3654), custom FP8 quantizer factories (#3998), and an FP32 LM head (fp32_lm_head) (#4096).
  • OpenTelemetry tracing through nemo-lens, off by default (#3655, #4052). See the observability docs.
  • Mooncake on the data plane: RDMA backend, GPUDirect RDMA (data_plane.mooncake_cpu.use_gdr), and data-plane checkpoints (#2935, #3501, #3898). VLM GRPO can also use the data plane (#2957).
  • vLLM refit options: refit through vLLM's reload_weights API (#3651, #4063), pausing generation during in-flight refit (#3839), and dropping colocated weights while asleep (discard_weights_on_sleep) to avoid host OOM (#4433).
  • Megatron config hooks: megatron_cfg.model_overrides (#3535), custom worker extensions (worker_extension_cls_fqn) (#3809), and tokenizer_kwargs (#3798).
  • Container can run as a non-root user: nemo-runtime (UID/GID 65532), with uv and its Python moved out of /root. The image still runs as root by default (#4260).
  • Fault-tolerance checkpoint retention: ft_save_period / ft_keep_latest_k (#3157).
  • New datasets: NuminaMath-1.5 (#3417) and OpenR1-Math-220k (#3131).
  • NeMo Gym is now the main evaluation path (#4687). See evaluation. examples/run_eval.py still works for quick checks.
  • Automodel context parallelism: refactored to use model-specific sequence sharding, with aligned loss inputs, automatic padding, and corrected gradient scaling (#3498). See the CP integration guide.
  • First shipped in v0.8: the v0.7.0 release notes listed these PRs, but they merged after the v0.7 branch was cut, so v0.8.0 is the first release that contains them: MXFP8 rollouts (#1887), the Qwen2.5-Omni audio + video recipe (#2823), Megatron async checkpointing (#2828), chunked linear fusion for GRPO (#2833), behavior penalties (#2885), batched port discovery (#2920), Ray GCS tuning (#3089, #3158), port-range fixes (#3103), lower Automodel logprob memory (#3124), the OpenR1-Math-220k dataset (#3131), and fault-tolerance checkpoint retention (#3157).
  • Major dependency upgrades: vLLM 0.20.0 → 0.29.0, PyTorch 2.13.0, Transformers 5.12.1, Ray 2.58.0, NeMo-Gym 0.4 → 0.7, NeMo-Automodel 0.6.1, Megatron-Bridge 0.7.0, NCCL 2.30.7, and Python 3.13.14 (#3280, #4080, #4415, #4828, #3498, #4852, #3453, #3183).

Notable Fixes

  • Fixed GRPO advantage spikes (near −2075) when use_leave_one_out_baseline and normalize_rewards are both on, which is the default in grpo_math_1B.yaml. This mostly showed up on easy data where all but one sample in a group were correct (#4158).
  • Fixed token positions filtered out by top-k/top-p still counting in the loss and KL (#4154).
  • Fixed refit sending stale weights: for Automodel non-colocated refit with large MoE experts (reward 0.1–0.2 before, 0.81 after on Nemotron 3 Nano Omni) (#4720), for Megatron with overlap_param_gather (#4086), and for vLLM's flashinfer_trtllm BF16 MoE backend (#3545, #3659).
  • Fixed the Automodel KL reference model moving to the resumed weights on every resume, instead of staying at the base weights (#3722).
  • Fixed the Automodel MoE router correction bias being cast to BF16 during refit, which could change expert routing in vLLM (#3540).
  • Fixed calibrated FP8 KV-cache scales being dropped on colocated refit (#3226), and GLM-5.2 FP8 sparse-attention scoring that made training and generation disagree (#4667).
  • Fixed async GRPO skipping about 30% of prompts across checkpoint save and restore (#3599), ignoring max_num_epochs (#3248), ignoring recompute_kv_cache_after_weight_updates without in-flight updates (#3152), and deadlocking on resume (#3066).
  • Fixed run_vlm_grpo.py always running sync GRPO even with async enabled (#3740).
  • Fixed EAGLE-3 draft gradients shrinking the policy update; train/grad_norm is now policy-only (#3323).
  • Fixed the Megatron optimizer updating random weights with CPU offload and the distributed optimizer (#2372), and gradient finalize being skipped for Nemotron Omni (#3789).
  • Fixed wrong eval pass@k when an answer could not be parsed (#3635).
  • Fixed a pipeline-parallel hang in MoE aux-loss metrics (#3419), a GPU memory leak after async checkpointing with colocated generation (#3322), and Mooncake port collisions with the kernel's ephemeral range (#4103).
  • Fixed DeepSeek-V3 with Transformers 5 (#4408).
  • Addressed security vulnerabilities and CVEs (#4343, #4415, #4736, #4766, #4885, #4924, #4970, #4988).

🛠️ Known Issues

  • LoRA with activation checkpointing on Automodel: with activation_checkpointing: true, the LoRA weights never reach vLLM, so rollouts come from the base model (#4470). Turn off activation checkpointing when using LoRA on Automodel.
  • vLLM async scheduling can hang large MoE runs with data parallelism (vllm-project/vllm#43547). The GLM-5.1 and Qwen3.5-397B recipes set policy.generation.vllm_kwargs.async_scheduling: false to avoid it (#4354, #4794). Use the same setting if a large MoE run with vLLM DP > 1 hangs in generation.
  • Qwen3.5 MoE with the Megatron backend and expert parallelism can still hang in vLLM's sample_tokens (vllm-project/vllm#36237). The Automodel path no longer hangs after the vLLM upgrade.
  • Quantized Nemotron 3 Super NVFP4 (W4A16/W4A4) recipes currently fail (#4585).
  • Automodel cannot load Nemotron-H checkpoints because of an embedding key name mismatch (#4211). This waits on an Automodel update.

What's Changed

New Contributors

Full Changelog: v0.7.0...v0.8.0

Contributors

yfw, ananthsub, and 78 other contributors
Assets 2
Loading