Skip to content

UniRL v0.2.0

Latest
Latest

Choose a tag to compare

View all tags
@CjhHa1 CjhHa1 released this 08 Oct 08:43
v0.2.0
6fd82cb
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

UniRL v0.2.0 is the first release since the initial v0.1.0 drop and folds in ~350 merged PRs. It focuses on four themes:

  • Much broader model coverage: 12 new model packages — Z-Image, Qwen-Image-Edit-2511, Boogu-Image, LTX-2 / LTX-2.3 (text-to-audio+video), WAN 2.2 V2V, MiniMax-H3 (text-to-video+audio), SenseNova-U1.5, Janus-Pro, Qwen3.5, Qwen3-MoE (VeOmni EP), Qwen3-Omni Thinker, and Cosmos3 — plus new modes for BAGEL and HunyuanImage 3.0.
  • New training modes: supervised fine-tuning (train_sft), multi-turn agentic RL (train_agentic), and disaggregated async RL for both AR (train_async_ar) and diffusion (train_async_diffusion).
  • More algorithms: CPPO, AR DPPO, GSPO, PPO with GAE + value head, DiffusionOPD (on-policy distillation), and a reference-model KL for DiffusionNFT.
  • Current rollout stacks & aligned performance components: rebuilt SGLang Diffusion and vLLM-Omni engines, a new direct vLLM TP engine and FastVideo engine, TP/EP weight sync for full weights and LoRA, unified batched-step diffusion replay (BatchedStepReplayMixin), cross-stage text-embedding deduplication and CPU LRU caching, guarded CUDA MPS reward workers, and fail-closed typed contracts across all layers, all on SGLang 0.5.19 / vLLM-Omni 0.28 / torch 2.13 + CUDA 13.

Models

  • Image diffusion: Z-Image with trainside and SGLang Diffusion rollout (#41, #73); Qwen-Image-Edit-2511 with DiffusionNFT and FlowGRPO (#128, #171); Boogu-Image-0.1 (#219); FLUX.2-Klein image-edit conditioning and ti2i rollout (#79, #242); Qwen-Image SGLang / vLLM rollout engines (#34).
  • Video diffusion: LTX-2 T2V and LTX-2.3 T2AV with a joint audio+video SDE policy (#91, #115, #233); WAN 2.2 V2V (#113); WAN 2.1 and WAN 2.2-A14B on SGLang Diffusion (#183, #184); MiniMax-H3 T2VA Flow-GRPO (#347, #403, #513); HunyuanVideo 1.5 FlashAttention (#427).
  • Unified models: BAGEL-7B-MoT multi-mode RL (it2i / t2t / i2t / it2t / t2ti) and UniGRPO on a single trainside (#62, #100, #148); BAGEL on vLLM-Omni with per-group canvas and sibling run packing (#89, #549, #550); HunyuanImage 3.0 all-modality generation, it2i GRPO, pure-T2I FlowGRPO, VeOmni expert parallelism, stage-pipelined rollout, batched AR recaption, and heterogeneous AR/DiT replica pools (#61, #186, #342, #140, #311, #609); SenseNova-U1.5 pixel flow with OCR recipes (#389, #407).
  • AR / omni: Janus-Pro i2t and t2i (#141); Qwen3.5 text + single-image (#174); Qwen3-Omni Thinker on vLLM-Omni with text, video, audio-in-video, standalone audio and standalone image training (#235, #260, #306, #309, #610).
  • World model: Cosmos3 omnimodal SFT adapters for video prediction and action BC (#200).

Training Modes & Trainers

  • SFT domain covering CE and flow-matching over the shared TrainStack (#209), with agent-conversation, interleaved image-text trajectory, Qwen3-Omni audio, and WAN 2.1 full-transformer SFT (#234, #352, #345, #320).
  • Agentic multi-turn rollout and training on Sample lineages (#214, #315, #363).
  • Disaggregated async trainers: AsyncARTrainer and AsyncDiffusionTrainer, with staleness budgeting, temporally consistent version metrics, dynamic prompt dispatch, and rollout refill before scoring (#68, #192, #310, #289, #213, #540).
  • Semi-independent reward on a dedicated GPU slab (#134); opt-in diffusion role residency (#302, #428).

Algorithms

  • CPPO (#58), AR DPPO (#85), GSPO (#178), AR PPO with GAE and a value head (#254, #256, #259).
  • DiffusionOPD on-policy distillation with multi-teacher and video-KL support (#232, #432); teacher-anchored capability negotiation in ARTrainer (#457).
  • DiffusionNFT under the EMA shadow adapter and with a reference-model KL penalty (#21, #411); Flow-DPPO beta KL (#112); FlowGRPO bounded-memory chunked replay backward (#465); WindowScheduler decay / exp_decay (#489).
  • ReFL / DRaFT-K differentiable-reward training, now hosted under experimental/ (#120, #210, #281).

Rollout Engines & Weight Sync

  • SGLang Diffusion v2 and vLLM-Omni v2 engines replace v1; a canonical SGLang LLM/VLM engine replaces sglang_llm (#35, #36, #74).
  • New direct vLLM TP rollout engine with vLLM-native IPC weight sync (#480); FastVideo rollout engine for WAN 2.1 / 2.2 (#149, #415).
  • SGLang TP/EP full-weight and LoRA sync, plus checkpoint-engine IPC sync (#226, #227, #225).
  • vLLM-Omni prompt-embed cache, sleep lifecycle, and MoE workspace offload on sleep (#449, #370).

Training Backend & Performance

  • Configurable HSDP, fsdp_mode=no_shard, copy-engine all-gather, and VeOmni expert parallelism and Ulysses sequence parallelism (#410, #321, #467, #158, #71).
  • Token-budget packed micro-batches and DP sequence-length balancing (#42, #45, #56); packed-varlen replay for Qwen3 and Qwen-VL (#75, #76).
  • Unified batched-step diffusion replay (BatchedStepReplayMixin) across SD3, Qwen-Image, Qwen-Image-Edit, Z-Image, FLUX.2-Klein, BAGEL, MiniMax-H3, WAN 2.1/2.2, HunyuanVideo 1.0/1.5, and LTX-2/2.3 (#156, #384, #394, #500, #613, #616); BAGEL prompt-KV caching and batched rollouts (#145, #203, #253).
  • In-batch prompt deduplication and shared bounded CPU LRU text-embedding caches across trainside diffusion and video stages (#571, #612, #614).
  • Lower memory and faster step execution: multi-tensor foreach AdamW for all-DTensor FSDP2 parameter groups (#541), CPU-resident eager AdamW step across offload (#441), released grads at the optimizer step (#460), expandable allocator segments for MiniMax-H3 (#583), unhydrated TensorRef anchored rollout outputs before DP scatter (#586), trainer-side skipping of unused text encoders and VAEs for dedicated rollout engines (#619), and a lower expert-gather peak (#422).
  • Guarded NVIDIA CUDA MPS runtime for colocated reward workers (#492) and overlapped remote reward scoring via micro-batched diffusion rollout (#445).
  • Side-by-side train stack layout, shared lifecycle/reporting, and direct typed contracts across trainers, models, rollout engines, reward, and distributed handles (#585, #592, #600, #601, #602, #603, #604, #605, #606, #607).

Rewards

  • New scorers: EditReward, HPSv3++, VideoAlign, CLAP / ImageBind audio rewards, EditScore, and rank-affine managed image scorers (#28, #142, #97, #168, #348, #303).
  • math-verify now runs in a bounded child process (#462).

Checkpointing, Eval & Observability

  • Checkpoint save/load for all trainers, LoRA checkpoints, and DCP sharded checkpoints with async save and an export reader (#33, #66, #83, #157).
  • Periodic eval for PE / ReFL / unified trainers and eval sampling decoupled from the RL rollout (#202, #346).
  • Unified logger, per-phase timing, Perfetto train-step profiler, GPU memory monitoring, and a rollout-replay K3 metric (#24, #37, #160, #179, #187).

Notable Fixes

  • DRPO: deterministic inference no longer clones every sample in a group, which zeroed the DRPO signal, and rollout throughput is restored (#208, #535).
  • AR loss aggregation is mask-aware and grouping-invariant, UnifiedModelTrainStack token-mean is DP-global, and text loss masks are respected (#448, #409, #566).
  • Non-persistent buffer dtypes (e.g. fp32 RoPE inv_freq) are preserved across meta-init and VeOmni wrap (#596); HI3 AR RoPE mode is pinned across replay and autoregress (#555).
  • Advantage normalization stays finite for invalid rewards (#335); diffusion replay keeps transition means at SDE-native fp32 (#444).
  • Distributed rank errors, inconsistent batch/DP shards, and cross-rank frozen teacher mismatches fail closed instead of hanging, and partially initialized AdamW state survives checkpoint resume/export (#258, #440, #575, #579, #582).

Upgrade Notes

  • Dependency stacks moved: SGLang 0.5.12.post1 → 0.5.19; vLLM / vLLM-Omni 0.20.0 → 0.28.0; torch 2.11.0 → 2.13.0, now +cu130 for both engines; transformers >=4.41 → >=5.12,<5.13; diffusers >=0.30 → >=0.38. The SGLang wheel needs glibc ≥ 2.34. See INSTALL.md.
  • Recipes moved: examples are grouped into per-model folders (#84), examples/sft/validation is flattened into examples/sft (#349), and HunyuanVideo 1.0 is now hunyuan_video10 (#387). Update --config-name paths accordingly.
  • Removed APIs: RolloutReq.stage_config / task_config are retired in favor of Part.control (#255, #263); repo-internal compatibility aliases and residual DiffRL identifiers are gone (#366, #498); the v1 SGLang Diffusion / vLLM-Omni engines and sglang_llm are removed (#35, #36, #74).
  • fastvideo extra removed (#573): it pinned a Git commit, which PyPI rejects, and never resolved against transformers>=5.12. Point $FASTVIDEO_PATH at the pinned FastVideo checkout instead (see INSTALL.md).

Full changelog: main was re-rooted on 2026-08-02, so the GitHub compare view between v0.1.0 and v0.2.0 is not available. See all PRs merged between the two releases.

Assets 2
Loading