v2.48.0: run LLMPPO actor and critic as separate or fused passes, picked by memory estimate

Features

• run LLMPPO actor and critic as separate or fused passes, picked by memory estimate

  • LLMPPO runs the actor (policy and KL loss) and the critic (value loss) as separate forward/backward passes per micro-batch, so peak activation memory is one batch of rows, not two. This covers both the Liger and unfused loss paths.
  • New fuse_actor_critic_pass: bool | None = None on LLMPPO and LLMPPOSpec. True runs one fused pass, False runs two. None fuses when the agilerl.arena.memory estimate of the fused pass fits the GPU, and fuses on non-CUDA devices. The choice is re-resolved on checkpoint load.
  • On the Liger path the critic pass runs under the critic adapter, so the value loss trains the critic LoRA.
  • The critic adapter stays trainable after FSDP2 replaces the model's parameters.
  • With activation_offload=True, the split Liger actor pass also offloads saved activations to CPU.
  • The fused-pass memory estimate uses the packed-expert LoRA path the model actually runs.
  • PPO learn reports actor and critic gradient norms before and after clipping.
  • agilerl-arena memory estimator: TrainingSettings gains fuse_actor_critic_pass (PPO only) and micro_batch_size.
  • agilerl now requires agilerl-arena>=1.18.0.

Full Changelog: v2.47.0...v2.48.0