v0.0.6

Highlights

Automodel trainers. A new worker runs NVIDIA NeMo Automodel trainers (#296). The worker supports data parallelism and context parallelism (#297). To use more than one GPU for a trainer, set openrl.trainer_gpus=N. The scheduler then starts one torchrun trainer on N GPUs (#307). To split one sequence across some of these GPUs, set openrl.trainer_cp. You can resume an adapter and its optimizer state from a checkpoint (#322).

Exclusive mode. If you set openrl.exclusive=true in user_metadata, the workload gets its own GPUs (#272). No other worker uses these GPUs, and the scheduler does not time-slice them.

Sampler replicas. If you set openrl.sampler_replicas=N, OpenRL starts N samplers (#308). Each sampler uses one GPU. All the samplers take requests from the same sampling queue.

vLLM 0.30. OpenRL now uses vLLM 0.30 (#281). One sampler now serves both LoRA models and FFT models. For delta weight sync, the sampler applies sparse patches with the vLLM function load_checkpoint_weight_patches.

Getting started guide. The new guide, docs/getting-started.md, starts from a GPU cluster (#294). It shows how to train a model on that cluster with SFT and RL.

Breaking changes

• FFT settings (#318). The API server no longer accepts these inputs:

  • the x-open-rl-fine-tuning-type header
  • the x-open-rl-weight-sync-strategy header
  • the full_config body field

  If a request sends one of them, the API server returns a 400 error. The error message gives the name of the new setting. Put the new settings in TINKER_TAGS or in user_metadata:

  Setting	Values
  openrl.fine_tuning_type	lora (default), full
  openrl.weight_sync	delta (default), full
  openrl.fft_seed	int
  openrl.fft_cpu_offload	true / false

• Sampler (#281). This release removes lora_sampler.py and MOCK_VLLM. The weight sync configuration now has one field, strategy. Its value is delta or full.

• Local setup (#285). This release removes make vllm and VLLM_URL. To sample with vLLM, set REDIS_URL. The API server then starts the sampler worker.

Fixes

• The Redis client now waits 30 seconds before a socket read stops (#287). Before this fix, the client could lose a request after Redis removed it from the queue.
• Worker pods now get VLLM_MAX_MODEL_LEN, OPEN_RL_TRAIN_TOKEN_BUDGET, and MAX_JOBS from the deployment (#307).

Examples

The Harvey LAB recipe has these changes:

• New default values for Qwen3.5-9B, and fixes to grading (#319)
• A backend that uses Agent Sandbox (#320)
• A Kubernetes Job that runs the recipe in the cluster (#321)

Full Changelog: v0.0.5...v0.0.6