v1.5.0: Any Post-Training
17 Algorithms — Any Post-Training
oxRL now ships 17 algorithms covering the full post-training landscape.
New in v1.5.0
Supervised Learning (12 algorithms):
| Algorithm | Description |
|---|---|
| SFT | Supervised Fine-Tuning |
| DPO | Direct Preference Optimization |
| ORPO | Odds Ratio Preference Optimization |
| KTO | Kahneman-Tversky Optimization |
| CPT | Continued Pre-Training (all-token loss) |
| KD | Knowledge Distillation (CE + KL) |
| RM | Reward Model Training (Bradley-Terry) |
| OnlineDPO | DPO with online rejection sampling |
| RFT | Rejection Sampling Fine-Tuning |
| SPIN | Self-Play Improvement |
| IPO | Identity Preference Optimization (squared loss) |
| SimPO | Simple Preference Optimization (reference-free) |
Reinforcement Learning (5 algorithms):
| Algorithm | Description |
|---|---|
| SGRPO | Stable GRPO with clipped surrogate loss |
| CISPO | Clipped importance-sampling policy optimization |
| PPO | Proximal Policy Optimization with GAE + value clipping |
| RLHF | GRPO with trained reward model |
| RLAIF | GRPO with AI-generated rewards |
Other Changes
oxrl/algs/__init__.pynow exports all 14 algorithm classes- 9 new example configs in
registry/examples/ - README updated: honest VLM support (VLA claim removed), complete algorithm tables
- 64 tests passing (was 33)
- New config fields:
simpo_gamma,ppo_vf_clip,ppo_tau,ppo_gamma,kd_alpha,kd_temperature,reward_threshold,reward_model_path
Breaking Changes
oxrl/algs/simpo.pyrewritten from Ray actor to SL-compatible DeepSpeed algorithmoxrl/algs/ppo.pyrefactored to self-initialize (like GRPO)