Skip to content
Sudhanshu edited this page Nov 11, 2026 · 1 revision

ORPO — Separate Preference Optimization Experiment

ORPO is an alternative legacy --mode orpo, not a continuation of SFT/DPO. The script constructs a new model/LoRA adapter and applies the five chosen/rejected pairs using ORPOTrainer.

flowchart TD
    Start["Fresh legacy execution"] --> Mode["--mode orpo"]
    Mode --> Check["Pinned stack and CUDA checks"]
    Check --> Model["SmolLM + LoRA adapter"]
    Model --> Data["5 chosen/rejected preference pairs"]
    Data --> Trainer["ORPOTrainer: likelihood + odds-ratio term"]
    Trainer --> Output["orpo_adapter + tokenizer"]
    Separate["NOT run after SFT → DPO"] -.-> Mode
Loading

The code specifies ORPOConfig(beta=0.1), batch size 1, gradient accumulation 2, default 3 epochs, and saves llm_training_output/orpo_adapter/.

The repository does not implement PPO, reward-model training, or a scalable human-feedback service. With only five preference pairs, training loss is not a useful quality or safety benchmark.

Clone this wiki locally