Repository navigation
ORPO
ORPO is an alternative legacy --mode orpo, not a continuation of SFT/DPO. The script constructs a new model/LoRA adapter and applies the five chosen/rejected pairs using ORPOTrainer.
flowchart TD
Start["Fresh legacy execution"] --> Mode["--mode orpo"]
Mode --> Check["Pinned stack and CUDA checks"]
Check --> Model["SmolLM + LoRA adapter"]
Model --> Data["5 chosen/rejected preference pairs"]
Data --> Trainer["ORPOTrainer: likelihood + odds-ratio term"]
Trainer --> Output["orpo_adapter + tokenizer"]
Separate["NOT run after SFT → DPO"] -.-> Mode
The code specifies ORPOConfig(beta=0.1), batch size 1, gradient accumulation 2, default 3 epochs, and saves llm_training_output/orpo_adapter/.
The repository does not implement PPO, reward-model training, or a scalable human-feedback service. With only five preference pairs, training loss is not a useful quality or safety benchmark.
Pre-Training and Fine-Tuning · Real prodcution grade PyTorch/Hugging Face training examples · Wiki Home · Architecture. This project is licensed under the Apache License, Version 2.0.© 2026 Sudhanshu Shukla · Licensed under the Apache License, Version 2.0