Repository navigation
Configuration
Sudhanshu edited this page Nov 11, 2026
·
1 revision
| Parameter | Qwen LoRA default | Legacy default |
|---|---|---|
| Base model | Qwen2.5-0.5B-Instruct | SmolLM-135M (Phi-2 opt-in) |
| LoRA | r=8, alpha=16, dropout=0.05 | r=8, alpha=16, dropout=0.05 |
| Target modules |
q_proj, v_proj
|
q_proj, v_proj or Phi-specific list |
| Epochs | 5 | SFT 3, DPO 2, ORPO 3 |
| Max sequence length | 256 | 256 |
| Batch size | 1 | 1 |
| Grad accumulation | 4 | 2 |
| Optimizer | Trainer default | AdamW Torch (paged AdamW for 4-bit) |
| Output | qwen_alien_romulus_finetuned/ |
llm_training_output/ |
Use python qwen_alien_romulus_finetune.py --help or python LLM_Training_Fixed_v2.py --help for accepted CLI switches. The Qwen original _fixed.py script has hardcoded constants rather than the notebook-export CLI. Legacy --quantize-4bit requires compatible CUDA and bitsandbytes support.
Pre-Training and Fine-Tuning · Real prodcution grade PyTorch/Hugging Face training examples · Wiki Home · Architecture. This project is licensed under the Apache License, Version 2.0.© 2026 Sudhanshu Shukla · Licensed under the Apache License, Version 2.0