Repository navigation
Phi2 SFT DPO Design
Not runnable:
DPO_SFT.pyfails Python syntax validation at line 44 (unterminated string), followed by additional indentation/text errors. The following is architectural intent inferred from the source, not an executed pipeline.
flowchart TD
A[6 chat-style SFT examples] -.-> B[Format prompts for Phi-2 tokenizer]
C[Phi-2 model weights] -.-> D[4-bit NF4 + LoRA adapter]
B -.-> E[SFTTrainer]
D -.-> E
E -.-> F[Saved SFT adapter]
F -.-> G[Trainable policy and reference configuration]
H[5 prompt/chosen/rejected pairs] -.-> I[DPOTrainer with beta]
G -.-> I
I -.-> J[Proposed DPO adapter and qualitative evaluation]
H -.-> K[ORPO code-string concept only]
K -.-> L[Not executed]
subgraph Warning[Design-stage ONLY — current DPO_SFT.py fails syntax validation]
E
I
end
- Build six chat-style SFT examples from hardcoded
SFT_DATA. - Load
microsoft/phi-2, configure bitsandbytes 4-bit NF4 quantization and PEFT LoRA. - Train a LoRA policy using
SFTTrainerand save a proposedsft_modeloutput. - Build five preference records in
DPO_DATAwithprompt,chosen,rejected. - Use
DPOTrainerwithbeta=0.1(subject to installed TRL API compatibility). - Compare generated answers, summarize preference lengths, save a proposed
dpo_model. - Print a non-executed ORPO source pattern.
The source contains no implementation of PPO. Manual chat delimiters such as <|system|> are not confirmed as valid Phi-2 tokenizer templates. The DPO reference-model behavior described by the draft must be verified against a corrected TRL version, not assumed. See Known-Issues and LoRA-and-Quantization.
Transformer Training Wiki · Implementation status: DistilBERT source available.This project is licensed under the Apache License, Version 2.0.© 2026 Sudhanshu Shukla · Licensed under the Apache License, Version 2.0
Overview
Architecture
Implemented classifier
- DistilBERT-Workflow
- Dataset-and-Labels
- Training-Configuration
- Evaluation-and-Inference
- Frozen-Backbone-Comparison
- Artifacts-and-Checkpoints
Experimental alignment draft
Operations