Skip to content

Phi2 SFT DPO Design

Sudhanshu edited this page Nov 11, 2026 · 1 revision

Phi-2 SFT → DPO (Unfinished Design)

Not runnable: DPO_SFT.py fails Python syntax validation at line 44 (unterminated string), followed by additional indentation/text errors. The following is architectural intent inferred from the source, not an executed pipeline.

flowchart TD
    A[6 chat-style SFT examples] -.-> B[Format prompts for Phi-2 tokenizer]
    C[Phi-2 model weights] -.-> D[4-bit NF4 + LoRA adapter]
    B -.-> E[SFTTrainer]
    D -.-> E
    E -.-> F[Saved SFT adapter]
    F -.-> G[Trainable policy and reference configuration]
    H[5 prompt/chosen/rejected pairs] -.-> I[DPOTrainer with beta]
    G -.-> I
    I -.-> J[Proposed DPO adapter and qualitative evaluation]
    H -.-> K[ORPO code-string concept only]
    K -.-> L[Not executed]
    subgraph Warning[Design-stage ONLY — current DPO_SFT.py fails syntax validation]
      E
      I
    end
Loading

Intended modules

  1. Build six chat-style SFT examples from hardcoded SFT_DATA.
  2. Load microsoft/phi-2, configure bitsandbytes 4-bit NF4 quantization and PEFT LoRA.
  3. Train a LoRA policy using SFTTrainer and save a proposed sft_model output.
  4. Build five preference records in DPO_DATA with prompt, chosen, rejected.
  5. Use DPOTrainer with beta=0.1 (subject to installed TRL API compatibility).
  6. Compare generated answers, summarize preference lengths, save a proposed dpo_model.
  7. Print a non-executed ORPO source pattern.

The source contains no implementation of PPO. Manual chat delimiters such as <|system|> are not confirmed as valid Phi-2 tokenizer templates. The DPO reference-model behavior described by the draft must be verified against a corrected TRL version, not assumed. See Known-Issues and LoRA-and-Quantization.

Clone this wiki locally