Preference Dataset

Preference Dataset Design

The draft DPO_SFT.py defines five intended comparison records, each containing three keys: prompt, chosen, and rejected. Since the source cannot be parsed, these are documented from inspection, not loaded or validated during execution.

flowchart LR
    A[Prompt] --> D[Preference record]
    B[Chosen preferred response] --> D
    C[Rejected less-preferred response] --> D
    D --> E{Data validation: future work}
    E -->|valid| F[DPO trainer: proposed]
    E -->|invalid| G[Reject / repair record]

Loading

Quality checks before a real DPO experiment

• Validate every record has all three nonempty string values.
• Normalize prompt formatting to the selected model's tokenizer/template.
• Confirm chosen responses are consistently preferable, avoid label leakage, and maintain a separate held-out preference evaluation set.
• Revisit whether length, formatting, or canned answers are spurious preference signals.
• Protect private or proprietary prompts from publishing.

The ORPO block is a triple-quoted string and is not executed.