Skip to content

Datasets and Validation

Sudhanshu edited this page Nov 11, 2026 · 1 revision

Datasets, Normalization and Validation

The original uploaded data.zip contains an Alien Q&A file formatted as 49 pretty-printed JSON objects, not strictly one JSON object per line. The publishing package preserves the original archive and additionally supplies:

data/alien_romulus_qa.jsonl — 49 proper JSONL records, with nonempty instruction and output string fields.

flowchart LR
    Uploaded["Uploaded data.zip"] --> Pretty["Pretty-printed 49 JSON objects"]
    Pretty --> Normalize["One JSON object per line"]
    Normalize --> Clean["data/alien_romulus_qa.jsonl"]
    Clean --> Validate["validate_jsonl(): 49 valid pairs"]
    Validate --> Chat["Chat template user + assistant"]
    Chat --> Tokenize["Tokenizer IDs"]
    Tokenize --> Train["Qwen Trainer"]
    Uploaded --> Unused["Smartphone / Wikipedia / space corpora: not used"]
Loading

Verify without model downloads:

python qwen_alien_romulus_finetune.py --validate-data
python LLM_Training_Fixed_v2.py --mode validate

The original data ZIP also contains smartphone, Wikipedia-derived, and space-missions files. They are not consumed by either training code path. They remain in the original archive for provenance and should undergo rights/licensing review before public distribution.

Legacy SFT data (6 examples) and preference data (5 triples) are directly embedded in LLM_Training_Fixed_v2.py; no external JSONL reader is used for that workflow.

Clone this wiki locally