Repository navigation
Datasets and Validation
The original uploaded data.zip contains an Alien Q&A file formatted as 49 pretty-printed JSON objects, not strictly one JSON object per line. The publishing package preserves the original archive and additionally supplies:
data/alien_romulus_qa.jsonl — 49 proper JSONL records, with nonempty instruction and output string fields.
flowchart LR
Uploaded["Uploaded data.zip"] --> Pretty["Pretty-printed 49 JSON objects"]
Pretty --> Normalize["One JSON object per line"]
Normalize --> Clean["data/alien_romulus_qa.jsonl"]
Clean --> Validate["validate_jsonl(): 49 valid pairs"]
Validate --> Chat["Chat template user + assistant"]
Chat --> Tokenize["Tokenizer IDs"]
Tokenize --> Train["Qwen Trainer"]
Uploaded --> Unused["Smartphone / Wikipedia / space corpora: not used"]
Verify without model downloads:
python qwen_alien_romulus_finetune.py --validate-data
python LLM_Training_Fixed_v2.py --mode validateThe original data ZIP also contains smartphone, Wikipedia-derived, and space-missions files. They are not consumed by either training code path. They remain in the original archive for provenance and should undergo rights/licensing review before public distribution.
Legacy SFT data (6 examples) and preference data (5 triples) are directly embedded in LLM_Training_Fixed_v2.py; no external JSONL reader is used for that workflow.
Pre-Training and Fine-Tuning · Real prodcution grade PyTorch/Hugging Face training examples · Wiki Home · Architecture. This project is licensed under the Apache License, Version 2.0.© 2026 Sudhanshu Shukla · Licensed under the Apache License, Version 2.0