Data

ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining
https://github.com/cxcscmu/ReScraper

NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute
https://qlabs.sh/10x

Structured Wikipedia
https://huggingface.co/datasets/wikimedia/structured-wikipedia

Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time
https://arxiv.org/abs/2605.15220

Chroma Context-1 Data Generation
https://github.com/chroma-core/context-1-data-gen

The Synthetic Data Playbook
https://huggingface.co/spaces/HuggingFaceFW/finephrase