Dataset | Source Url | ||||||||
---|---|---|---|---|---|---|---|---|---|
The Pile | https://the-eye.eu/public/AI/pile/ | ||||||||
Wikipedia | https://dumps.wikimedia.org/ | ||||||||
Common crawl | https://commoncrawl.org/ | ||||||||
百度百科 | https://baike.baidu.com/ | ||||||||
搜狗百科 | https://baike.sogou.com/ | ||||||||
知乎 | https://www.zhihu.com/ | ||||||||
Quora | https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs | ||||||||
书生万卷/wanjuan-1.0 | https://github.com/opendatalab/WanJuan1.0 | ||||||||
C4 | https://huggingface.co/datasets/allenai/c4 | ||||||||
Wudao | https://github.com/BAAI-WuDao/Data | ||||||||
The Stack | https://huggingface.co/datasets/bigcode/the-stack | ||||||||
OLMO | https://github.com/allenai/OLMo | ||||||||
project gutenberg | https://gutenberg.org/ | ||||||||
openwebtext | https://huggingface.co/datasets/Skylion007/openwebtext |