Ordering of deduplicated datasets?

[zplizzi]

I just wanted to confirm that the 3 versions you have of the deduplicated data have the same data ordering?

• https://huggingface.co/datasets/EleutherAI/pythia_deduped_pile_idxmaps (tokenized + sharded)
• https://huggingface.co/datasets/EleutherAI/the_pile_deduplicated (jsonl)
• https://huggingface.co/datasets/EleutherAI/raw_deduplicated_pile (raw, parquet)

I was hoping to use the jsonl one but wanted to ensure it will accurately replicate the data ordering in your tokenized dataset that you used for training.

[haileyschoelkopf]

Hi! The latter two datasets have the exact same ordering of data points.

preprocess_data.py in the GPT-NeoX repo should be deterministic in creating shuffle order and tokenizing data, but there’s the off chance that if you rerun (maybe due to very strange multiprocessing issues, etc.)

So, if you’d like to replicate the tokenized Deduplicated Pile in our exact data order, in the Megatron .bin and .idx format, the tokenized + sharded version is the safest bet—If you do prefer to tokenize it yourself though, I could probably provide you a checksum of the tokenized files to confirm things look ok!

[zplizzi]

Thank you!

[stabilize-ai]

going from @zplizzi 's original message, do the 2nd and 3rd links below contain the same examples as the ones used for pythia training and in the same order ?

• https://huggingface.co/datasets/EleutherAI/pythia_deduped_pile_idxmaps (tokenized + sharded)
• https://huggingface.co/datasets/EleutherAI/the_pile_deduplicated (jsonl)
• https://huggingface.co/datasets/EleutherAI/raw_deduplicated_pile (raw, parquet)

[haileyschoelkopf]

The 2nd and 3rd links do not. They theoretically do contain the same data if you run prepare_data.py on the 2nd link, but due to multiprocessing weirdness I can't guarantee the shuffle would end up the same, which is why we supply the first link