Question For Training Dataset

[dumpmemory]

Except the database and index data on huggingface, the train_data.json in the repo could thought to be an example right? Would you mind releasing the full version of train and test dataset for reproducing the result ?

[bling0830]

We explain how to retrieve the library and how to handle training data in index-server/README.md. The pile is used as training corpus.

We used the first 400,000 texts of pile_00.json and the first 400,000 texts of pile_29.json for a total of 800,000 texts as training corpus. Do the truncate operation for text over 1025.

[dumpmemory]

Thanks for your support.