Intermittent memory errors while running preprocess.py

[EvanCrabtree]

While running preprocess.py, I have received intermittent memory access errors.

I have been using the ramen dataset from the link provided and running python preprocess.py --dataset_path data/ramen.

Randomly, it will give me memory errors like double free or corruption (!prev) Aborted (core dumped) , corrupted size vs. prev_size Aborted and additionally a pytorch error that has been patched in pytorch 2.0.1 when trying to run on multithreaded CPU.

Setting the threads to 1 with torch.set_num_threads(1) fixes the pytorch issue but makes the preprocessing speed very slow.

I was wondering if there was any advice on how to fix these issues happening during the preprocess script execution.