Data preprocessing with multi-gpus

[xingweiqu]

Hi LCM teams, thanks for your great work, I am trying to use your prepare_wikipedia script to deal with my own data. Right now, I have a 1 node with 8 gpus, but I found that this pipeline is super slow, and it seems that only this is only using one gpu. Could u help me to figure out how to use multi-gpus and prallel, with this script?

[chengshuang18]

+1