pretrain a model with the MLM objective

[liujiqiang999]

Hi, How many GPU are used when training a model with the MLM objective?

[glample]

Hi,

In practice, we observed that bigger batches seem to help, and it significantly accelerates the overall training time. I would suggest to use at least 8 GPUs, especially if you use a big model.

[jiahuigeng]

Have you successfully implemented accumulated gradients or multi-gpu settings?

[glample]

Not yet. It will be easy in the next version of PyTorch distributed, so we will wait for that.