Memory explosion when pretrain Bidirectional LSTM

[shangqing-liu]

Hi,

Thanks for the wonderful work. May I ask a question, when I pretrain LSTM model with the default settings, the memory is overflow. My server has 180G RAM, so may I ask how much RAM needed for pretraining?

Thanks and best regards.

[ajayjain]

HI @shangqing-liu -- apologies about the delay. I am a coauthor on this work.

Can I ask what GPU you are using to train this model? Are you encountering a CUDA malloc error (out of memory on GPU) or host-side out of memory error?

At the moment, the dataloader reads the whole dataset in RAM before training. The pre-training dataset is quite large (almost 20GB) so this can be expensive. We pre-trained our models on machines with around 256GB of RAM. However, we didn't encounter OOMs.

If you could provide some more details on your setup, I can help debug!

[shangqing-liu]

Hi, @ajayjain

Thanks for the reply, I have fixed this problem and now close this issue.