My training crashes with large corpus.

[Srj]

I have a corpus of 10GB and i tried to train it on Google Colab with 25 GB ram.But during training time it crashes. Is there any way to batchify or anything else so that my model doesn't crash? Any suggestion will be helpful.

[mingruimingrui]

Wow, that's really large... probably like 100 million sentences.
Typically a training set of 10 million sentences is already considered large.

With so much data, you can probably sample a subset of your dataset and get the same performance.
Perhaps read the first ~ 10 million sentences and split into train and test (validation optional)?
Training should be straight forward. From experience, you should take note of the following during evaluation.

• out of vocabulary % (or % unknown token)
• average token per word

Ensure consistency and you should be good.

It might also help to have a validation set to generate sentence piece vocabularies if your data is extremely noisy.
A vocabulary threshold like 20 or 50 would be good.

These are advice from a guy who mainly does machine translation.

[taku910]

Please try --train_extremely_large_corpus=true of spm_train. You computer needs to have reasonable memory size to enable this mode though.

[Mistobaan]

Got these silent crashes too on large text files (~50GB)

[tanreinama]

I get 'std::bad_alloc' error on 11GB japanease corpus with --train_extremely_large_corpus=true option.
It seems work well in AWS r5.24xlarge instance. memory usage is 180GM+. thx.

[thusinh1969]

I am too. 30G corpus, 180 millions sentences. Only know about -train_extremely_large_corpus=true now, will try and let you know. It works with "bpe" though, but crashed on unigram.

Steve

[thusinh1969]

I am too. 30G corpus, 180 millions sentences. Only know about -train_extremely_large_corpus=true now, will try and let you know. It works with "bpe" though, but crashed on unigram.

Steve

Keep crashing on UNIGRAM !!!! Any hint ?
Steve