How to set nCodes?

[shawnzaru]

I am wondering based on what the number 40000 or 60000 is chosen. Is it like a rough estimate of the size of corpora vocabulary?

Thanks.

[glample]

Yes, it is a rough estimate of the vocabulary size. If you generate 40000 BPE codes, you should roughly get around 41k - 43k unique words in your vocabulary.

A small number of codes (like 8k) will result in a very small vocabulary, so faster training, but lower performance. 80k is a bit slow, usually 40k or 60k is a good trade-off between speed and final performance.

[ljl199141]

I am training a multi-language model. My training data is from Wikipedia, so it has over lots of languages. How do I determine nCode in this case? Thanks!

[glample]

There is no really good way to do this. Best is to try different values, and see what works best.