Subsampling frequent outputs

[stephanpeitz]

Hi,

thanks for sharing your code!
I'm just wondering if you have implemented the subsampling of frequent outputs (can't find it in your code) and if it was crucial for the performance.

Cheers,
Stephan

[glample]

Hi,

Not sure about what you mean by frequent outputs, but the code that selects the words to mask is here: https://github.com/facebookresearch/XLM/blob/master/src/trainer.py#L295-L305

sample_alpha == 0 will do the same thing as in the original BERT paper, i.e. sample 15% of words at random. A non-zero value of sample_alpha will sample 15% of the words in a batch, but each word will have a different probability of being masked out (i.e. the rare words will have a higher probability to be masked out than the frequent ones).

[stephanpeitz]

Yes, I was referring to

we also subsample the frequent outputs using an approach similar to Mikolov et al. (2013b)

Thanks for pointing me to the function.

[stephanpeitz]

Sorry, one more question: which value for sample_alpha have you used in your experiments?

[glample]

We used sample_alpha = 0.5