Scaled embedded dropout mask

[plpplp]

Hello,

why is the embedded dropout mask weighted by / (1 - dropout)?

hedwig/models/reg_lstm/embed_regularize.py

Line 38 in 98634d3

mask = embed.weight.data.new().resize_((embed.weight.size(0), 1)).bernoulli_(1 - dropout).expand_as(embed.weight) / (1 - dropout)

I looked into the references in the corresponding paper but
Gal and Ghahramani (2016) are not using it and a Merity et al. (2018) are just stating that they do it. So I wonder what the idea behind the step is.

[plpplp]

I found the answer. Scaling is to account for the change in magnitude of activations when dropout is applied and seems to be the standard way to do dropout.