Fix bug in BernoulliDropout

[kaikaun]

Divide activations by (1-weight) rather than weight

[lmjohns3]

Can you explain why this is the correct scaling factor instead of just weight?

[kaikaun]

Consider the case where weight = 0.1. This means that 0.9 of the
activations remain, while 0.1 are zeroed out. In order for the total
activation to be on average the same as before, the remaining activations
are scaled up. The current code would scale them up by 10 times (i.e. 1 /
weight), which is wrong, since 0.9 of the activations remain -- the total
activation would now be on average 9 times the original. The correct
scaling is by 1 / 0.9, so you should be dividing by (1 - weight), not
weight.

In addition, I have been testing with the new (1 - weight) scaling in my
own application using my own fork, and I get much better performance both
in and out of sample.

Regards,
David Khoo

On 20 March 2016 at 12:46, Leif Johnson notifications@github.com wrote:

Can you explain why this is the correct scaling factor instead of just
weight?

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#123 (comment)

[lmjohns3]

Ok, fantastic, thanks for the explanation!