GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
For the batch normalization layer what does the line "The running sum is kept with a default momentum of 0.1" means. How does momentum is used here to updates mean and variance?
The running mean and variances are computed using an exponential moving average with smoothing factor 0.1.
i.e running_mean[t] = running_mean[t-1]*0.9 + batch_mean[t]*0.1
running_mean[t] = running_mean[t-1]*0.9 + batch_mean[t]*0.1
Thanks @colesbury, its really helpful.
@n3011, perhaps you'd like to improve the documentation with @colesbury's answer 😬