Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BatchNorm Layer: it seems to be safer to add a epsilon to the standard deviation #303

Closed
cook opened this issue Mar 23, 2018 · 3 comments
Closed

Comments

@cook
Copy link
Contributor

cook commented Mar 23, 2018

In the following codes cited from the file src/layer/batchnorm, if the sqrt_var is zero, there will be a zero-division error. So I advice to add a epsilon to the standard deviation or add some protection codes.

    float sqrt_var = sqrt(var_data[i]);
    a_data[i] = bias_data[i] - slope_data[i] * mean_data[i] / sqrt_var;
    b_data[i] = slope_data[i] / sqrt_var;

(虽然可能性不大, 但加上去毕竟更安全些)

@nihui
Copy link
Member

nihui commented Mar 24, 2018

epsilon value is handled in model converter that is pre-added to variance

@cook
Copy link
Contributor Author

cook commented Mar 26, 2018

Why there is not a parameter for epsilon in the layer BatchNorm like in the layers such as InstanceNorm and MVN, I've found eps parameter in these layers. Is it inconsistency?

@nihui
Copy link
Member

nihui commented Mar 26, 2018

It is for some historical reason, and it is fine to add eps parameter with default value of zero for consistency.
welcome to open pull request : )

@nihui nihui closed this as completed in 2e9da1b Mar 27, 2018
ghimiredhikura pushed a commit to ghimiredhikura/NCNNSFD that referenced this issue Jul 26, 2020
…encent#311)

* Add the epsilon parameter to the BatchNorm layer. (fix Tencent#303)

* Move the eps into the sqrt.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants