-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
double backward always returns nan when dtype is float16 and cudnn is enabled. #6251
Comments
With numpy, the errors are also too large. |
I will check if this issue is related to #6323 . |
@takagi cupy/cupy#2072 and #6497 seem to fix this issue, at least the above snippet. |
I guess cupy/cupy#2060, #6323, and #5924 did. |
@crcrpar Could you try with the current master branch? |
@grafi-tt I tried the snippet with current master, and it worked. |
I've also confirmed that the snippet works.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When a pair of
F.reshape
andF.batch_normalization
is used under the condition that dtype is fload16 and use_cudnn='always', double backward of the pair goes so unstable that it returnsnan
with high probability.One use-case of the pair is
F.group_normalization
:chainer/chainer/functions/normalization/group_normalization.py
Lines 61 to 72 in afe9033
For backward,
For double backward,
The text was updated successfully, but these errors were encountered: