-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix elementwise_gradient bug #10150
Fix elementwise_gradient bug #10150
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add your test as well?
612685e
to
5b345d1
Compare
5b345d1
to
d06c79c
Compare
}, | ||
fetch_list=['x@GRAD', 'y@GRAD']) | ||
self.__assert_close(x_grad, out[0], "x@GRAD") | ||
self.__assert_close(y_grad, out[1], "y@GRAD", atol=2.0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
atol=2.0 ? So large?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The process of getting gradient involves accumulative operation and the shape of input is too large, x.shape = [2,32,220,220], y.shape = [32], so the diff of the result between Python and C++(CUDA) is bigger.
Python result
array([96771.625, 96801.26 , 96760.63 , 96893.32 , 96946.375, 96911.95 ,
96837.16 , 96716.88 , 96931.305, 96746.445, 96781.81 , 96827.22 ,
97108.19 , 96866.8 , 96894.17 , 96776.36 , 96720.61 , 96992.8 ,
96641.664, 96772.305, 96698.16 , 96675.64 , 96805.86 , 96710.6 ,
96733.37 , 96858.41 , 96771.516, 97033.516, 96820.48 , 96726.09 ,
96784.66 , 96740.76 ], dtype=float32)
C++ result
array([96771.1 , 96801.05 , 96761.125, 96892.69 , 96946.73 , 96910.93 ,
96837.516, 96716.27 , 96930.99 , 96748.18 , 96781.41 , 96828.016,
97108.17 , 96866.62 , 96893.836, 96775.74 , 96720.98 , 96992.24 ,
96642.36 , 96771.14 , 96698.195, 96675.055, 96805.88 , 96711.305,
96732.086, 96858.97 , 96770.734, 97033.92 , 96819.95 , 96725.72 ,
96784.35 , 96740.76 ], dtype=float32)
CUDA result
array([96738.2 , 96891.58 , 96752.25 , 96972.39 , 96706.016, 96755.234,
96731.42 , 96850.01 , 97060.125, 96790.375, 96587.12 , 96833.305,
96709.3 , 96703.7 , 96842.61 , 96727.95 , 96966.586, 96791.33 ,
97077.234, 96715.03 , 96850.26 , 96898.47 , 96780.97 , 96839.516,
96835.61 , 96600.41 , 96517.08 , 96787.58 , 96758.37 , 96555.484,
96882.94 , 96578.2 ], dtype=float32)
So the max diff between Python result
and C++ result
is 1.28125, the max diff between Python result
and CUDA result
is 0.015625.
e2f89a1
to
3623f02
Compare
… fix_elementwise_gradient
3623f02
to
0f5d5b1
Compare
fix #10122 |
There is a lack of synchronization in
reduceSum
, so the results of it have some randomness when the thread block excess 256. But the reason of why randomness doesn't happen when thread block is less than 256 is not clear.