-
Notifications
You must be signed in to change notification settings - Fork 74.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GPU MaxPool gradient ops do not yet have a deterministic XLA implementation #69417
Comments
@wx0608, |
This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. |
I am using DeepLabV3+ with Resent backbone and I am getting the same error when I want to have reproducible results. I put these lines as well: os.environ['TF_DETERMINISTIC_OPS'] = '1' from here I realied that TF_CUDNN_DETERMINISTIC. When set to 'true' or '1', this selects deterministic gradient algorithms for tf.nn.max_poold and tf.keras.layers.MaxPoolD , before I has same problem with Upsampling layers in which we should use bilinear interpolation not Nearestneighbor , but with MaxPooling layers I dont know what should be changed. Please leave any meassage that might be helpful. |
wx0608 any findings? |
This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. |
Same here. I guess, as enable_op_determinism() documentation states:
Can you please implement MaxPool2D deterministic? AveragePooling2D layer does not have that problem. My Setting:
to make code deterministic, causes:
using that Tensorflow code:
tensorflow: 2.11.0 |
Seems when disabling XLA, error disappers for MaxPooling2D:
|
This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. |
Hi. If possible, please also consider deterministic XLA for |
@wx0608, Thank you! |
Tested with Keras 3.0, Tensorflow 2.17. Still same error:
For me MaxPool1D is not an option. |
I am having the same issue with MaxPool1D |
this issue still exist. |
Issue type
Feature Request
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
binary
TensorFlow version
tf 2.16
Custom code
Yes
OS platform and distribution
Linux Ubuntu 22.04
Mobile device
No response
Python version
3.9.19
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
12.4/8.9.7.29
GPU model and memory
NVIDIA GeForce RTX 3090
Current behavior?
When TF deterministic was set, runtime exception was thrown at MaxPooling2D().
Standalone code to reproduce the issue
When TF deterministic was set, runtime exception was thrown at MaxPooling2D().
Relevant log output
The text was updated successfully, but these errors were encountered: