Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Feature Request: Support for configuring deterministic options of cudNN Conv routines #18096
Please go to Stack Overflow for help and support:
If you open a GitHub issue, here is our policy:
Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.
You can collect some of this information using our environment capture script:
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the problem
Can a user-facing option be added, perhaps in
Source code / logs
It's possible #16889 can be rolled into this issue. @ekelsen did work relating to determinism in #12871. @drpngx closed a "we trade determinism for speed" doc contribution in #10636, saying we're working on the problem.
1 similar comment
If you're referring to conv ops, TF currently does autotuning underneath to pick the best algorithm for the input shape, by first running a few trial steps. I imagine disabling autotune would give you determinism in conv.
You can set
@yzhwang to confirm.
In my opinion, it would be helpful to get a fine-grained option to enable / disable the specific cudNN operations that are non-deterministic. For example, theano's config surfaces the
This is opposed to using a big hammer to enable or disabling autotuning for all cudNN routines, which couples determinism to how autotuning logic behaves.
1 similar comment
From TF2 release notes:
For more information about GPU determinism in TensorFlow, please see: https://github.com/NVIDIA/tensorflow-determinism
This feature request can now be closed.