Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the possibility to run with a fixed seed #3

Closed
davidsandberg opened this issue Feb 12, 2016 · 1 comment
Closed

Add the possibility to run with a fixed seed #3

davidsandberg opened this issue Feb 12, 2016 · 1 comment

Comments

@davidsandberg
Copy link
Owner

Add the possibility to run with a fixed seed to simplify troubleshooting

@davidsandberg
Copy link
Owner Author

This seems more difficult than expected.
When running on the GPU the backward pass is not deterministic.
Made a very simple test case (test/seed_test.py) to check this.
The problem is due to the gradient computation in CuDNN. When running it on CPU (i.e. without CuDNN) the test case passes. After upgrading to CuDNN 4.0.7 (from v3) also the test case with only linear layers breaks.
It is described in the CuDNN user guide (section 2.5, see below).
The problem has been discussed in https://groups.google.com/forum/#!topic/theano-users/718YYXwaYEk

2.5. Reproducibility (determinism)
By design, most of cuDNN's routines from a given version generate the same bit-wise
results at every run when executed on GPUs with the same architecture and the same
number of SMs. However, bit-wise reproducibility is not guaranteed across versions,
as the implementation of a given routine may change. With the current release, the
following routines do not guarantee reproducibility because they use atomic add
operations:
cudnnConvolutionBackwardFilter when
CUDNN_CONVOLUTION_BWD_FILTER_ALGO_0 or
CUDNN_CONVOLUTION_BWD_FILTER_ALGO_3 is used
cudnnConvolutionBackwardData when
CUDNN_CONVOLUTION_BWD_DATA_ALGO_0 is used
cudnnPoolingBackward when CUDNN_POOLING_MAX is used

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant