Unable to create a reproduceable code

[alizzz]

I've tried adding numpy.random.seed before importing anything from keras (as was suggested in #439). However, each run produces different result.

[desilinguist]

I am having the exact same problem. I ran the bundled LSTM example 4 times as follows (I installed both keras and Theano from their git repositories directly instead of from PyPI):

for i in `seq 4`; do python imdb_lstm.py; done

And I get the following test set scores and accuracies for the four runs which as you can see vary widely:

Run #1 
Test score: 0.474129207468
Test accuracy: 0.8304

Run #2
Test score: 0.405411259794
Test accuracy: 0.828

Run #3
Test score: 0.451528955483
Test accuracy: 0.8324

Run #4
Test score: 0.490154602194
Test accuracy: 0.8276

The full log for the command is below. Shouldn't every epoch yield exactly the same result across runs if numpy.random.seed is set? Or is there another source of randomness that we are not controlling?

Using gpu device 0: Tesla K20Xm (CNMeM is disabled)
Loading data...
20000 train sequences
5000 test sequences
Pad sequences (samples x time)
X_train shape: (20000, 100)
X_test shape: (5000, 100)
Build model...
Train...
Train on 20000 samples, validate on 5000 samples
Epoch 1/4
20000/20000 [==============================] - 46s - loss: 0.5733 - acc: 0.6961 - val_loss: 0.3945 - val_acc: 0.8222
Epoch 2/4
20000/20000 [==============================] - 46s - loss: 0.3169 - acc: 0.8729 - val_loss: 0.3788 - val_acc: 0.8340
Epoch 3/4
20000/20000 [==============================] - 46s - loss: 0.1948 - acc: 0.9281 - val_loss: 0.3752 - val_acc: 0.8326
Epoch 4/4
20000/20000 [==============================] - 46s - loss: 0.1257 - acc: 0.9558 - val_loss: 0.4741 - val_acc: 0.8304
5000/5000 [==============================] - 4s
Test score: 0.474129207468
Test accuracy: 0.8304
Using gpu device 0: Tesla K20Xm (CNMeM is disabled)
Loading data...
20000 train sequences
5000 test sequences
Pad sequences (samples x time)
X_train shape: (20000, 100)
X_test shape: (5000, 100)
Build model...
Train...
Train on 20000 samples, validate on 5000 samples
Epoch 1/4
20000/20000 [==============================] - 46s - loss: 0.6699 - acc: 0.5927 - val_loss: 0.5938 - val_acc: 0.7136
Epoch 2/4
20000/20000 [==============================] - 46s - loss: 0.4760 - acc: 0.7772 - val_loss: 0.4008 - val_acc: 0.8246
Epoch 3/4
20000/20000 [==============================] - 47s - loss: 0.2920 - acc: 0.8812 - val_loss: 0.3977 - val_acc: 0.8302
Epoch 4/4
20000/20000 [==============================] - 46s - loss: 0.1994 - acc: 0.9266 - val_loss: 0.4054 - val_acc: 0.8280
5000/5000 [==============================] - 4s
Test score: 0.405411259794
Test accuracy: 0.828
Using gpu device 0: Tesla K20Xm (CNMeM is disabled)
Loading data...
20000 train sequences
5000 test sequences
Pad sequences (samples x time)
X_train shape: (20000, 100)
X_test shape: (5000, 100)
Build model...
Train...
Train on 20000 samples, validate on 5000 samples
Epoch 1/4
20000/20000 [==============================] - 46s - loss: 0.6477 - acc: 0.6252 - val_loss: 0.4986 - val_acc: 0.7772
Epoch 2/4
20000/20000 [==============================] - 46s - loss: 0.3893 - acc: 0.8312 - val_loss: 0.3874 - val_acc: 0.8290
Epoch 3/4
20000/20000 [==============================] - 46s - loss: 0.2540 - acc: 0.9009 - val_loss: 0.3813 - val_acc: 0.8260
Epoch 4/4
20000/20000 [==============================] - 46s - loss: 0.1783 - acc: 0.9353 - val_loss: 0.4515 - val_acc: 0.8324
5000/5000 [==============================] - 4s
Test score: 0.451528955483
Test accuracy: 0.8324
Using gpu device 0: Tesla K20Xm (CNMeM is disabled)
Loading data...
20000 train sequences
5000 test sequences
Pad sequences (samples x time)
X_train shape: (20000, 100)
X_test shape: (5000, 100)
Build model...
Train...
Train on 20000 samples, validate on 5000 samples
Epoch 1/4
20000/20000 [==============================] - 46s - loss: 0.5874 - acc: 0.6794 - val_loss: 0.3825 - val_acc: 0.8342
Epoch 2/4
20000/20000 [==============================] - 46s - loss: 0.3242 - acc: 0.8661 - val_loss: 0.3941 - val_acc: 0.8220
Epoch 3/4
20000/20000 [==============================] - 46s - loss: 0.2155 - acc: 0.9190 - val_loss: 0.3737 - val_acc: 0.8386
Epoch 4/4
20000/20000 [==============================] - 47s - loss: 0.1415 - acc: 0.9517 - val_loss: 0.4902 - val_acc: 0.8276
5000/5000 [==============================] - 4s
Test score: 0.490154602194
Test accuracy: 0.8276

[hlin117]

@nouiz in #439 suggests the following:

Be sure to use Theano dev version, otherwise sometimes Theano use numpy
random generator in way that is very hard to predict, so this look like the
seed don't work:

http://deeplearning.net/software/theano/install.html#bleeding-edge-install-instructions

Just in case this is your problem.

[hlin117]

Nevermind, I tried his suggestion, but I still can't get consistent results.

[dbonadiman]

Are you using Convolutions on GPU? It may be you need to use a special
'deterministic' theano flag.
Check there
http://deeplearning.net/software/theano/library/sandbox/cuda/dnn.html it
may slow down your results but it helps to fix this problem.

By default cuDNN uses a non deterministic algorithm to compute gradients
for convolutions.
To debug if it is the case try a CPU run (if it is possible) then check for
the flag.

Daniele Bonadiman

On Thu, Mar 3, 2016 at 12:42 AM, Henry Lin notifications@github.com wrote:

Nevermind, I tried his suggestion, but I still can't get consistent
results.

—
Reply to this email directly or view it on GitHub
#850 (comment).

[hlin117]

I'm not using a CNN. Just a simple, feedforward neural network.

[hlin117]

EDIT: Okay, I actually figured out my problem. I thought that fit would actually reset the parameters of the network per call, but it doesn't. (scikit-learn does this.)

Looks like I could reproduce the results now. Thanks.

[tdeboissiere]

Adding to @dbonadiman :

The documentation on http://deeplearning.net/software/theano/library/sandbox/cuda/dnn.html is not up to date with theano 0.8.1
If you want a deterministic behaviour when using the GPU and cuda dnn, you should add

[dnn.conv]
algo_bwd_filter = deterministic
algo_bwd_data = deterministic

to your .theanorc file (algo_bwd = deterministc is deprecated)

[bapla]

@hlin117 What did you do to solve the issue?

I have the same situation: a very simple feedforward neural network. I set the numpy seed as the very first thing in the script. However, I do get different results for different runs.

I'm using the latest theano dev (as suggested by #439) + keras version:
Successfully installed Theano-0.9.0.dev0
Successfully installed Keras-1.0.1

``import numpy as np
np.random.seed(1113)

[...]

print("build model")
model = Sequential()
model.add(Dense(num_labels, input_dim=input_size, init='uniform'))
model.add(Activation('tanh'))
model.add(Dense(num_labels))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer="sgd", metrics=['accuracy'])
model.fit(X_train, y_train,
nb_epoch=args.iters,
batch_size=100, validation_data=(X_dev, y_dev))
``

[hlin117]

Can you double check you're using those packages, using keras.version
and theano.version?
On Apr 25, 2016 7:45 AM, "bapla" notifications@github.com wrote:

@hlin117
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_hlin117&d=CwMCaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=IgVP_sCIQHhgRgxQoo7gufqSQOvlFIckTGgYIpXBa_4&m=kdjM0XSrMDzDOn17_RI_G4FQfyy7sagfryn_MgvnrvI&s=11H-KguseLniOVLQCd40srIWyw-Cquuf1LZ2kO3N2ek&e=
What did you do to solve the issue?

I have the same situation: a very simple feedforward neural network. I set
the numpy seed as the very first thing in the script. However, I do get
different results for different runs.

I'm using the latest theano dev (as suggested by #439
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_fchollet_keras_issues_439&d=CwMCaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=IgVP_sCIQHhgRgxQoo7gufqSQOvlFIckTGgYIpXBa_4&m=kdjM0XSrMDzDOn17_RI_G4FQfyy7sagfryn_MgvnrvI&s=AtUaOb_tpu7cUZ8qMhQ-ySuUIpkc60Jqeeth3lJJCSE&e=)

• keras version:
  Successfully installed Theano-0.9.0.dev0
  Successfully installed Keras-1.0.1

import numpy as np
np.random.seed(1113)
[...]
print("build model")
model = Sequential()
model.add(Dense(num_labels, input_dim=input_size, init='uniform'))
model.add(Activation('tanh'))
model.add(Dense(num_labels))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer="sgd",
metrics=['accuracy'])
model.fit(X_train, y_train,
nb_epoch=args.iters,
batch_size=100, validation_data=(X_dev, y_dev))

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_fchollet_keras_issues_850-23issuecomment-2D214299031&d=CwMCaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=IgVP_sCIQHhgRgxQoo7gufqSQOvlFIckTGgYIpXBa_4&m=kdjM0XSrMDzDOn17_RI_G4FQfyy7sagfryn_MgvnrvI&s=I1ngqB1LuTr-SInFOUnOkkJJvNxsC1WPwFBBjMl869E&e=

[bplank]

hi @hlin117
here is the check - they are the right versions.

import keras keras.__version__ '1.0.1' import theano theano.__version__ '0.9.0dev0.dev-c95bea5047dd8f18dc9cc1fa6415a785639d38f7'

However, the seeding still doesn't work, I get different results with different runs.

[hlin117]

Can you make sure that you're using the same training and testing set for every run? (Truthfully, that was actually my problem.)

[bplank]

No, that's not the problem, I'm using pre-split separate files.

Weird..

[nouiz]

There is a few source of non-deterministic things in Theano. We are in the
process of making a flag to disable all of them at the same time. You can
do it by disabling cudnn and applying this diff on this file:

https://github.com/Theano/Theano/pull/4212/files#diff-c55c00bc8b5d7f77c3795170a0a519ad

On Thu, Apr 28, 2016 at 2:43 AM, Barbara Plank notifications@github.com
wrote:

No, that's not the problem, I'm using pre-split separate files.

Weird..

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#850 (comment)

[bplank]

Hi @nouiz!
Thanks for this update. I see.. that flag would be really handy!

I tried now, changed the two lines in theano/sandbox/cuda/opt.py, ran setup.py, and added THEANO_FLAGS='optimizer_excluding=conv_dnn' in front of the script. However, it's still not deterministic.. Did I do something wrong? When do you expect the flag to be ready?

Btw: here's the simple sample code that I'm running. https://github.com/bplank/example_ff.git

[kepler]

@bplank, after spending some time trying to find a solution to this problem, I think I nailed down a major source of randomness.

Using python 2 I'm able to get consistent results with your code running theano on a GPU without any special THEANO_FLAGS.

However, with python 3 results are not deterministic, even with the proposed flags above. What turned results deterministic in python 3 was setting the flag

PYTHONHASHSEED=0

before running the code. Details about this flag can be found at the python docs.

[bplank]

@kepler Thanks, great! I just checked, your suggestion of setting PYTHONHASHSEED=0 works for python 3 with the theano backend.
Unfortunately this is not the case for the tensorflow backend..

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs, but feel free to re-open it if needed.

[bplank]

We can close this issue (it works for theano) - it is still an open issue for the tensorflow backend - see: #2280

[bplank]

Follow up: This is how it works for me for the theano backend:

np.random.seed(113) #set seed before any keras import
from keras.models import Sequential

[bplank]

Again follow up: The two runs I compared to were accidentally very similar, but now I do no longer get the same results back..

Did something change with the Theano backend?

[bplank]

Update: With Theano (0.9.0 dev version) my sample code actually works IF I set the PYTHONHASHSEED. We can close this issue. Keras with Theano backend is reproducible code.

[riley-x]

@bplank can you summarize the versions of Theano, Keras, and Python you're using, and what steps you took to get deterministic results? I see setting PYTHONHASHSEED=0 and np.random.seed from the above comments; was there anything else? Were you able to get it to work with Theano 0.9.0 stable version?

[mbollmann]

I discussed this issue with @bplank and did reproducibility tests myself, and we both got deterministic results with the setup you described:

• Theano 0.9.0 stable
• PYTHONHASHSEED=0 (or some other number; wasn't always needed for reproducibility)
• np.random.seed() (always tried setting it first thing before importing anything Theano-/Keras-related)

I confirmed that this produced deterministic results with Keras 1.2.2 and 2.0.4 on different example scripts, though not when running with CUDA. For this, a patch for Theano is needed, which has also been discussed above already. I use this minimal patch in my virtualenv:

https://gist.github.com/mbollmann/fda3b219ef1416649b79ecc725174152

I didn't try Tensorflow backend, and I only tried a subset of the available layers/models in Keras, so this was not strictly an exhaustive test.

[bplank]

@riley-x I needed to use the PYTHONHASHSEED. And setting the seed before any import, as @mbollmann said above.

I did get deterministic results with Python 3.6 Anaconda, theano '0.9.0.dev-c697eeab84e5b8a74908da654b66ec9eca4f1291' and Keras 2.0.2. See the run_example.sh script in the example github folder mentioned earlier.

[JWGoodwin]

I succeeded in getting reproducibility with:
Win7 64 bit,
Python 2.7.13
Theano 0.10.0beta2.dev-c3c477df9439fa466eb50335601d5c854491def8
Keras 2.0.8
Both on CPU and on GEForce 1060 GPU.

See my comments on Jason Brownlee's blog at
https://machinelearningmastery.com/reproducible-results-neural-networks-keras/#comment-414394

[wt-huang]

Closing as this is resolved