TensorFlow MNIST accuracy test is failing

[mahnerak]

Completed model training.
Traceback (most recent call last):
File "test_mnist_accuracy.py", line 71, in
app.run()
File "/home/mahnerak/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "test_mnist_accuracy.py", line 67, in main
assert float(accuracy) >= 0.98, accuracy
AssertionError: 0.9639

[goodfeli]

It just passed three times in a row for me.

• Could it be flaky? If you re-run it, does it still fail?
• Are you running from a clean copy of the master branch?
• What version of keras and tensorflow are you using?

[mahnerak]

• Tried just now. Failed 5 times in a row. For each run got ~0.96 accuracy.
• Running from clean copy (current version) of the master branch
• TensorFlow v0.12, Keras v1.2.0

One more thing. I noticed that Adadelta from TensorFlow behaves different than implementations found in Keras and Lasagne (Theano versions), when I was implementing theano version of FGSM. Then I ran Keras version of Adadelta on TensorFlow, and its results were also different than the ones from TensorFlow's default Adadelta. It was a bit confusing.

Today I updated TensorFlow, and its Adadelta behaves like the other implementations.

[npapernot]

The test is passing on my machine as well

[goodfeli]

@mahnerak : is there a 3rd digit to your tensorflow version? I am also running keras 1.2.0. My tensorflow is 0.12.1. Have you checked cleverhans.file to make sure your PYTHONPATH is set correctly and that you are in fact importing the clean unmodified master branch?
@npapernot : what are your versions?

[npapernot]

TF: 0.10.0rc0 and same keras version

[mahnerak]

TensorFlow version is 0.12-head (compiled from sources today).
Checked PYTHONPATH again, it's set correct.
I'll try to do clean install tensorflow (with prebuild binaries) and keras on another machine and run the tests.

[goodfeli]

It's possible tensorflow 0.12-head is broken.
I suggest checking cleverhans.__file__ rather than checking PYTHONPATH.

[mahnerak]

Installed 0.12.1 from prebuilt binaries. Tests are passing now.

[goodfeli]

@mahnerak I would suggest reporting this on the tensorflow repository issues tracker if you haven't already