MNIST Example returns difference in predictions: 1.0

[donigian]

I'm using Google Collab GPU runtime to run MNIST example using Keras 1.2.0.

This snippet from MNIST Example

from deeplift.util import compile_func
import numpy as np
from keras import backend as K

deeplift_model = revealcancel_model
deeplift_prediction_func = compile_func([deeplift_model.get_layers()[0].get_activation_vars()],
                                       deeplift_model.get_layers()[-1].get_activation_vars())
original_model_predictions = keras_model.predict(X_test, batch_size=200)
converted_model_predictions = deeplift.util.run_function_in_batches(
                                input_data_list=[X_test],
                                func=deeplift_prediction_func,
                                batch_size=200,
                                progress_update=None)
print("difference in predictions:",np.max(np.array(converted_model_predictions)-np.array(original_model_predictions)))
assert np.max(np.array(converted_model_predictions)-np.array(original_model_predictions)) < 10**-5
predictions = converted_model_predictions

However, I'm getting the following result:

difference in predictions: 1.0
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-4-3f4204ab927a> in <module>()
     13                                 progress_update=None)
     14 print("difference in predictions:",np.max(np.array(converted_model_predictions)-np.array(original_model_predictions)))
---> 15 assert np.max(np.array(converted_model_predictions)-np.array(original_model_predictions)) < 10**-5
     16 predictions = converted_model_predictions

I used the provided shell script to download .h5 model file.

Furthermore, the Compute Importance Scores snippet results in the following error:

RuntimeError: You set the target layer to an activation layer, which is unusual so I am throwing an error - did you mean to set the target layer to the layer *before* the activation layer instead? (recommended for  classification)

[AvantiShri]

Hi Armen,

Did you load the model using the theano or the tensorflow backend?

[donigian]

Hi @AvantiShri,
Using TensorFlow backend. Thanks for your prompt response, really enjoyed your talk at Nvidia GTC.

[AvantiShri]

Are you using the version of DeepLIFT that is on the master branch, or the version that is on the dev-tf branch? The master branch is intended for models trained with the theano backend.

Also, the keras2compat branch is designed for keras 2.0 models, but I have not updated the example notebooks on that branch. Do you specifically want to use keras 1.2? If not, I can prioritize getting the example notebooks on the keras 2.0 branch and just making that into the master branch.

[donigian]

I'm using master branch. Ultimately, I'm trying to reproduce the examples before doing any further work.

I see, let me try to use theano backend & try to reproduce the MNIST & Genomics example notebooks.

It would be nice to get examples working with keras 2.0, but it's less of a priority so long as I get the above to work. Out of curiosity, have you gotten DeepLift to work w/ Tensorflow backend? If so, were you using Keras 1.2 or 2.0?

[AvantiShri]

I’ve gotten DeepLIFT to work with the tensorflow backend with both keras 2.0 and keras 1.2. DeepLIFT for keras 2.0 and the tensorflow backend is implemented on the keras2compat branch. DeepLIFT for keras 1.2 with tensorflow is on the dev-tf branch.

[donigian]

Excellent, thanks for the clarification, I'll give it a test drive.

[AvantiShri]

Btw, I added an example notebook that works on Keras 2 to the keras2compat branch: https://github.com/kundajelab/deeplift/blob/keras2compat/examples/mnist/MNIST_replicate_figures.ipynb - let me know if you have any issues.