How to convert models saved using tensorflow.keras?

[hartonen]

Thank you for a very useful tool!

Am I correct in assuming that models saved using tensorflow.keras are not supported? Is the only way to visualize these models to manually create an identical model in keras and load the weights from my model saved with tensorflow.keras (as done in deeplift/examples/convert_models/keras1.2to2/MNIST.ipynb)? Just wanted to make this sure for myself :)

[AvantiShri]

Yes, as of now I have not implemented support for loading from the tensorflow.keras saved file format.

[hartonen]

Thank you for a swift answer!

I ran the training again for my network using keras instead of tensorflow.keras. However, when trying to calculate the importance scores as instructed in the "Quickstart"-section, I get a following error message:

"AttributeError: 'BatchNormalization' object has no attribute 'set_scoring_mode'"

This occurs when I try to compile the function to compute the contribution scores, i.e.

find_scores_layer_idx = 0
deeplift_contribs_func = deeplift_model.get_target_contribs_func(find_scores_layer_idx=find_scores_layer_idx,target_layer_idx=-2)

I was able to convert the model without issues with:

deeplift_model = kc.convert_model_from_saved_files('model-900-0.81.h5',nonlinear_mxts_mode=deeplift.layers.NonlinearMxtsMode.DeepLIFT_GenomicsDefault)

My model looks like this:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv1d_1 (Conv1D)            (None, 76, 32)            640       
_________________________________________________________________
batch_normalization_1 (Batch (None, 76, 32)            128       
_________________________________________________________________
activation_1 (Activation)    (None, 76, 32)            0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 76, 32)            0         
_________________________________________________________________
conv1d_2 (Conv1D)            (None, 76, 32)            3072      
_________________________________________________________________
batch_normalization_2 (Batch (None, 76, 32)            128       
_________________________________________________________________
activation_2 (Activation)    (None, 76, 32)            0         
_________________________________________________________________
dropout_2 (Dropout)          (None, 76, 32)            0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 2432)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 2)                 4864      
_________________________________________________________________
batch_normalization_3 (Batch (None, 2)                 8         
_________________________________________________________________
activation_3 (Activation)    (None, 2)                 0         
=================================================================
Total params: 8,840
Trainable params: 8,708
Non-trainable params: 132
_________________________________________________________________

I am a little lost as to what the error message means. Am I setting the target layer index incorrectly? As far as I understand, the target layer should be layer "dense_1" (I am doing binary classification). But if I try:

deeplift_contribs_func = deeplift_model.get_target_contribs_func(find_scores_layer_idx=find_scores_layer_idx,target_layer_idx=-3)

I get the following error:

RuntimeError: There is a layer after your target layer but it is not an activation layer, which seems odd...if doing regression, make sure to set the target layer to the last layer

So maybe I cannot use BatchNormalization after the final dense layer? Sorry for a rather lengthy post, any help would be greatly appreciated :)

[AvantiShri]

Hi @hartonen, yes, the target layer should be dense_1 and you are doing things correctly. This one is my fault; that "RuntimeError" should just be a warning because I put it in to catch cases where people accidentally specified the wrong index for the target layer (e.g. because they copied the settings from the quickstart without adjusting the settings for their specific architecture). I was not expecting an architecture like yours where batchnorm is applied to the very last layer. I have made a pull request where I change the error to just be a warning at https://github.com/kundajelab/deeplift/pull/93/files - could you try it out and see if it addresses your issues? If so, I can merge it into the master branch.

[hartonen]

Hi @AvantiShri and thank you for taking the time to make the pull request!

When trying out the pull request you linked, I ran into another problem. It seems that I am only able to convert my model using deeplift version 0.6.9.0, but if I try newer versions I get an error. Here is first the output when using deeplift 0.6.9.0 (this is the release I had installed):

pip install deeplift==0.6.9.0
Collecting deeplift==0.6.9.0
Requirement already satisfied: numpy>=1.9 in /home/local/thartone/virtualenv_test/p3venv/lib/python3.6/site-packages (from deeplift==0.6.9.0) (1.15.1)
Installing collected packages: deeplift
Successfully installed deeplift-0.6.9.0

Then opening ipython and converting the keras model to deeplift model:

In [4]: import deeplift
   ...: from deeplift.conversion import kerasapi_conversion as kc
   ...: 
   ...: 

In [5]: deeplift_model = kc.convert_model_from_saved_files('model-900-0.81.h5',nonlinear_mxts_mode=deeplift.layers.NonlinearMxtsMode.DeepLIFT_GenomicsDefault)
nonlinear_mxts_mode is set to: DeepLIFT_GenomicsDefault
For layer 2 the preceding linear layer is 0 of type Conv1D;
In accordance with nonlinear_mxts_mode=DeepLIFT_GenomicsDefault we are setting the NonlinearMxtsMode to Rescale
For layer 6 the preceding linear layer is 4 of type Conv1D;
In accordance with nonlinear_mxts_mode=DeepLIFT_GenomicsDefault we are setting the NonlinearMxtsMode to Rescale
Heads-up: I assume sigmoid is the output layer, not an intermediate one; if it's an intermediate layer then please bug me and I will implement the grad func
For layer 11 the preceding linear layer is 9 of type Dense;
In accordance with nonlinear_mxts_modeDeepLIFT_GenomicsDefault we are setting the NonlinearMxtsMode to RevealCancel

In [6]: deeplift.__version__
Out[6]: '0.6.9.0'

So the model is converted nicely!

Then if I install the newest version of deeplift (this is also the version I get when I clone deeplift repo and do 'pip install --editable', which I did before trying the pull request):

pip install deeplift
Collecting deeplift
Requirement already satisfied: numpy>=1.9 in /home/local/thartone/virtualenv_test/p3venv/lib/python3.6/site-packages (from deeplift) (1.15.1)
Installing collected packages: deeplift
Successfully installed deeplift-0.6.9.1

I get the following error:

In [1]: import deeplift
   ...: from deeplift.conversion import kerasapi_conversion as kc
   ...: 
   ...: 

In [2]: deeplift_model = kc.convert_model_from_saved_files('model-900-0.81.h5',nonlinear_mxts_mode=deeplift.layers.NonlinearMxtsMode.DeepLIFT_GenomicsDefault)
nonlinear_mxts_mode is set to: DeepLIFT_GenomicsDefault
WARNING: Logging before flag parsing goes to stderr.
W0108 12:31:10.828082 140135447795456 deprecation_wrapper.py:119] From /home/local/thartone/virtualenv_test/p3venv/lib/python3.6/site-packages/deeplift/layers/core.py:201: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-2-dafd3396951c> in <module>()
----> 1 deeplift_model = kc.convert_model_from_saved_files('model-900-0.81.h5',nonlinear_mxts_mode=deeplift.layers.NonlinearMxtsMode.DeepLIFT_GenomicsDefault)

/home/local/thartone/virtualenv_test/p3venv/lib/python3.6/site-packages/deeplift/conversion/kerasapi_conversion.py in convert_model_from_saved_files(h5_file, json_file, yaml_file, **kwargs)
    398             layer_config["config"]["weights"] = layer_weights
    399 
--> 400     return model_conversion_function(model_config=model_config, **kwargs)
    401 
    402 

/home/local/thartone/virtualenv_test/p3venv/lib/python3.6/site-packages/deeplift/conversion/kerasapi_conversion.py in convert_sequential_model(model_config, nonlinear_mxts_mode, verbose, dense_mxts_mode, conv_mxts_mode, maxpool_deeplift_mode, layer_overrides, custom_conversion_funcs)
    455                 maxpool_deeplift_mode=maxpool_deeplift_mode,
    456                 converted_layers=converted_layers,
--> 457                 layer_overrides=layer_overrides)
    458     converted_layers[-1].build_fwd_pass_vars()
    459     return models.SequentialModel(converted_layers)

/home/local/thartone/virtualenv_test/p3venv/lib/python3.6/site-packages/deeplift/conversion/kerasapi_conversion.py in sequential_container_conversion(config, name, verbose, nonlinear_mxts_mode, dense_mxts_mode, conv_mxts_mode, maxpool_deeplift_mode, converted_layers, layer_overrides, custom_conversion_funcs)
    494                                        else "")+str(layer_idx),
    495                                  verbose=verbose,
--> 496                                  **modes_to_pass)) 
    497         else:
    498             print("Encountered an Input layer in sequential container; "

/home/local/thartone/virtualenv_test/p3venv/lib/python3.6/site-packages/deeplift/conversion/kerasapi_conversion.py in conv1d_conversion(config, name, verbose, nonlinear_mxts_mode, conv_mxts_mode, **kwargs)
    151                         else "")+name,
    152             kernel=config[KerasKeys.weights][0],
--> 153             bias=config[KerasKeys.weights][1],
    154             stride=config[KerasKeys.strides],
    155             padding=config[KerasKeys.padding].upper(),

IndexError: list index out of range

And weirdly enough now, python claims the version is 0.6.8.1:

In [3]: deeplift.__version__
Out[3]: '0.6.8.1'

even though pip says it is 0.6.9.1:

pip show deeplift
Name: deeplift
Version: 0.6.9.1
Summary: DeepLIFT (Deep Learning Important FeaTures)
Home-page: https://github.com/kundajelab/deeplift
Author: None
Author-email: None
License: UNKNOWN
Location: /home/local/thartone/virtualenv_test/p3venv/lib/python3.6/site-packages
Requires: numpy
Required-by:

The virtual environment used is otherwise identical in test made with deeplift versions 0.6.9.1 and 0.6.9.0. My model does not use bias terms in convolutional filters so could this be something that in version 0.6.9.1 this is required? Or have I somehow messed up my installation since after installing the version 0.6.9.1 with pip, python still claims the deeplift version is 0.6.8.1?

[AvantiShri]

Hi @hartonen,

Don’t worry about “0.6.8.1” showing up when you do deeplift.___version__; there are 3 places where I need to update the version each time I make a new release and I often forget to update the version number in __init__.py, which is the one that is accessed by deeplift.__version__.

I am surprised that the model conversion worked with v0.6.9.0 because I would have expected that error to be thrown if you supplied a model that had no bias terms even with v0.6.9.0. I can make a fix that would avoid the error. Can you upload a saved model file so I can try reproducing the error?

As an aside, the latest version on the master branch is v0.6.9.2, and the version on the branch I made for you is v0.6.9.3, so I think you may not have done a “git pull” to get the latest version - however I think even if you had, you would have still gotten the error, which is why I am requesting a model file for debugging purposes (if you don’t want to share your actual model that’s fine; any model that can reproduce the error would do).

[AvantiShri]

Ok, I pushed some updates to #93 that should address the error. You could test it out on your computer (remember to switch to the targetlayerwarningfix branch), and if you have any issues feel free to upload a model that I could use for testing purposes

[hartonen]

Hi @AvantiShri and thanks again for your response and help!

So I have now cloned the latest version of deeplift repo and swithced to branch targetlayerwarningfix:

(deeplift_test) thartone@dx4-csb-01:/home/ad/home/t/thartone/GitHub/deeplift$ git status
On branch targetlayerwarningfix
Your branch is up-to-date with 'origin/targetlayerwarningfix'.
nothing to commit, working directory clean

These are the packages and their versions I have installed:

(deeplift_test) thartone@dx4-csb-01:/home/ad/home/t/thartone/GitHub/deeplift$ pip freeze
absl-py==0.9.0
astor==0.8.1
backcall==0.1.0
decorator==4.4.1
-e git+https://github.com/kundajelab/deeplift.git@2fd2a3e80a65291fb050970928cb12bc6bf077f1#egg=deeplift
gast==0.3.2
google-pasta==0.1.8
grpcio==1.26.0
h5py==2.9.0
ipython==7.11.1
ipython-genutils==0.2.0
jedi==0.15.2
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
Markdown==3.1.1
numpy==1.15.1
parso==0.5.2
pexpect==4.7.0
pickleshare==0.7.5
prompt-toolkit==3.0.2
protobuf==3.11.2
ptyprocess==0.6.0
Pygments==2.5.2
PyYAML==5.3
scipy==1.1.0
six==1.13.0
tensorboard==1.14.0
tensorflow==1.14.0
tensorflow-estimator==1.14.0
termcolor==1.1.0
traitlets==4.3.3
wcwidth==0.1.8
Werkzeug==0.16.0
wrapt==1.11.2

This is the output I get when trying to convert my model using the branch targetlayerwarningfix:

In [8]: import deeplift

In [9]: from deeplift.conversion import kerasapi_conversion as kc

In [10]: deeplift_model = kc.convert_model_from_saved_files('model-02-0.75.h5',nonlinear_mxts_mode=deeplift.layers.NonlinearMxtsMode.DeepLIFT_GenomicsDefault) 
nonlinear_mxts_mode is set to: DeepLIFT_GenomicsDefault
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-11-6c3dc1984fa6> in <module>()
----> 1 deeplift_model = kc.convert_model_from_saved_files('model-02-0.75.h5',nonlinear_mxts_mode=deeplift.layers.NonlinearMxtsMode.DeepLIFT_GenomicsDefault)

/home/ad/home/t/thartone/GitHub/deeplift/deeplift/conversion/kerasapi_conversion.py in convert_model_from_saved_files(h5_file, json_file, yaml_file, **kwargs)
    401             layer_config["config"]["weights"] = layer_weights
    402 
--> 403     return model_conversion_function(model_config=model_config, **kwargs)
    404 
    405 
...

I have truncated the error message as it is quite long.

...
/home/ad/home/t/thartone/GitHub/deeplift/deeplift/layers/convolutional.py in _build_activation_vars(self, input_act_vars)
     66                                 input_act_vars,
     67                                 kernel=self.kernel)
---> 68         return conv_without_bias + self.bias[None,None,:]
     69 
     70     def _build_pos_and_neg_contribs(self):

TypeError: 'float' object is not subscriptable

Anyway, the conversion works with otherwise the same setup when I replace the latest version of deeplift with the version 0.6.9.0. I have attached below a model that should reproduce the error as a .gz-file. Structure of the model should look like this:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv1d_1 (Conv1D)            (None, 76, 32)            640       
_________________________________________________________________
batch_normalization_1 (Batch (None, 76, 32)            128       
_________________________________________________________________
activation_1 (Activation)    (None, 76, 32)            0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 76, 32)            0         
_________________________________________________________________
conv1d_2 (Conv1D)            (None, 76, 32)            3072      
_________________________________________________________________
batch_normalization_2 (Batch (None, 76, 32)            128       
_________________________________________________________________
activation_2 (Activation)    (None, 76, 32)            0         
_________________________________________________________________
dropout_2 (Dropout)          (None, 76, 32)            0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 2432)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 2)                 4864      
_________________________________________________________________
batch_normalization_3 (Batch (None, 2)                 8         
_________________________________________________________________
activation_3 (Activation)    (None, 2)                 0         
=================================================================
Total params: 8,840
Trainable params: 8,708
Non-trainable params: 132
_________________________________________________________________

This is the gz-file contianing my model, let e know if it does not work: model-02-0.75.h5.gz

[AvantiShri]

Thanks for the info. I pushed a fix for the TypeError: 'float' object is not subscriptable error but I have not had a chance to test it yet. I have to work on some other stuff at the moment but let me know if you get a chance to try out the fix first.

[hartonen]

Hi @AvantiShri ,

I pulled the latest fixes for branch targetlayerwarningfix, and this is the output I get when trying to convert my model:

In [6]: deeplift_model = kc.convert_model_from_saved_files('model-02-0.75.h5',nonlinear_mxts_mode=deeplift.layers.NonlinearMxtsMode.DeepLIFT_GenomicsDefault)
nonlinear_mxts_mode is set to: DeepLIFT_GenomicsDefault
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-4f6c37faaf05> in <module>()
----> 1 deeplift_model = kc.convert_model_from_saved_files('model-02-0.75.h5',nonlinear_mxts_mode=deeplift.layers.NonlinearMxtsMode.DeepLIFT_GenomicsDefault)

/home/ad/home/t/thartone/GitHub/deeplift/deeplift/conversion/kerasapi_conversion.py in convert_model_from_saved_files(h5_file, json_file, yaml_file, **kwargs)
    401             layer_config["config"]["weights"] = layer_weights
    402 
--> 403     return model_conversion_function(model_config=model_config, **kwargs)
    404 
    405 

/home/ad/home/t/thartone/GitHub/deeplift/deeplift/conversion/kerasapi_conversion.py in convert_sequential_model(model_config, nonlinear_mxts_mode, verbose, dense_mxts_mode, conv_mxts_mode, maxpool_deeplift_mode, layer_overrides, custom_conversion_funcs)
    458                 maxpool_deeplift_mode=maxpool_deeplift_mode,
    459                 converted_layers=converted_layers,
--> 460                 layer_overrides=layer_overrides)
    461     converted_layers[-1].build_fwd_pass_vars()
    462     return models.SequentialModel(converted_layers)

/home/ad/home/t/thartone/GitHub/deeplift/deeplift/conversion/kerasapi_conversion.py in sequential_container_conversion(config, name, verbose, nonlinear_mxts_mode, dense_mxts_mode, conv_mxts_mode, maxpool_deeplift_mode, converted_layers, layer_overrides, custom_conversion_funcs)
    497                                        else "")+str(layer_idx),
    498                                  verbose=verbose,
--> 499                                  **modes_to_pass)) 
    500         else:
    501             print("Encountered an Input layer in sequential container; "

/home/ad/home/t/thartone/GitHub/deeplift/deeplift/conversion/kerasapi_conversion.py in conv1d_conversion(config, name, verbose, nonlinear_mxts_mode, conv_mxts_mode, **kwargs)
    153             kernel=config[KerasKeys.weights][0],
    154             bias=(config[KerasKeys.weights][1]
--> 155                   if len(config[KerasKeys.weights]) > 1 else np.zeros(1.0)),
    156             stride=config[KerasKeys.strides],
    157             padding=config[KerasKeys.padding].upper(),

TypeError: 'float' object cannot be interpreted as an integer

Hopefully this is of help!

[AvantiShri]

Hello @hartonen, I implemented a fix and tested it out with your model: https://colab.research.google.com/gist/AvantiShri/d3344f1c3a64354d397b621a889c21ce/testingnobiasfix.ipynb - the model conversion and get_target_contribs_func seem to work; let me know if you run into any other issues

If this works for you, I will merge the feature into the master branch.

As an aside, it turns out that the reason it worked with v0.6.9.0 on pypi is that I had accidentally created that pypi build from a different branch than the master branch, and that different branch had some extra features (including this fix). I didn't finish working on those features and created the build for subsequent versions from the master branch, which is why they did not contain this fix.

Thanks for your patience,
Av

[hartonen]

Hi @AvantiShri , I pulled the new fixes and I can confirm that I was able to convert my model and to calculate importance scores. So everything seems to be working. Thank you very much for taking the time to look at my issues!

Cheers,
Tuomo