convert our model(s) from onnx to tf

[cwmeijer]

No description provided.

[cwmeijer]

use
https://github.com/onnx/onnx-tensorflow/graphs/contributors
first try it on our models.

[cwmeijer]

Explored it with a single model (leafsnap) and predictions were equivalent to the original onnx. Script can be found here:
https://github.com/dianna-ai/dianna-exploration/blob/56-onnx-to-tf-exploration/onnx_conversion_scripts/onnx_to_tensorflow.ipynb

[elboyran]

Can somebody who has time, please test the Tutorial notebook of onnx-tensorflow executing exactly the same notebook and just appending it with the shap step to see if it works?

[loostrum]

That tutorial didn't work for me even without changing anything, the model they provide fails when running prepare (see also the discussion in the teams channel). I couldn't find which versions of e.g. tensorflow they used.

I did do some tests together with Chris and Yang, see this notebook: https://github.com/dianna-ai/dianna-exploration/blob/test-shap-tf-onnx/onnx_conversion_scripts/test_shap_tf_onnx.ipynb
We're getting further, i.e. we can generate an explainer object, but getting the shap values still fails.
shap also fails on a native keras model (see the same notebook), it looks like the DeepExplainer is not (fully) compatible with tensorflow 2, or at least not with recent versions.

[cwmeijer]

2 remarks:

• Make a draft PR with a failing test at https://github.com/slundberg/shap/pulls.
• Post error messages with version details here to get help easier

[elboyran]

Note that the relevant branches are in dianna-exploration. If it results in a PR we should not forget it in the stand-ups.

[elboyran]

My interpretations of the errors we got in https://github.com/dianna-ai/dianna-exploration/blob/test-shap-tf-onnx/onnx_conversion_scripts/test_shap_tf_onnx.ipynb:

For our model- the warning after cell [21] about exporting a tensor dict might be related to the error later after cell [24]:

TypeError: Expected any non-tensor type, got a tensor instead.

Looks like versioning issues indeed. But maybe we can try not with mnist. but another or our models? I recall @geek-yang saying the mnist had a different structure after conversion than the others.

For the native Keras model: I can report our error also in their issue, hoping someone will pay attention. Also worth looking at how the skomatin guy changed his model and avoided the error (post from 11 Sep 2020). He thinks shap does not support TF 2.0 :-( @loostrum what shap and TF versions did you use?

[elboyran]

One more tough: if DeepExplainer attempts ultimately fail, any point of trying the slower Shap's KernalExplainer?

https://github.com/slundberg/shap

[loostrum]

I tried the latest version of shap (0.40), a slightly older version (0.38) and the latest master branch.
Using tensorflow 2.6