Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Keras images) Add an optional image argument, and other improvements #329

Merged
merged 21 commits into from Aug 10, 2019

Conversation

@teabolt
Copy link
Contributor

teabolt commented Aug 5, 2019

API improvements for #315.

For Keras image classifiers, the call for explanations is eli5.explain_prediction(model, doc) . We will add the ability to pass an optional image argument like this: eli5.explain_prediction(model, doc, image=image), where image is a Pillow image corresponding to doc. In this case we will no longer use doc to create an image. This is useful for experimental models or unusual input shapes.

These changes should've been introduced in #325 but need to be merged earlier. See also #315 (comment). (Originally image was going to be a required argument, but we decided to keep it optional).

@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Aug 5, 2019

Codecov Report

Merging #329 into master will increase coverage by 0.02%.
The diff coverage is 96.36%.

@@            Coverage Diff             @@
##           master     #329      +/-   ##
==========================================
+ Coverage   97.29%   97.31%   +0.02%     
==========================================
  Files          49       49              
  Lines        3107     3134      +27     
  Branches      581      584       +3     
==========================================
+ Hits         3023     3050      +27     
  Misses         44       44              
  Partials       40       40
Impacted Files Coverage Δ
eli5/keras/gradcam.py 100% <100%> (ø) ⬆️
eli5/formatters/image.py 100% <100%> (+2.73%) ⬆️
eli5/keras/explain_prediction.py 96.38% <95.65%> (-1.83%) ⬇️
@teabolt teabolt force-pushed the teabolt:keras-gradcam-img-v2 branch from 0503324 to c4eb039 Aug 5, 2019
@kmike

This comment has been minimized.

Copy link
Contributor

kmike commented Aug 6, 2019

@teabolt I'm not sure I follow the comments in previous PR regarding this. Could you please summarize, why should image argument be required, not optional?

@teabolt

This comment has been minimized.

Copy link
Contributor Author

teabolt commented Aug 6, 2019

I'm not sure I follow the comments in previous PR regarding this. Could you please summarize, why should image argument be required, not optional?

@kmike making image required makes a couple of things easier:

  • We can dispatch to an image or text explanation function easier (if image is passed dispatch to image, if tokens is passed dispatch to text). Otherwise we'd need to check the shape of the doc array which may not be ideal (eg: if we later end up supporting video with array shape (height, width, frames), this has the same shape as an RGB image (height, width, 3)).
  • We don't need code to convert doc to an image. Classic (height, width, channels) arrays can be easily converted to images, but the user would have issues if doc is something else but still represents an image.

Edit: I can see how my arguments are edge cases. We can try infer things from doc and if that fails the user can explicitly pass image?

@kmike

This comment has been minimized.

Copy link
Contributor

kmike commented Aug 6, 2019

Otherwise we'd need to check the shape of the doc array which may not be ideal

Interesting - in most other places we're dispatching based on model, not based on input data type. E.g. you've got ResNet, with 2d convolutions and stuff => dispatch to image handling.

Edit: I can see how my arguments are edge cases. We can try infer things from doc and if that fails the user can explicitly pass image?

Yeah, that's what I was thinking - don't require user to pass image if we can figure it out automatically; if it works in 80% cases that's fine, as soon as there is a way to pass image explicitly.

@teabolt

This comment has been minimized.

Copy link
Contributor Author

teabolt commented Aug 6, 2019

you've got ResNet, with 2d convolutions and stuff => dispatch to image handling.

That is a good idea. For text we can check for Embedding, recurrent, or 1D layers. For images we'd expected 2D layers.

docs/source/libraries/keras.rst Outdated Show resolved Hide resolved
@teabolt teabolt changed the title (Keras images) Add a required image argument for explanations (Keras images) Add an optional image argument, and other improvements Aug 7, 2019
eli5/formatters/image.py Outdated Show resolved Hide resolved
Copy link
Contributor

lopuhin left a comment

Thanks @teabolt , looks great 👍
Left some minor comments above

README.rst Outdated
@@ -51,6 +51,9 @@ It provides support for the following machine learning frameworks and packages:
* sklearn-crfsuite_. ELI5 allows to check weights of sklearn_crfsuite.CRF
models.

* Keras_ - explain predictions of image classifiers via Grad-CAM visualizations.

This comment has been minimized.

Copy link
@kmike

kmike Aug 8, 2019

Contributor

To make it look more 😎 we may move Keras upper the list, e.g. right after scikit-learn (both here and in overview); it can be easy to miss otherwise, as with lightning / sklearn-crfsuite the list starts to be a bit obscure.

Co-Authored-By: Konstantin Lopuhin <kostia.lopuhin@gmail.com>
teabolt and others added 6 commits Aug 8, 2019
Co-Authored-By: Konstantin Lopuhin <kostia.lopuhin@gmail.com>
@lopuhin
lopuhin approved these changes Aug 8, 2019
Copy link
Contributor

lopuhin left a comment

Looks great, thanks @teabolt 👍


def test_explain_prediction_not_supported():
res = eli5.explain_prediction(Sequential(), np.zeros((0,)))

This comment has been minimized.

Copy link
@kmike

kmike Aug 8, 2019

Contributor

is this check for "image task / non-image task" faling only because of doc shape, or because of model as well? If that's only doc, what do you think about adding a case where it fails because of a model?

This comment has been minimized.

Copy link
@teabolt

teabolt Aug 9, 2019

Author Contributor

Exactly, here it fails because of doc and short-circuits. Added another test case.

@teabolt

This comment has been minimized.

Copy link
Contributor Author

teabolt commented Aug 9, 2019

Added a Grad-CAM image (https://github.com/teabolt/eli5/blob/keras-gradcam-img-v2/docs/source/static/gradcam-catdog.png) to the README after text highlighting image.

eli5/keras/explain_prediction.py Outdated Show resolved Hide resolved
@kmike
kmike approved these changes Aug 10, 2019
Copy link
Contributor

kmike left a comment

👍

Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>
@teabolt

This comment has been minimized.

Copy link
Contributor Author

teabolt commented Aug 10, 2019

Thanks @kmike. I'll have to clean up my backslashes in other PR's.

@kmike kmike merged commit ac35c0b into TeamHG-Memex:master Aug 10, 2019
1 of 2 checks passed
1 of 2 checks passed
codecov/patch 96.36% of diff hit (target 97.29%)
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@kmike

This comment has been minimized.

Copy link
Contributor

kmike commented Aug 10, 2019

Thanks @teabolt!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.