Extend `IntegratedGradients` `target` parameter selection to support rank > 2 model outputs #635

gipster · 2022-04-21T09:17:05Z

This pull request addresses the issue related to target dimensionality in the IntegratedGradients class.

The present code behaves correctly for the most common use cases (classification and regression models with output dimensionality <= 2) when the targets passed are correctly formatted. However, the _select_target function returns incorrect outputs for output dimensions > 2 and in some case doesn't raise errors when unusual targets are passed.

This pull request should address this issue with the following changes

Substitute the line
```
preds = tf.linalg.diag_part(tf.gather(preds, targets, axis=1))
```
with the line
```
preds = tf.gather_nd(preds, tf.expand_dims(targets, axis=1), batch_dims=1)
```
in the function _select_target . In this new version, passing out or range target will raise an error for any output dimension.
Added the function _check_target ensuring that the target dimensionality is compatible with the model's output dimensionality

codecov · 2022-04-21T15:58:54Z

Codecov Report

Merging #635 (3e44b6c) into master (61f0c6a) will increase coverage by 0.03%.
The diff coverage is 88.88%.

@@            Coverage Diff             @@
##           master     #635      +/-   ##
==========================================
+ Coverage   82.06%   82.10%   +0.03%     
==========================================
  Files          77       77              
  Lines       10519    10570      +51     
==========================================
+ Hits         8632     8678      +46     
- Misses       1887     1892       +5

Impacted Files	Coverage Δ
alibi/explainers/integrated_gradients.py	`88.70% <82.85%> (-0.35%)`	⬇️
...libi/explainers/tests/test_integrated_gradients.py	`97.06% <100.00%> (+0.14%)`	⬆️

gipster · 2022-04-22T08:21:02Z

Added exceptions in the _check_target function to ensure that any badly formatted targets will result in a error.
Added tests for the functions _check_target and _select_target.
Converted the output of _get_target_from_target_fn into integers types. Targets must always be integers.

jklaise · 2022-04-22T14:48:47Z

alibi/explainers/integrated_gradients.py

+            raise ValueError(f"First dimension in target must be egual to nb of samples. "
+                             f"Found target 1st dimension {target.shape[0]}; nb samples: {nb_samples}")


egual -> equal
dimension -> dimension:

jklaise · 2022-04-22T14:51:21Z

alibi/explainers/integrated_gradients.py

+        if len(target.shape) > 2:
+            raise ValueError("Targets must be 1-d or 2-d arrays. In 2-d arrays, each column must contain "
+                             "the target index of the corresponding dimension in the model's output tensor.")


Slightly confused why target can be 2d, can you elaborate? Probably would need to update the explain docstring then too as it only talks about None, int, List, np.ndarray and no dimensions.

ok, I think I used the word dimension when I should have used the word rank.

With gather_nd, you can actually get the correct target for output tensors with rank higher than 2.

For examples, if you have a classification problem with 10 classes, your output tensor will be a rank 2 tensor (a matrix) of dimension nb_samples X 10 . Now in this case you target must be rank 1 (1-d), of length nb_samples, and with range 0-9, each value representing the column position in the output tensor.

If you have a tensor of higher rank as output, for example an auto encoder with (nb_samples, 28, 28, 3) rank 3 tensor in the output (ignoring the nb_samples dimension), you need a rank 2 target (a matrix) nb_samples X 3 , where the first 2 columns have range 0-27 and the last column have range 0-2.

In other words, for output ranks > 1 , if you have a rank-n tensor as output, you need a rank 2 tensor with dimensions nb_samples x n. Each row represents the location of an element in the output tensor.

Not sure if it's clear enough

I think that makes sense, perhaps worth writing a bit about it in the docstrings (not necessary to be comprehensive as we don't currently have examples on outputs of >2 rank. captum describes it quite well: https://captum.ai/api/integrated_gradients.html

alibi/explainers/integrated_gradients.py

jklaise · 2022-04-22T14:58:32Z

alibi/explainers/integrated_gradients.py

+            tmax, tmin = target.max(axis=0), target.min(axis=0)
+
+            if tmax > 1:
+                raise ValueError(f"Targets values {tmax} out of range for output shape {output_shape[-1]} ")


Please check grammar here and elsewhere.

Also perhaps need to make the message less confusing in general (here and elsewhere), e.g. consider:
ValueError: Targets value 4 out of range for output shape 2

But output_shape is the whole tuple not the length of the last dimension.

jklaise · 2022-04-22T15:00:22Z

alibi/explainers/integrated_gradients.py

+            raise ValueError("Targets must be 1-d or 2-d arrays. In 2-d arrays, each column must contain "
+                             "the target index of the corresponding dimension in the model's output tensor.")
+
+        if len(output_shape) == 1:


This whole branch is only valid if the (implicit) task is classification and outputs are labels rather than probabilities (hence output shape is 1-d), right? Can you add a comment here to clarify?

It's valid when you have the binary classification with single squash output (probability of class 1 but returns tensor of shape (nb_samples) instead of (nb_samples, 1) ).

Hmm remind me, is it valid to pass a model that outputs labels instead of probabilities? Or does the model always have to be probabilistic (in the classification case that is)?

Not sure about that.

alibi/explainers/tests/test_integrated_gradients.py

jklaise · 2022-04-22T15:12:36Z

alibi/explainers/tests/test_integrated_gradients.py

+               {'preds': np.array([[[0.0, 0.1], [1.0, 1.1]],
+                                   [[2.0, 2.1], [3.0, 4.1]]]),
+                'target': np.array([[0, 0],
+                                   [0, 0]]),
+                'expected': np.array([[0.0],
+                                      [2.0]])}]


I see that this test case tests a 3-dimensional output and a 2-dimensional target which partially answers my question above why 2D targets are allowed. However, I though we only support 2-dimensional outputs anyway?

Does this mean that this PR implicitly allows up to 3D output?

yes, that's right. Any rank for the output tensor are allowed now, not only 3D. Refer to comment above

jklaise · 2022-04-22T16:21:46Z

alibi/explainers/integrated_gradients.py

            For regression models whose output is a scalar, target should not be provided.
            For classification models `target` can be either the true classes or the classes predicted by the model.
+            It must be provided if the model output dimension is higher than 1.


Make the language consistent to always talk about the rank of the output (instead of dimesnion)?

no, but in this case dimension doesn't refer to the rank of the tensor, it refers to the number of classes

Ok so in that case the sentence should be changed so it's clear it applies to classification and talks about classes (as opposed to the niche case of 2-class classification with 1-class output probability?).

jklaise · 2022-04-22T16:22:44Z

alibi/explainers/integrated_gradients.py

            For regression models whose output is a scalar, target should not be provided.
            For classification models `target` can be either the true classes or the classes predicted by the model.
+            It must be provided if the model output dimension is higher than 1.
+            If the model's output is a rank-n tensor with n > 2,
+            the target must a rank-2 numpy array or a list of lists (a matrix) with dimensions nb_samples X (n-1) .


must -> must be

jklaise · 2022-04-22T16:26:50Z

alibi/explainers/integrated_gradients.py

+            tmax, tmin = target.max(axis=0), target.min(axis=0)
+
+            if tmax > 1:
+                raise ValueError(f"Target value {tmax} out of range for output shape = 1 ")


Should this be either out of range for output shape {output_shape} or out of range for rank-1 output instead?

ok, I think it's actually better to put for output shape {output_shape} everywhere

jklaise · 2022-04-22T16:31:28Z

alibi/explainers/integrated_gradients.py

+        if out_rank != target_rank:
+            raise ValueError(f"The last dimension of target  must match the rank of the model's output tensor. "
+                             f"Found target last dimension: {target_rank}; model's output rank: {out_rank}")


Should this be done as the very first check as it could mess up everything immediately if not passed right?

Although I'm confused since it seems target_rank is not always len(target.shape) as it's defined as target.shape[-1] for rank>2 outputs...

The case with output_shape > 2 is different because the last dimension of the target must match the rank of the output tensor (excluding batch dimension )

jklaise · 2022-04-22T16:32:21Z

alibi/explainers/integrated_gradients.py

+                raise ValueError(f"Target value {tmax} out of range for output shape = 1 ")
+
+        elif len(output_shape) == 2:
+            out_rank, target_rank = 1, len(target.shape)


I fee like out_rank should be 2 here but it seems you are using different conventions in different places which makes it more confusing...

I.e. are we ignoring batch dimension in some cases but not others?

no, because out_rank doesn't include the dimension referring to the number of samples

we are not ignoring it only in the case of the squash output (output_shape = 1)

But if len(output_shape) == 1 then out_rank=1 so we are not ignoring batch dimension in this case, but are ignoring it in all other cases?

It's just confusing to read that out_rank=1 in both len(output_shape) == 1 and len(output_shape) == 2... (and similar for the target_rank). Is there a better way to code this up or at least comment in the code?

(I think supporting squash-output for 2-class classification is giving a lot of headache here and elsewhere...)

well it gave me a headache yesterday when I was writing these warnings and getting errors for squash output

gipster added 3 commits April 19, 2022 18:03

improved _select_target function

2192589

_check_target function

57e9f1c

_check_target function corrected

c85adeb

gipster linked an issue Apr 21, 2022 that may be closed by this pull request

Integrated Gradients target #627

Closed

gipster added 4 commits April 21, 2022 11:32

fixed _check_target position

349c609

added range check to _check_target

afa575a

flake8 f string placeholder

572e2d6

raise error if target is not int

8c7f21c

gipster added 3 commits April 21, 2022 18:52

_check_target and _select_target test added

0da82f7

transform target_fn in integers

294c1f1

test _check_target also with good targets

3a0b508

my god, flake8

f468188

jklaise reviewed Apr 22, 2022

View reviewed changes

alibi/explainers/integrated_gradients.py Outdated Show resolved Hide resolved

jklaise reviewed Apr 22, 2022

View reviewed changes

alibi/explainers/tests/test_integrated_gradients.py Outdated Show resolved Hide resolved

jklaise reviewed Apr 22, 2022

View reviewed changes

gipster added 3 commits April 22, 2022 17:57

better messages, doc string modified, variable names changed

11c93b4

docstring

d4b45b7

docstring 2

f89b4be

jklaise reviewed Apr 22, 2022

View reviewed changes

'must be' must be correct

c3900fe

jklaise reviewed Apr 22, 2022

View reviewed changes

gipster added 2 commits April 22, 2022 18:44

grammar fixes in warnings

ae9c755

docstrig changed

3e44b6c

jklaise added this to the 0.7.0 milestone Apr 26, 2022

jklaise approved these changes Apr 26, 2022

View reviewed changes

jklaise changed the title ~~Intgrads select target~~ Extend IntegratedGradients target parameter selection to support rank > 2 model outputs Apr 26, 2022

jklaise merged commit 7d898c1 into SeldonIO:master Apr 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend `IntegratedGradients` `target` parameter selection to support rank > 2 model outputs #635

Extend `IntegratedGradients` `target` parameter selection to support rank > 2 model outputs #635

gipster commented Apr 21, 2022

codecov bot commented Apr 21, 2022 •

edited

Loading

gipster commented Apr 22, 2022

jklaise Apr 22, 2022

jklaise Apr 22, 2022

gipster Apr 22, 2022

jklaise Apr 22, 2022

jklaise Apr 22, 2022

jklaise Apr 22, 2022

jklaise Apr 22, 2022

gipster Apr 22, 2022

jklaise Apr 22, 2022

gipster Apr 22, 2022

jklaise Apr 22, 2022

gipster Apr 22, 2022

jklaise Apr 22, 2022

gipster Apr 22, 2022

jklaise Apr 22, 2022

jklaise Apr 22, 2022

jklaise Apr 22, 2022

gipster Apr 22, 2022

jklaise Apr 22, 2022

gipster Apr 22, 2022

jklaise Apr 22, 2022 •

edited

Loading

jklaise Apr 22, 2022

gipster Apr 22, 2022

gipster Apr 22, 2022

jklaise Apr 22, 2022 •

edited

Loading

jklaise Apr 22, 2022

gipster Apr 22, 2022

		raise ValueError(f"First dimension in target must be egual to nb of samples. "
		f"Found target 1st dimension {target.shape[0]}; nb samples: {nb_samples}")

Extend IntegratedGradients target parameter selection to support rank > 2 model outputs #635

Extend IntegratedGradients target parameter selection to support rank > 2 model outputs #635

Conversation

gipster commented Apr 21, 2022

codecov bot commented Apr 21, 2022 • edited Loading

Codecov Report

gipster commented Apr 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jklaise Apr 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jklaise Apr 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Extend `IntegratedGradients` `target` parameter selection to support rank > 2 model outputs #635

Extend `IntegratedGradients` `target` parameter selection to support rank > 2 model outputs #635

codecov bot commented Apr 21, 2022 •

edited

Loading

jklaise Apr 22, 2022 •

edited

Loading

jklaise Apr 22, 2022 •

edited

Loading