Skip to content

Commit

Permalink
docs causal fidelity: add remarks
Browse files Browse the repository at this point in the history
  • Loading branch information
AntoninPoche committed Nov 26, 2021
1 parent 9b2d80d commit ea27182
Show file tree
Hide file tree
Showing 2 changed files with 46 additions and 2 deletions.
24 changes: 23 additions & 1 deletion docs/api/deletion.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,29 @@ the important pixels.

-- <cite>[RISE: Randomized Input Sampling for Explanation of Black-box Models (2018)](https://arxiv.org/abs/1806.07421)</cite>[^1]

The better the method, the smaller the score.

## Score interpretation

If explanations are accurate, the score will quickly fall from the score on non-perturbed input to the score of a random predictor.
Thus, in this case, a lower score represent a more accurate explanation.


## Remarks

This metric only evaluate the order of importance between features.

The parameters metric, steps and max_percentage_perturbed may drastically change the score :

- For inputs with many features, increasing the number of steps will allow you to capture more efficiently the difference between attributions methods.

- The order of importance of features with low importance may not matter, hence, decreasing the max_percentage_perturbed,
may make the score more relevant.

Sometimes, attributions methods also returns negative attributions,
for those methods, do not take the absolute value before computing insertion and deletion metrics.
Otherwise, negative attributions may have higher absolute values, and the order of importance between features will change.
Therefore, the score may not represent the pertinence of the explanation.


## Example

Expand Down
24 changes: 23 additions & 1 deletion docs/api/insertion.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,29 @@ The Insertion Fidelity metric measures how well a saliency-map–based explanati

-- <cite>[RISE: Randomized Input Sampling for Explanation of Black-box Models (2018)](https://arxiv.org/abs/1806.07421)</cite>[^1]

The better the method, the higher the score.

## Score interpretation

If explanations are accurate, the score will quickly rise to the score on non-perturbed input.
Thus, in this case, a higher score represent a more accurate explanation.


## Remarks

This metric only evaluate the order of importance between features.

The parameters metric, steps and max_percentage_perturbed may drastically change the score :

- For inputs with many features, increasing the number of steps will allow you to capture more efficiently the difference between attributions methods.

- The order of importance of features with low importance may not matter, hence, decreasing the max_percentage_perturbed,
may make the score more relevant.

Sometimes, attributions methods also returns negative attributions,
for those methods, do not take the absolute value before computing insertion and deletion metrics.
Otherwise, negative attributions may have higher absolute values, and the order of importance between features will change.
Therefore, the score may not represent the pertinence of the explanation.


## Example

Expand Down

0 comments on commit ea27182

Please sign in to comment.