Negative ignore_index for the Accuracy metric

[yassersouri]

As with the CrossEntropy loss in Pytorch, it is usual to have negative value for ignore_index.
Currently the check in _check_classifiction_inputs (specifically _basic_input_validation) does not allow a negative value for target.

I have currently created some tests to show the problem.
If this is something that you think should be fixed, I can write a fix for it.

My proposal is to add ignore_index as an additional input to the _basic_input_validation and slightly change the implementation.
Please let me know if this is something that makes sense and you guys want it fixed.

[pep8speaks]

Hello @yassersouri! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-07-26 07:40:46 UTC

[codecov]

Codecov Report

Merging #362 (e804319) into master (21ba650) will decrease coverage by 0%.
The diff coverage is 96%.

@@          Coverage Diff          @@
##           master   #362   +/-   ##
=====================================
- Coverage      95%    95%   -0%     
=====================================
  Files         167    167           
  Lines        6906   6931   +25     
=====================================
+ Hits         6565   6585   +20     
- Misses        341    346    +5     

[yassersouri]

Codecov Report

Merging #362 (6667e8c) into master (3de06f9) will decrease coverage by 20.05%.

How is it possible to decrease the coverage by adding a test?!

[yassersouri]

@SkafteNicki All I need is a thumbs up or thumbs down on this.
Currently for me it makes a lot of sense to have negative value for ignore_index.

[SkafteNicki]

@yassersouri fine by me. The changes probably need to be done to the StatScore class as the other classification metric inherit from this.

[Borda]

@yassersouri @SkafteNicki how is it going here? 🐰

[yassersouri]

Hi again! Sorry for disappearing for a while. I will work on this pull request again.
There were some issues regarding the way the ignored items are removed from the predictions and targets that won't work in this new scenario. I am thinking about two ways to solve it.

1. Adding a new index (num_classes + 1) for the ignore index and deal with it that way.
2. Converting the inputs from d-dimensional to 1-dimensional so it is easier to remove the ignored indices.

Currently I think option 1 is messy as it requires making changes at multiple locations. I will try to implement version 2 and then gather your feedback.

[yassersouri]

I just resolved the conflicts, please ignore the review request.

[Borda]

This seems quite abundant; shall we close it?
cc: @PyTorchLightning/core-metrics

[tridao]

Having this option would be quite useful.
For example, training masked language models (e.g. BERT) involves computing accuracy only on the masked tokens, while the other tokens are replaced with index -100. Currently torchmetrics.Accuracy errors with "The target has to be a non-negative tensor".

[stancld]

@Borda @SkafteNicki Could you please have a look at this PR? I tried to fix some minor issues but I'm still not fully familiar with the torchmetrics classification package :]

[Borda]

lgtm

[SkafteNicki]

Also LGTM, this should probably be extended to other classes when we try to refactor classification