Bug in AUC metric when TP = 100%?

[seth-brown]

As an example, this works correctly:

In [13]: import numpy as np                                                                                                                         

In [14]: from sklearn import metrics                                                                                                                

In [15]: true = [1, 1, 1, 1, 1, 1, 1, 1, 1, 0.99]                                                                                                   

In [16]: pred = [0, 1, 0, 1, 0, 1, 0, 1, 0, 1]                                                                                                      

In [17]: fpr, tpr, thresholds = metrics.roc_curve(true, pred)                                                                                       

In [18]: metrics.auc(fpr, tpr)                                                                                                                      
Out[18]: 0.22222222222222221

However, if there are no true negatives (e.g. there is only one class), an error is thrown:

In [19]: true = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

In [20]: pred = [0, 1, 0, 1, 0, 1, 0, 1, 0, 1]                                                                                                      

In [21]: fpr, tpr, thresholds = metrics.roc_curve(true, pred)                                                                                       
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-21-35631f51a7c5> in <module>()
----> 1 fpr, tpr, thresholds = metrics.roc_curve(true, pred)

    132     # ROC only for binary classification
    133     if classes.shape[0] != 2:
--> 134         raise ValueError("ROC is defined for binary classification only")
    135 
    136     y_score = np.ravel(y_score)

ValueError: ROC is defined for binary classification only

Is this the correct behavior?

[jaquesgrobler]

Well with

In [11]: true = [1, 1, 1, 1, 1, 1, 0, 1, 1, 1]

In [12]: pred = [0, 1, 0, 1, 0, 1, 1, 1, 0, 1]

which has no true negatives either, there's no problem.
As the error message states, it's only for binary classification, so it makes sense to me that it would moan for just one class. Seems like correct and intended behavior in that context IMHO
Regards,J

[ogrisel]

I think we could raise an even more explicit error message when len(np.unique(true)) == 1: for instance:

raise ValueError("true should have exactly 2 distinct possible labels: got only one instead.")

[amueller]

On 10/22/2012 10:29 AM, Olivier Grisel wrote:

I think we could raise an even more explicit error message when
|len(np.unique(true)) == 1|: for instance:

|raise ValueError("true should have exactly 2 distinct possible labels: got only one instead.")
|

—
Reply to this email directly or view it on GitHub
#1257 (comment).

But shouldn't this also work in this case? It should be enough if either
y_true or y_pred
has two labels.

Btw, true=0.9 doesn't make any sense. I'm not sure what the code does there.
Have you switched y_true and y_pred?

[seth-brown]

My interpretation was that y_true was an array of the ground truth classifications. For example:

y_true = [1, 1, 0]

(the first two elements are of class 1, while the third element is of class 0)

I was not expecting an error to be thrown if all y_true values were of one class. Thanks for the clarification.

[amueller]

This is the correct interpretation. I still don't understand what the 0.9 does there ;)

That aside:
I think we should be able to handle the case, and I consider what you discovered a bug.

[weilinear]

If I have understood it right, actually the y_pred here is a y_score which means with a thresholds, y_score can be transformed into a y_pred. Then if we have a y_true all 1 or 0:

y_true all 1:
False positive rate will always be zero with each of the thresholds. Only true positive changes with thresholds. All point on the y-axis

y_true all 0:
True positive rate will always be zero with each of the thresholds. Only false positive changes with thresholds. All point on x-axis.

IMHO, no error actually as sometimes I just use it to see how y_true changes with thresholds :) Maybe a warning should be fine.

[GaelVaroquaux]

On Wed, Oct 24, 2012 at 01:14:16AM -0700, Wei LI wrote:

IMHO, no error actually as sometimes I just use it to see how y_true changes
with thresholds :) Maybe a warning should be fine.

+1 for warning

[weilinear]

I am preparing a PR for this and I find another problem here: should we let the user set which is the true classes or we just take the 1 as default. If yes, maybe a further check on y_true to be a binary label. As roc_curve([1,2,1],[0,1,2]) would not generate any warning. Currently the unique(y_true)[1] will be the default true class which I think is a little confusing.

[amueller]

Hm so the problem is that it is hard to do a consistent "unique".
I think we should use "LabelEncoder", training with "y_true" and predicting on "y_pred".
We should also provide a "classes" parameter, so that people can say which classes they expect (if one
is not present in y_true but in y_pred).

I'm not sure "classes" is the right name, but I'm sure you'll find it in the code somewhere.

[amueller]

how is this handled in other score functions?

[weilinear]

@amueller This differs from other score functions as it has a true class which cannot be directly inferred from y_true. The only generally accepted way to me is to make the 1 as true classes.

So I suggest the logic here is, we allow the users to input a true class, and using a one against all manner, all others to become negative classes. If not provided, then a binary label will be assumed, y_true contains other labels should issue an error.

Then roc_curve([1,2,1],[0,1,2]) will issue an error but roc_curve([1,2,1],[0,1,2],1) will not, it will transform label [1,2,1] to [1,0,1].

(y_pred here is just y_score, it is only our scores for the sample being positive. y_pred should be inferred with thresholds, so y_pred, y_score does not quite matter in this issue).

[amueller]

I would rather throw an error when more as two classes are present.

[weilinear]

@amueller maybe we can close this one?