Zero-One Loss in Classification

[KarelZe]

Thanks for this awesome library. 💯

In #7 you discussed the use of alternative loss functions in classification.

I'm working on a use case, where I perform classification with different classifiers, but I mainly care about the accuracy of the prediction, rather than the predicted probabilities, as some classifiers yield hard probabilities only. As such, I wanted to swap the cross-entropy loss for the zero-one loss.

I extended the utils.py (see here.) and added the new loss function:

class ZeroOneLoss:
    '''zero-one loss that expects probabilities.'''
    def __init__(self, reduction='mean'):
        assert reduction in ('none', 'mean')
        self.reduction = reduction

    def __call__(self, pred, target):

        # Add a dimension to prediction probabilities if necessary.
        if pred.ndim == 1:
            pred = pred[:, np.newaxis]
        if pred.shape[1] == 1:
            pred = np.append(1 - pred, pred, axis=1)

        if target.ndim == 1:
        # Class labels.
            loss = (np.argmax(pred, axis=1) != target).astype(float)
        elif target.ndim == 2:
        # Probabilistic labels.
            loss = (np.argmax(pred, axis=1) != np.argmax(target, axis=1)).astype(float)
        else:
            raise ValueError('incorrect labels shape for zero-one loss')

        if self.reduction == 'mean':
            return np.mean(loss)
        return loss

and call it like this:

imputer = sage.MarginalImputer(model, train)
estimator = sage.KernelEstimator(imputer, 'zero one')

I was wondering, if there is more to consider, when using alternative loss functions, in particular zero-one-loss, with SAGE?
Would you also be interested in a PR?

[iancovert]

Hi, thanks for adding this! It looks reasonable to me and I can't think of anything else you would need to consider when implementing it. The existing modules should make it easy to swap in a new loss function like this.

Submitting a PR would be great, if you're willing to. Would you also consider adding a notebook, or an example with this loss function to one of the existing notebooks? Just so we can verify that it yields reasonable results.

[KarelZe]

@iancovert Thanks for your feedback. 🎉

I'd love to contribute this feature. I'll also prepare a notebook and maybe some tests. It will take me some time, as I'm working on my thesis.

[iancovert]

That sounds great! And no rush, I'm also working on my thesis so I understand :)

[iancovert]

Sorry it took me a while, but I just merged your PR. I checked out the code and it all looks reasonable, thanks for the contribution and for using the package!