Optimize AUC for binary classification

[pplonski]

Right now, all metrics as minimized. AUC should be maximized.

Please add a check in AutoML constructor for correct metric name.

[shahules786]

@pplonski Can you elaborate this a little more?

[pplonski]

We minimize logloss for binary classification. There is no option to optimize different metrics in AutoML. This issue is to add an option to optimize AUC as eval_metric. During the implementation of this issue, it will be worth to check what other metrics can be optimized for classification (F1, Acc, MCC) and for regression.

It is worth to check which metrics can be optimized directly by algorithms. I would avoid callbacks in algorithms because they slow down the training 20x (yes, 20x, even empty list of callbacks).

[shahules786]

Cool @pplonski , will look at it.

[pplonski]

This ticket is adding NEGATIVE AUC!

There is a problem with Learning curves plots. They have wrong y-axis label.

[alberto-jj]

Dear Piotr, many thanks for all the work done on this fantastic package.
I was wondering if is it possible to optimize based on balanced accuracy in classification tasks.
It might very helpful for unbalanced classes.

[pplonski]

Hi @alberto-jj,

You should be able to set custom eval metric. It will not be used directly for model training, but it will be used for early stopping and model selection, and ensemble building - it should help a lot.

Please take a look at this comment for implementation details #390 (comment)

[puneeshkhanna]

@pplonski - Quick q is log loss minimized based upon class imbalance ?

[pplonski]

What do you mean?

[puneeshkhanna]

Is it weighted log loss ?

[pplonski]

You can pass sample_weight and then it will be weighted loss.