Add area under the Precision-Recall curve (AUPRC): a useful metric for class imbalance.

[gm039]

The area under the Precision-Recall curve (AUPRC) is very useful metric for class imbalance. Here is the source (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html).

I tried using the add_metric feature in Pycaret 2.2 as below.

from sklearn.metrics import average_precision_score
add_metric('AUPRC_ID','AUC_PRC',average_precision_score, greater_is_better = True)

But the scores are different from the score obtained using the evaluate_model(tuned_model_best) precision-recall curve (See the snapshot below).

Additionally, the AUC_PRC score seems to takes Target and Predicted Label as input argument instead of Target and Predicted Score.

[gm039]

@pycaret What is the default input argument to the metric defined through the add_metric feature in Pycaret 2.2?

[pycaret]

@gm039 You can actually access the metrics PyCaret is using by calling the get_metrics function.

[gm039]

@pycaret Thanks! Below are the metrics used.

The input argument to the metric should be Target and Predicted Score but its taking Target and Predicted Label as verified in the previous snapshot.

Unfortunately, both the average precision calculated is different from the score obtained using the evaluate_model(tuned_model_best) precision-recall curve (See the previous snapshot's PR curve).

[pycaret]

@gm039 You can pass the target = 'pred_proba' inside the add_metric call and it will work just fine :) See below:

from pycaret.datasets import get_data
data = get_data('juice')

from pycaret.classification import *
s = setup(data, target = 'Purchase', session_id = 123, silent = True)

from sklearn.metrics import average_precision_score
add_metric('apc', 'APC', average_precision_score, target = 'pred_proba')

lr = create_model('lr')

predict_model(lr);

plot_model(lr, plot = 'pr')

Hope this helps?

[pycaret]

@gm039 Please close the issue, if this answers the question.

[gm039]

@pycaret Thankyou so much. It worked.