[BUG]: Custom metrics all report 0.0000 for classification in Pycaret 3.3.1

[JeffCanadaUser]

pycaret version checks

• I have checked that this issue has not already been reported here.

• I have confirmed this bug exists on the latest version of pycaret.

• I have confirmed this bug exists on the master branch of pycaret (pip install -U git+https://github.com/pycaret/pycaret.git@master).

Issue Description

The AUC 0.0000 metric was just fixed in Pycaret 3.3.1 but custom metrics still seems to be broken. This bug is present in 3.3.0 and 3.3.1 but working properly in 3.2.0. Tested using Python 3.9.18.

Using the example for custom metrics from the Binary Classification tutorial ( https://github.com/pycaret/pycaret/blob/master/tutorials/Tutorial%20-%20Binary%20Classification.ipynb ) the metric results from any binary classification is always 0.0000

This is the case for create_model() and compare_models()

and

Reproducible Example

import pycaret
pycaret.__version__
# loading sample dataset from pycaret dataset module
from pycaret.datasets import get_data
data = get_data('diabetes')
# import pycaret classification and init setup
from pycaret.classification import *
s = setup(data, target = 'Class variable', session_id = 123)

# create a custom function
import numpy as np

def custom_metric(y, y_pred):
    tp = np.where((y_pred==1) & (y==1), (100), 0)
    fp = np.where((y_pred==1) & (y==0), -5, 0)
    return np.sum([tp,fp])

# add metric to PyCaret
add_metric('custom_metric', 'Custom Metric', custom_metric)

rf = create_model('rf')

compare_models()

Expected Behavior

Expected behaviour with works fine in Pycaret 3.2.0 is:

Actual Results

Actual results are all 0.00000 for each "Custom Metric" entry in the table

Installed Versions

System: python: 3.9.18 | packaged by conda-forge | (main, Dec 23 2023, 16:33:10) [GCC 12.3.0] executable: /opt/conda/envs/data-sci/bin/python machine: Linux-5.15.0-101-generic-x86_64-with-glibc2.35

PyCaret required dependencies:
pip: 24.0
setuptools: 69.1.1
pycaret: 3.3.0
IPython: 8.12.3
ipywidgets: 8.1.2
tqdm: 4.66.2
numpy: 1.25.2
pandas: 2.1.4
jinja2: 3.1.3
scipy: 1.10.1
joblib: 1.3.2
sklearn: 1.4.1.post1
pyod: 1.1.3
imblearn: 0.12.0
category_encoders: 2.6.3
lightgbm: 4.3.0
numba: 0.58.1
requests: 2.31.0
matplotlib: 3.7.5
scikitplot: 0.3.7
yellowbrick: 1.5
plotly: 5.19.0
plotly-resampler: Not installed
kaleido: 0.2.1
schemdraw: 0.15
statsmodels: 0.14.1
sktime: 0.26.1
tbats: 1.1.3
pmdarima: 2.0.4
psutil: 5.9.8
markupsafe: 2.1.5
pickle5: Not installed
cloudpickle: 3.0.0
deprecation: 2.1.0
xxhash: 3.4.1
wurlitzer: 3.0.3

PyCaret optional dependencies:
shap: 0.44.1
interpret: Not installed
umap: Not installed
ydata_profiling: 4.6.5
explainerdashboard: Not installed
autoviz: Not installed
fairlearn: Not installed
deepchecks: Not installed
xgboost: 2.0.3
catboost: 1.2.3
kmodes: Not installed
mlxtend: Not installed
statsforecast: 1.7.3
tune_sklearn: Not installed
ray: Not installed
hyperopt: Not installed
optuna: Not installed
skopt: Not installed
mlflow: Not installed
gradio: Not installed
fastapi: Not installed
uvicorn: Not installed
m2cgen: Not installed
evidently: Not installed
fugue: 0.8.7
streamlit: Not installed
prophet: 1.1.5

[celestinoxp]

ping @moezali1 @Yard1

[kuanhan]

@moezali1 @Yard1

[CMobley7]

This should work for binary classification. I haven't figured out multiclass classification yet.

def custom_metric(y, y_pred, **kwargs):
    tp = np.where((y_pred==1) & (y==1), (100), 0)
    fp = np.where((y_pred==1) & (y==0), -5, 0)
    return np.sum([tp,fp])

[cantones1]

@CMobley7 this did not work for me when I tried it in the tutorial, even after updating my pycaret from github with pip install git+https://github.com/pycaret/pycaret.git@master --upgrade . Can you give some guidance on how did you make this work ?
Thanks!

[kuanhan]

@CMobley7 this did not work for me when I tried it in the tutorial, even after updating my pycaret from github with pip install git+https://github.com/pycaret/pycaret.git@master --upgrade . Can you give some guidance on how did you make this work ? Thanks!

Sadly, it didn't work for me either.

[cantones1]

ooops..., @kuanhan and @CMobley7 , I think the issue is that there was no "import numpy as np" anywhere in my code

So this actually did work for me:

import numpy as np

def custom_metric(y, y_pred, **kwargs):
    tp = np.where((y_pred==1) & (y==1), (100), 0)
    fp = np.where((y_pred==1) & (y==0), -5, 0)
    return np.sum([tp,fp])

add_metric('custom_metric', 'Custom Metric', custom_metric)