CalibratedClassifierCV doesn't work well with CatBoostClassifier . Gives wrong output.

[zoj613]

Description

Problem:
When trying to calibrate the class probability estimates with scikit-learn's CalibratedClassifierCV, all I get are 1's for the negative target and 0's for the positive target in a binary classification problem. If I use CatBoostClassifier indipendently I get normal looking probabilities. This leads me to believe that this Classifier is not compatible with the calibration technique. Is there a way I can go about fixing this issue?

Steps/Code to Reproduce

from catboost import CatBoostClassifier
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import make_classification

X, y = make_classification(100, 10)

cat = CatBoostClassifier(verbose=0)
calib = CalibratedClassifierCV(base_estimator=cat, method='sigmoid', cv=2)
cat.fit(X, y)
calib.fit(X, y)
print(cat.predict_proba(X))
print(calib.predict_proba(X))

Expected Results

The result is supposed to be calibrated probabilities but instead I get all 1's for the first column and all zeros for the second one. This is clearly the wrong output.

Actual Results

See above

Versions

System:
    python: 3.7.3 (default, Jul  8 2019, 11:40:34)  [GCC 6.5.0 20181026]
executable: ~/.pyenv/versions/3.7.3/envs/absa-py37/bin/python
   machine: Linux-5.0.0-27-generic-x86_64-with-debian-buster-sid

Python deps:
       pip: 19.0.3
setuptools: 40.8.0
   sklearn: 0.21.3
     numpy: 1.17.0
     scipy: 1.3.1
    Cython: None
    pandas: 0.25.0

[amueller]

sigmoid calibration doesn't usually work well for tree-based models (as mentioned in the documentation). Do you observe the same with isotonic regression?

[zoj613]

sigmoid calibration doesn't usually work well for tree-based models (as mentioned in the documentation). Do you observe the same with isotonic regression?

I get the same behavior regardless of the parameter I pick. All other tree-based classifiers I have used work as expected. This is the only one producing this bug.

[amueller]

Feel free to reach out to the CatBoost authors as well, not sure if any of us have time to look into this.

[zoj613]

Feel free to reach out to the CatBoost authors as well, not sure if any of us have time to look into this.

I did several days ago, no one replied to the issue there. Hence why I decided to also post it here hoping that someone might have a workaround.
catboost/catboost#999

[adamlin859]

It think this issue is cause by the CatBoostClassifier classifier have a empty classes_ attribute after fit. You can see this by the code below catboost/catboost#984:

from catboost import CatBoostClassifier
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import make_classification

X, y = make_classification(100, 10)

cat = CatBoostClassifier(verbose=0)
calib = CalibratedClassifierCV(base_estimator=cat, method='sigmoid', cv=2)
cat.fit(X, y)
calib.fit(X, y)
print(cat.predict_proba(X))
print(calib.predict_proba(X))
print(cat.classes_)

This is causing idx_pos_class to be empty so no classifier is actually being fit.

scikit-learn/sklearn/calibration.py

Lines 313 to 316 in d6d1d63

	idx_pos_class = self.label_encoder_.\
	transform(self.base_estimator.classes_)
	return df, idx_pos_class

[amueller]

If classes_ is indeed empty, CatBoostClassifier doesn't satisfy our api and doesn't pass check_estimator so I think we can close?

[thomasjpfan]

Closing this issue as CatBoostClassifier does not satisfy our api.

(As a quick fix, one can subclass CatBoostClassifier and define classes_.)

[MatanBendak]

Is there an update about this topic?
Does sklearn Isotonic regression support CatBoostClassifier?
If not- Is there an easy fix for this?