ValueError when using Pipeline in GridSearchCV from version 1.3.0

[dokato]

I've got the following code:

import platform
import scipy
import sklearn
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import StratifiedKFold
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import IsolationForest
from sklearn.metrics import average_precision_score, make_scorer, roc_auc_score

import numpy as np
from copy import deepcopy

print(platform.python_version())
print(sklearn.__version__)

def unsupervised_score(y_true, y_pred, scorer):
    y_pred_copy = convert_unsupervised_labels(y_pred)
    return scorer(y_true, y_pred_copy)

def convert_unsupervised_labels(y):
    yc = deepcopy(y)
    yc[np.argwhere(yc == 1)] = 0
    yc[np.argwhere(yc == -1)] = 1
    return yc

# ---------------------

estimator = sklearn.pipeline.Pipeline([("estimator", None)])

grid = [
    dict(
        estimator=[IsolationForest(random_state=0)],
        estimator__n_estimators=range(50, 201),
        estimator__contamination=[0.001],
    ),
]

scoring_unsup = {
    "roc_auc": make_scorer(
        unsupervised_score,
        greater_is_better=True,
        scorer=roc_auc_score,
        needs_threshold=True,
    )
}
cv = StratifiedKFold(n_splits=2)

rs = GridSearchCV(estimator=estimator,
                  param_grid=grid,
                  cv=cv,
                  scoring=scoring_unsup,
                  refit="roc_auc",
                  error_score="raise", return_train_score=True,)

# Generate random data for binary classification
X, y = make_classification(
  n_samples=10,  # Number of samples
  n_features=3,  # Number of features
  n_informative=2,  # Number of informative features
  n_redundant=1,  # Number of redundant features
  n_clusters_per_class=1,  # Number of clusters per class
  random_state=42,
)

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

rs.fit(X_train, y_train)
print(rs.best_estimator_)

that for the latest version of sklearn 1.3.0 results in error:

ValueError: Pipeline should either be a classifier to be used with response_method=decision_function or the response_method should be 'predict'. Got a regressor with response_method=decision_function instead.

The code works fine in 1.2.2.

The error is not quite informative as it suggests adding response_method argument which is no longer used there(?).

Why this breaking change and how to fix it? I'm looking for an easy (can be hacky) fix so it doesn't change much in this pipeline.

[lesteve]

Thanks a lot for the reproducing script! I did a small edit to replace undefined y_copy by yc.

I don't get the error in the development version, #27565 may be the PR that fixed it (not 100% sure), can you try installing the development version to confirm if that fixes it for you?

[dokato]

Thank you @lesteve for a prompt reply. (and apologies for c&p typo...) Took me a bit to install dev on M1, but finally got it, tested and indeed works on 1.4.dev0, cheers!

[lesteve]

Thanks for your feedback! The next release (1.4) is likely to be released in January.

[wsperat]

@lesteve I have a similar problem when using permutation_importance with a custom classifier that implements both predict and predict_proba (but not decision_function). Do you think it's related to the same issue?

[lesteve]

Which version of scikit-learn are you using? Have you tried scikit-learn 1.4.0 to see if it gets rid of the issue?

[wsperat]

Apparently I had built my class incorrectly, and inherited from TransformerMixin instead of ClassifierMixin; that fixed the issue for me. Thanks!