PyCaret returns slightly different results than scikit-learn

[guillepl]

Hello. I am currently working with PyCaret 3.4.0, since 4.0 lacks some configuration parameters that are useful for my case.
I tried to replicate PyCaret results using scikit-learn.

This is my script, after running Pycaret's setup and obtaining the transformed data:

compare_models(include=['rf'],cross_validation=True)

scoring=['accuracy','precision','recall','f1','f1_macro','f1_weighted','f1_micro', 'roc_auc']
rfc = RandomForestClassifier(random_state=42, n_jobs=-1)
scores = cross_validate(rfc, xtrain_trans, ytrain_trans, scoring=scoring, cv=cv)

compare_models return these results:

but I get these results from sklearn:

• Accuracy=0.6948717948717947
• AUC=0.7411665257819103
• Recall=0.6908791208791208
• Precission=0.7005056185644422
• F1=0.6867142420587947
• F1_macro=0.6910345427878428
• F1_weighted=0.6911863570749788
• F1_micro=0.6948717948717947

Just to clarify, I am using the exactly same CV splitter on both PyCaret and sklearn.

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=42)

The setup used was:

setup(xtrain, 
            target = 'Group',
            session_id=42,
            test_data=xtest, #None default
            imputation_type=None, 
            remove_multicollinearity=True,
            multicollinearity_threshold=0.70,
            remove_outliers=True,
            transformation=True,
            transformation_method='quantile',
            normalize=True,
            feature_selection=False,
            fold_strategy=cv,
            use_gpu=True)

Any idea on what might be happening?

[guillepl]

Ok, so I think I found the answer, also I am not absolutely sure.

I think PyCaret creates a pipeline that is run on each split in the CV phase. On each split, train set is preprocessed with normalization and transformation, then test set is transformed. After that, the model is fitted and the prediction is made using the transformed test set.

If I turn off all preprocessing options in PyCaet's setup, the averaged CV results returned from compare_models and sklearn are exactly the same.

I coudln't find any confirmation on PyCaret guides, and since the GitHub changed recently due to the 4.0 update, it’s not so easy to explore the scripts anymore.

[JayeshSuryavanshi]

You have basically diagnosed it correctly, and yes, it is the preprocessing. The difference is where it gets fit.

PyCaret builds the transforms and the model as one pipeline and refits that pipeline inside every CV fold: on each split it fits the transforms on that fold's training rows only, applies them to the fold's validation rows, then fits and scores the model. That is the leak-free way to do it. In your sklearn version you passed xtrain_trans / ytrain_trans, which were already transformed once on the full training set before any split, so your quantile transform, normalizer, multicollinearity filter, and outlier removal all saw every fold's validation rows before the CV even started. The two are not running the same experiment, which is why the averages drift, and it is exactly why turning all preprocessing off lines them up.

To replicate PyCaret in sklearn, put the transforms and the model in a Pipeline and hand cross_validate the raw (untransformed) X, so everything refits per fold:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import QuantileTransformer, StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_validate, RepeatedStratifiedKFold

# match PyCaret's transformer settings and the order it applies them (transform, then normalize)
pipe = Pipeline([
    ("quantile", QuantileTransformer(output_distribution="normal")),
    ("scale", StandardScaler()),
    ("rf", RandomForestClassifier(random_state=42, n_jobs=-1)),
])

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=42)
scores = cross_validate(pipe, xtrain_raw, ytrain, scoring=scoring, cv=cv)

Two things can still keep them from matching exactly even after that:

• remove_outliers=True drops rows from each fold's training set, and an ordinary sklearn transformer cannot change the number of samples. PyCaret does this as a resampling step inside the pipeline, so to mirror it you would need an imblearn Pipeline with a custom outlier-removal sampler, or just leave it off while comparing. remove_multicollinearity is likewise a PyCaret-specific step you would have to reproduce as its own transformer.
• use_gpu=True. If PyCaret ran the RF on GPU it may have used cuML's RandomForestClassifier rather than sklearn's, and those are different implementations. Even with identical folds and identical data they will not produce bit-identical trees, so a small gap is expected. Set use_gpu=False if you want a clean apples-to-apples comparison.

So your own fix (all preprocessing off matches) is the confirmation that it is the per-fold preprocessing. Rebuild it as a Pipeline on raw data, keep outlier removal and GPU in mind, and you can get them to agree.