[BUG]: return_train_score not working for tune_model

[ngupta23]

pycaret version checks

• I have checked that this issue has not already been reported here.

• I have confirmed this bug exists on the latest version of pycaret.

• I have confirmed this bug exists on the develop branch of pycaret (pip install -U git+https://github.com/pycaret/pycaret.git@develop).

Issue Description

return_train_score works for create_model but not for tune_model. Reproducible example can be found here:

https://gist.github.com/ngupta23/9bfb054f048bd16f2c2afbb2a496b29b

Reproducible Example

from pycaret.datasets import get_data
dataset = get_data('credit')

from pycaret.classification import ClassificationExperiment
exp = ClassificationExperiment()
exp.setup(data = dataset, target = 'default', session_id=123)

# Works ----
model = exp.create_model('dt', return_train_score=True)

# Does not work ----
tuned = exp.tune_model(model, return_train_score=True)

Expected Behavior

Tuning should return the train scores (similar to create_model)

• Also, please add some unit tests to test this behavior for various methods (create_model, tune_model etc.)

Actual Results

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   3360             try:
-> 3361                 return self._engine.get_loc(casted_key)
   3362             except KeyError as err:

13 frames
/usr/local/lib/python3.7/dist-packages/pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

/usr/local/lib/python3.7/dist-packages/pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'Mean'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
<ipython-input-5-8c68ef6820b4> in <module>()
      1 # Does not work ----
----> 2 tuned = exp.tune_model(model, return_train_score=True)

/usr/local/lib/python3.7/dist-packages/pycaret/classification/oop.py in tune_model(self, estimator, fold, round, n_iter, custom_grid, optimize, custom_scorer, search_library, search_algorithm, early_stopping, early_stopping_max_iters, choose_better, fit_kwargs, groups, return_tuner, verbose, tuner_verbose, return_train_score, **kwargs)
    947             tuner_verbose=tuner_verbose,
    948             return_train_score=return_train_score,
--> 949             **kwargs,
    950         )
    951 

/usr/local/lib/python3.7/dist-packages/pycaret/internal/pycaret_experiment/supervised_experiment.py in tune_model(self, estimator, fold, round, n_iter, custom_grid, optimize, custom_scorer, search_library, search_algorithm, early_stopping, early_stopping_max_iters, choose_better, fit_kwargs, groups, return_tuner, verbose, tuner_verbose, return_train_score, display, **kwargs)
   2613                 groups=groups,
   2614                 fit_kwargs=fit_kwargs,
-> 2615                 display=display,
   2616             )
   2617 

/usr/local/lib/python3.7/dist-packages/pycaret/internal/pycaret_experiment/supervised_experiment.py in _choose_better(self, models_and_results, compare_dimension, fold, fit_kwargs, groups, display)
    175         for model, result in models_and_results:
    176             if result is not None and is_fitted(model):
--> 177                 result = result.loc["Mean"][compare_dimension]
    178             else:
    179                 self.logger.info(

/usr/local/lib/python3.7/dist-packages/pandas/core/indexing.py in __getitem__(self, key)
    929 
    930             maybe_callable = com.apply_if_callable(key, self.obj)
--> 931             return self._getitem_axis(maybe_callable, axis=axis)
    932 
    933     def _is_scalar_access(self, key: tuple):

/usr/local/lib/python3.7/dist-packages/pandas/core/indexing.py in _getitem_axis(self, key, axis)
   1162         # fall thru to straight lookup
   1163         self._validate_key(key, axis)
-> 1164         return self._get_label(key, axis=axis)
   1165 
   1166     def _get_slice_axis(self, slice_obj: slice, axis: int):

/usr/local/lib/python3.7/dist-packages/pandas/core/indexing.py in _get_label(self, label, axis)
   1111     def _get_label(self, label, axis: int):
   1112         # GH#5667 this will fail if the label is not present in the axis.
-> 1113         return self.obj.xs(label, axis=axis)
   1114 
   1115     def _handle_lowerdim_multi_index_axis0(self, tup: tuple):

/usr/local/lib/python3.7/dist-packages/pandas/core/generic.py in xs(self, key, axis, level, drop_level)
   3769             try:
   3770                 loc, new_index = index._get_loc_level(
-> 3771                     key, level=0, drop_level=drop_level
   3772                 )
   3773             except TypeError as e:

/usr/local/lib/python3.7/dist-packages/pandas/core/indexes/multi.py in _get_loc_level(self, key, level, drop_level)
   3110                 return indexer, maybe_mi_droplevels(indexer, ilevels, drop_level)
   3111         else:
-> 3112             indexer = self._get_level_indexer(key, level=level)
   3113             return indexer, maybe_mi_droplevels(indexer, [level], drop_level)
   3114 

/usr/local/lib/python3.7/dist-packages/pandas/core/indexes/multi.py in _get_level_indexer(self, key, level, indexer)
   3202         else:
   3203 
-> 3204             idx = self._get_loc_single_level_index(level_index, key)
   3205 
   3206             if level > 0 or self._lexsort_depth == 0:

/usr/local/lib/python3.7/dist-packages/pandas/core/indexes/multi.py in _get_loc_single_level_index(self, level_index, key)
   2853             return -1
   2854         else:
-> 2855             return level_index.get_loc(key)
   2856 
   2857     def get_loc(self, key, method=None):

/usr/local/lib/python3.7/dist-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   3361                 return self._engine.get_loc(casted_key)
   3362             except KeyError as err:
-> 3363                 raise KeyError(key) from err
   3364 
   3365         if is_scalar(key) and isna(key) and not self.hasnans:

KeyError: 'Mean'

Installed Versions

pip install git+https://github.com/pycaret/pycaret.git@develop

System: python: 3.7.13 (default, Apr 24 2022, 01:04:09) [GCC 7.5.0] executable: /usr/bin/python3 machine: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic

PyCaret required dependencies:
pip: 21.1.3
setuptools: 57.4.0
pycaret: 3.0.0
ipython: Not installed
ipywidgets: 7.7.0
numpy: 1.21.6
pandas: 1.3.5
jinja2: 2.11.3
scipy: 1.7.3
joblib: 1.1.0
sklearn: 1.0.2
pyod: Installed but version unavailable
imblearn: 0.8.1
category_encoders: 2.4.1
lightgbm: 3.3.2
numba: 0.55.1
requests: 2.27.1
matplotlib: 3.5.2
scikitplot: 0.3.7
yellowbrick: 1.4
plotly: 5.5.0
kaleido: 0.2.1
statsmodels: 0.13.2
sktime: 0.11.3
tbats: Installed but version unavailable
pmdarima: 1.8.5

PyCaret optional dependencies:
shap: Not installed
interpret: Not installed
umap: Not installed
pandas_profiling: Installed but version unavailable
explainerdashboard: Not installed
autoviz: Not installed
fairlearn: Not installed
xgboost: 0.90
catboost: Not installed
kmodes: Not installed
mlxtend: 0.14.0
tune_sklearn: Not installed
ray: Not installed
hyperopt: 0.2
optuna: Not installed
skopt: Not installed
mlflow: Not installed
gradio: Not installed
fastapi: Not installed
uvicorn: Not installed
m2cgen: Not installed
evidently: Not installed
nltk: 3.2.5
pyLDAvis: Not installed
gensim: 3.6.0
spacy: 2.2.4
wordcloud: 1.5.0
textblob: 0.15.3
psutil: 5.4.8
fugue: Not installed
streamlit: Not installed
prophet: Not installed

[Yard1]

thanks, will take a look. seems to be a simple oversight!