[BUG] Incompatible API between julearn and scikit-learn pipelines

[fraimondo]

Describe the bug
When using any Grouped CV scheme for hyperparameter tuning, the fitting of the final estimator fails because there is no groups parameter defined:

julearn/julearn/api.py

Line 211 in 1c57332

pipeline.fit(df_X_conf, y)

Adding the group parameter at this line fails because Julearn pipeline API is not exactly as in scikit_learn: https://github.com/scikit-learn/scikit-learn/blob/15a949460dbf19e5e196b8ef48f9712b72a3b3c3/sklearn/pipeline.py#L314

We are missing the **fit_params.

To Reproduce
Steps to reproduce the behavior:

1. Check the associated PR
2. Run:

from test_api import test_tune_hyperparam
test_tune_hyperparam()

3. Enjoy the stack trace

Expected behavior
A clear and concise description of what you expected to happen.

No error, hyperparameter tuning should work fine. Test should pass.

[samihamdan]

I see the problem, but before I can solve it we need to clarify the api as the test does not provide the needed information.
In the test you set up GroupedKFold for inner cv.
You do not set groups argument in run_cross_validation and we do not have a fit_params argument for run_cross_validation.
As far as I understand, in sklearn cross_validate you would set groups = groups for the outer cv
or for inner cv fit_params = {groups:groups} in cross_validate.

Once we decide where in run_cross_validation we want to set groups for inner cv, I should be able to fix the rest as well.

The question is now:
Where should groups for inner cv go to:

• search_params inside model_params as other arguments for searchers,
  even though this is not for the constructor of the searcher.
• new added fit_params for run_cross_valdiation
• we use groups of run_cross_validation for both inner and outer cv
  => which would not work when both are GroupedKFold
• Or some other option I did not yet consider.

[fraimondo]

You are right, forgot to add the groups parameter in the CV test. I'm updating the PR.

However, I don't think it's possible to do grouped stuff in the inner CV in sklearn. The "groups" parameter will depend on the outer CV split.

[fraimondo]

https://stackoverflow.com/questions/60996995/use-groupkfold-in-nested-cross-validation-using-sklearn

[samihamdan]

We checked and found that the function _fit_and_score which is used by cross_validate (
https://github.com/scikit-learn/scikit-learn/blob/2beed55847ee70d363bdbfe14ee4401438fba057/sklearn/model_selection/_validation.py#L576 ) uses _check_fit_params which subsets array params to only include the train indeces.
(https://github.com/scikit-learn/scikit-learn/blob/2beed55847ee70d363bdbfe14ee4401438fba057/sklearn/utils/validation.py#L1456)

Together with manually checking this shows that one can pass in groups as fit_params in cross_validate.
Therefore, we decided to always include groups if its not None into the fit_params
This allows users to use groups for both inner and outer cv, but not different groups for each.

[fraimondo]

Coming back to the issue. I think this is what triggered a big refactor.