Update the hyperparameter tuning methods

[SvenKlaassen]

Reworking the tuning procedures from doubleml to not only use basic grid search would be a good addition.

I think the current framework might as well be quite adaptable to be combined with Optuna.
A tune_optuna method could accept parameter dictionaries for the specific learners and set up an optuna study, see example.

Any comments or suggestions on hyperparameter tuning?

[OliverSchacht]

Good idea, I think tuning beyond grid search is an important issue for practitioners!

I think FLAML AutoML is also a promising option. Both would be viable options to integrate.

[SvenKlaassen]

Yes, Flaml would also be great.

[JanTeichertKluge]

Hi all,

I made an initial implementation on the following branch to try out Optuna: https://github.com/DoubleML/doubleml-for-py/tree/j-optuna
The argument optuna_settings=None is added to the tune method and passed to the _nuisance_tuning method and finally to the _dml_tune function from utils. If optuna_settings is not None, the new helper function _dml_tune_optuna is used instead of the previous tuning procedure using sklearns methods. The optuna tuning is done after sevceral checks and configurations
One can pass a parameter dict as optuna_settings, as @SvenKlaassen suggested, e.g.:

 optuna_settings_tpe = {
    "n_trials": 30,
    "sampler": optuna.samplers.TPESampler(seed=42),
    "show_progress_bar": False,
    "verbosity": optuna.logging.WARNING,
}

I know that the implementation isn't great at the moment, I see this first version more as a proof of concept (created with some help from AI).

You can also find a quick simulation in the following notebook: optuna_tuning_comparison.ipynb

At first glance, it definitely seems to be faster than sklearns tuning. Other advantages may arise in the case of more complex DGPs?

[SvenKlaassen]

Looks already very interesting.
I think that might be helpful but I would suggest that the optuna should not rely on the previous types of grids but allow for standard parameter distributions, i.e. trial.suggest_float('n_estimators', 10, 1000).

That might require a bit more general type of tuning function. And mainly the objective for each learner.

[SvenKlaassen]

One specific issue with tuning is to allow for tuning on folds, i.e. fold specific nuisance parameters.
Fixing #108 would be very simple if only one set of nuisance parameters per self learner would exist in the _params params property.

I would suggest to remove the possiblity to tune on folds completely and set parameters just per learner (even optionally allow for updated/different learners (´g1´ vs ´g0´).