Demonstration of using fit along with hyperparameter optimization

[dionman]

Is there a notebook demonstrating how to use the optuna hyperparameter optimization capabilities of the models as part of fitting?

[bcebere]

Hello @dionman

Thank you for your question. It would be an interesting tutorial.

We will try a notebook this week with an Optuna example.

[tztsai]

Hello, is there anyone working on this issue? If not, I will work on it.

[bcebere]

Hello @tztsai

Absolutely, thank you for working on this.
Let us know if you need help.

Just a few guidelines:

• For a dataset and a task, you need to define an objective in Optuna.
• To that end, you need to "translate" the hyperparam_space from a plugin to the optuna Trial suggestions.
• Finally, given a set of args, you construct and evaluate the plugin.
• For the evaluation, a good metric could be the detection.mlp metric, which discriminates real data from synthetic data. In other words, you ask Optuna to give you the hyperparams which create the closest synthetic data to the real data. Of course, many other metrics can be relevant here, it is up to you.

[tztsai]

Hello @bcebere

I have finished the tutorial. In order to let Optuna sample from the hyperparam_space, I added a script optuna_sample.py under utils/. Please check them in the new pull request.

[dionman]

Does the current implementation of the plugins support HPO search algorithms that allow early pruning of unpromising trials, e.g. optuna's SuccessiveHalvingPruner?

[tztsai]

I don't think so. To support early pruning, a plugin must support arbitrary callbacks in each iteration of training. A custom callback then can be provided that reports intermediate performance scores and raises optuna.TrialPruned() so as to prune this trial. However, the DDPM plugin does support a list of callbacks during fitting and I also implemented the Callback class in "synthcity/utils/callbacks.py". If needed, I can implement a PruneOptunaTrial callback class for early pruning.

[robsdavis]

The tutorial created by @tztsai is available here, which closes this issue.

[tztsai]

Hi, sorry this does not support minibatching. 发件人: DM 发送时间: 2023年6月8日 14:27 收件人: vanderschaarlab/synthcity 抄送: Tianzhang Cai; Mention 主题: Re: [vanderschaarlab/synthcity] Demonstration of using fit along withhyperparameter optimization (Issue #166) does the computation of the validation score in the tutorial support minibatching? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[tztsai]

Hi, sorry the notebook does not include optuna timeout. 发件人: DM 发送时间: 2023年6月9日 19:00 收件人: vanderschaarlab/synthcity 抄送: Tianzhang Cai; Mention 主题: Re: [vanderschaarlab/synthcity] Demonstration of using fit along withhyperparameter optimization (Issue #166) is optuna timeout working for this notebook? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>