Getting information about each trial as soon as it completes?

[hbredin]

It would be nice to have a programmatic way to get information about each trial as soon as it completes (at least in the sequential optimization case...).

Something like:

for trial in study.optimize_iter(objective, n_trials=10):
    # trial.params contains the set of parameters that were last tested
    # trial.value contains the corresponding objective value
    # trial.trial_id contains the trial ID
    pass

My main use case for this is to be able to run some kind of "callback function" every time a new best set of hyper-parameters is found.

As far as I understand, there is no easy way to do this -- except maybe by doing this directly in the objective function but this is not very convenient.

Does that make sense?

[g-votte]

Sounds cool. We Optuna dev team will discuss possible designs and implementations for the proposed feature.

[hbredin]

Great. Thanks. Two possible API designs that I can think of are:

1. making .optimize iterable thanks to a boolean flag:

for trial in study.optimize(objective, n_trials=10, iter=True):
    # trial.params contains the set of parameters that were last tested
    # trial.value contains the corresponding objective value
    # trial.trial_id contains the trial ID
    pass

2. making .optimize support an optional callback parameter

def callback(trial):
    # trial.params contains the set of parameters that were last tested
    # trial.value contains the corresponding objective value
    # trial.trial_id contains the trial ID
    pass

study.optimize(objective, n_trials=10, callback=callback)

I believe the former is more powerful and flexible than the latter.

[g-votte]

Thanks for your suggestion. I think option 2 is preferable from the perspective of implementation cost and compatibility with current Optuna API.

Out of curiosity, in your use case, what are you trying to do in the callback function?

[hbredin]

Here are a few things the callback function might do:

• every time a better set of hyper-parameters is found, run a validation experiment (possibly asynchronously)
• log trial details to tensorboard (e.g. using tensorboardX)
• display personalized progress report to the user (instead of optuna's builtin logging)

The last point might be tricky to do with option 2 (callback) but very easy with option 1 (iterator).

[sile]

I close this issue because this feature was implemented in #480.
Feel free to re-open this issue if you have further concerns.