[ENH] callbacks for `BaseOptimizer` or `BaseExperiment`

[fkiraly]

It would be nice to have callbacks for BaseOptimizer or BaseExperiment.

One common use case is logging the sequence of parameters and parameter values obtained throughout the optimization.

My current thinking is, the best place to inject callbacks is the boilerplate layer of evaluate in BaseExperiment.

When called from within BaseOptimizer, it logs to a singleton or multiton attached to the optimizer instance, or to a logger attached to the optimizer.

How are callbacks passed? I can see a few options:

• through set_config with a callback or callback_pre / callback_post field
• as __init__ arguments, though that would require that all experiments and optimizers have to take it (which might be invasive)

[fkiraly]

Thoughts, @SimonBlanke?

[SimonBlanke]

I agree, a callback parameter should be readded to Hyperactive. I would see the parameter in the experiment class. Maybe by a separate method?

class BaseExperiment(BaseObject):
...

def callbacks(pre: list = None, post: list = None):
  if not pre:
    pre = []
...

[fkiraly]

I would suggest we map out explicitly where callbacks could be added, and what their form should be. Is it prior and post individual evaluate calls?

[SimonBlanke]

I would suggest we map out explicitly where callbacks could be added, and what their form should be.

We could add it to the evaluate method in the experiment class.

Is it prior and post individual evaluate calls?

I am not sure I understand the question. We can have both, right? pre-eval and post-eval callbacks.

[fkiraly]

I am not sure I understand the question. We can have both, right? pre-eval and post-eval callbacks.

Yes, exactly what I meant - where callbacks would be useful.