-
Notifications
You must be signed in to change notification settings - Fork 65
Description
I noticed an issue with the current API design - the sign (pun intended).
Namely, experiments can be minimization or maximization, e.g., sklearn experiments with the accuracy score, vs the MSE; or more classical optimization.
Currently, the API implicitly assumes minimization everywhere. But this becomes confusing ot the user as the SklearnCvExperiment expects metrics, so it raises the expectation that it handles the sign.
It is unclear where to handle this and is left to the user - there are two natural places of which one or both could be used:
- the experiment - a tag
lower_is_better: boolordirection: {"max", "min", "neither"}- or an__init__arg - the optimizer, which could get an
__init__arg, or an arg torun
It seems more natural to have the sign "minmize" or "maximize" as part of the optimization problem, hence the experiment.
My favourite solution is the tag, from a user perspective, although I wonder what this would imply about the optimizer. I am confident that we could assume minimization and let the boilerplate handle the sign, although this has to be thought through.
FYI @SimonBlanke