Except method

[jvmncs]

It would be helpful to have an except method for the BayesianOptimization class. The idea would be to pass parameter values that you don't want to search. This would come in handy, for example, when tuning learning rates close to zero. I'm tuning three separate learning rates close to zero and it automatically searches each learning rate exactly equal to zero. I feel like this is buggy behavior, not sure if it comes from sklearn or bayes-opt. The model I'm trying to train is particularly tricky, and often the random initializations are degenerately better than the trained models. Thus, the optimization process exploits these degenerate cases thinking that they are optimal points.

It should be an easy fix, simply a matter of checking the right condition over an input list/dict. I'll try to submit a pull request and fix it myself if I get some time this weekend.

[jvmncs]

Turns out the print_step method is hard-coded to include up to five decimal points, and the points that the GP was probing were below 0.00001. The idea of excluding some points may still be worthwhile, so I won't close the issue just yet.

[jvmncs]

Looking into providing constraints for the scipy minimize function in acq_max from helpers.py. Should only be a one or two line fix with minimal impact on performance.

[jvmncs]

For exclusion of specific points, each point must be reformulated into an equality or inequality constraint. We want a non-equality constraint, i.e. x not equal to some y. This can be reformulated as sum(abs(x_i-y_i))>0, which can be approximated by sum(abs(x_i-y_i))-tol>=0 for some tol very close to zero. Numerical stability shouldn't be a problem for the right choice of tol. I'm on the fence about whether exclusion points should be passed to the __init__ method or passed to their own method the same way points are passed to the explore method (i.e. dict with values as lists). I'll implement one of these possibilities over the weekend.

[jvmncs]

Oy, I just realized the acq_max function uses L-BFGS-B. I'd have to change that to SLSQP. We could still use L-BFGS-B in cases where no exclusion points have been passed. @fmfn Thoughts?

[fmfn]

Bayesian Optimization (particularly with UCB acquisition function) is known to having a "thing" for edges, there are even a few papers in the literature of people trying to tackle this, however I didn't consider implementing it since I thought it was out of scope.

Before acq_max returns it clips the output, making sure it lies within the given bounds. However, clip operates on a closed interval, therefore the edges can be probed exactly.

I rather have the object giving bad, but predictable results (exact edge probing), than performing magic under the hood without the user's knowledge. So in this situation I think it would be better to let the user pass {"learning_rate": 0.0 + 1e-10}, than to try to figure out that learning_rate==0.0 leads to pathological behaviour.

Finally, one interesting idea would be to generalize the optimization manifold from a box as is currently the case (only params lower and upper bounds are defined), to more interesting topologies. Similarly the ability to put constraints on bounds (such as x > y), would be quite interesting. But I believe this would be challenging.

[jvmncs]

Awesome, closing this issue. I'll follow up with a new issue for implementing constraints. I think it will be easier than it seems.