Ability to tune the underlying bayes_opt params

[robert1826]

Some problems require tuning the underlying bayes_opt params such as the utility function being used or even the underlying gp params ... it seems that there is no way to change them using nevergrad

[jrapin]

Indeed, the current interface of on optimizer is not very flexible in this regard.
I am trying to push some sort of factory / family of optimizers which could be use for this though. It is mostly hidden for now but some algorithms, including the BO variants have one:

nevergrad/nevergrad/optimization/optimizerlib.py

Lines 1044 to 1067 in d5bda38

	class ParametrizedBO(base.ParametrizedFamily):
	"""Bayesian optimization
	initialization: str
	Initialization algorithms (None, "Hammersley", "random" or "LHS")
	middle_point: bool
	whether to sample the 0 point first
	"""
	no_parallelization = True
	_optimizer_class = _BO
	def __init__(self, *, initialization: Optional[str] = None, middle_point: bool = False) -> None:
	assert initialization is None or initialization in ["random", "Hammersley", "LHS"], f'Unknown init {initialization}'
	self.initialization = initialization
	self.middle_point = middle_point
	super().__init__()
	BO = ParametrizedBO().with_name("BO", register=True)
	RBO = ParametrizedBO(initialization="random").with_name("RBO", register=True)
	QRBO = ParametrizedBO(initialization="Hammersley").with_name("QRBO", register=True)
	MidQRBO = ParametrizedBO(initialization="Hammersley", middle_point=True).with_name("MidQRBO", register=True)
	LBO = ParametrizedBO(initialization="LHS").with_name("LBO", register=True)

In this class, we could add close to anything (though the more we add, the more likely there will be new issues so I try to be cautious about it). Is there anything in particular you would like to have access to?

[robert1826]

I would like to have access to the utility function args (kind, kappa and xi) ... and for the GP, I think the alpha parameter will suffice.

[robert1826]

another (maybe related) issue is the seed of the BayesianOptimization object ... as I noticed that it's random but I think it should be reproducible / controllable

[lazyoracle]

another (maybe related) issue is the seed of the BayesianOptimization object ... as I noticed that it's random but I think it should be reproducible / controllable

The results of BO can be repeatedly reproduced inside Nevergrad by setting the seed to numpy random as such:

import numpy as np
np.random.seed(any_seed)

Check out previously closed issue #4

However there are some reported issues with the inability to reproduce BO results across different systems, you might want to take a look at this:
Different result on different machines for same input and random seed #155

[robert1826]

@lazyoracle I think you are missing this :

nevergrad/nevergrad/optimization/optimizerlib.py

Line 994 in d5bda38

seed = np.random.randint(2**32, dtype=np.uint32)

[lazyoracle]

@robert1826 Won't setting a fixed seed in np.random() generate the same randint() to be used as the seed you mentioned above thus ensuring reproducibility?

[jrapin]

@robert1826, as @lazyoracle mentioned, this line is precisely what makes seeding before initialization of the optimizer work. They are unit tests that make sure the results are reproducible (though I did notice slight differences from a computer to another one, which seems consistent with the issue mentioned by @lazyoracle)

For the future, I'm still considering changing how seeding works in nevergrad though, providing a default random state that optimizers should pull from. But I'd rather avoid adding a seed argument if I can. That does not seem necessary most of the time.

[jrapin]

Concerning the parameters, I'll try to add them next week.

[jrapin]

Above is the diff to provide access to more parameters, @robert1826 would that work for you? Can you give more details to what was useful for you and why? This is just for my own culture, I am not yet very familiar with BO.

[robert1826]

This looks perfect to me and it's exactly what I asked for .... but for completeness please add another param for the 'init' budget .. instead of just setting it as the sqrt(budget) here:

nevergrad/nevergrad/optimization/optimizerlib.py

Line 1004 in b8fdfa7

init_budget = int(np.sqrt(self.budget)) - midpoint

[jrapin]

Ok, this is now changed as well. I am a bit worried that this will create a lot of border cases though, but nevermind for now. On the longer term though, this may mean that the interface will evolve. I see a couple of things that will probably change:

• the first 3 parameters (about the initialization) could(/should?) probably be changed to a unique parameter init_optimizer for instance.
• the gaussian cumulative density transform which is used to convert from [0, 1] to R saturates too fast in my experience, I doubt you can get any value beyond 14 or something like this. We may change it to arctan if this makes sense as well.
• the whole ParametrizedOptimizer thing may move inside nevergrad or change names when we properly "release" it.

In a nutshell, don't be surprise if you see some changes (or don't hesitate to ask for more changes, or propose some), this is still under development (sorry for the bother...)

[jrapin]

I'll merge now, but am unable to comprehensively test that all the parameters work as intended. Hopefully all should go well but let me know if you see anything weird ;)

[lazyoracle]

How well does this integrate with the family/factory of optimizers plan you were mentioning? Do you plan to have a ParametrizedOptimizer-like parameter for initialising all the optimizers?

[robert1826]

My logic is that the proposed changes are just a way to 'expose' the params of bayes_opt to the user of nevergrad ... so any border cases in this change is probably handled by the bayes_opt package ... anyway I'm an active user and I'll try to test and fix bugs if I find some.

[jrapin]

@robert1826 some of the initialization part is not strictly from bayes_opt so I won't be able to blame them if it fails :D
@lazyoracle for now there is no clear path, it's more of a tentative API. I will push it whenever this makes sense (it removed a looot of duplicated code one OnePlusOne for instance), there is issue #100 to track this process. Still, not all optimizers require parametrization (and not all users want to deal with parametrization) so both optimizers and optimizer family may live together if nobody complains too much...