gp_minimize and company can not take points as dictionaries

[pavelkomarov]

I had an issue a few days ago (#581) where I wondered whether I could use BaseEstimator's set_params-like syntax when defining spaces and points with BayesSearchCV. That is paramName or subObject__subObjectParamName as keys to a dictionary-defined space. It turns out you can, so hooray.

However, I realized I can not actually use BayesSearchCV because my data is too large to fit in memory, so there is no way to call .fit(data). As a workaround, I have defined my own objective function which handles batch-reads and does its own cross validation and returns a score. But now I am finding out that the X_minimize functions only take lists as their dimensions parameter, which is wildly inconvenient because it means I can't just call estimator.set_params(**point) inside my objective function, and all the spaces I defined based on dictionaries can't work.

I suggest updating all X_minimize functions to support points and spaces defined as dictionaries rather than requiring lists. The list-solution seems to me an outdated one.

[pavelkomarov]

I did manage to find a hack to work around this issue, in case anyone else cares (and so I can't forget before I have a chance to implement it):

1. Use the Run object I defined to manage some of my configuration and batching and such to also store, instead of the dictionary-defined space, the alphabetically-sorted keys and the result of skopt's utils.dimensions_aslist(space) separately.
2. Pass the listified values-only-space result from step 1 as the space to X_minimize with Run.objectiveFn as the function to minimize.
3. In Run's objectiveFn use the saved, alphabetized keys and the point passed by base_minimize on line 225 or 248 to reconstruct the dictionary, which is possible because dimensions are sorted by alphabetized key, and then use this dictionary-defined space to call estimator.set_params(**dict).

Phew.

[betatim]

Does

scikit-optimize/skopt/utils.py

Line 381 in 6deff29

def point_asdict(search_space, point_as_list):

not work for you?

We decided to follow the interface of scipy.optimize which does not require objective functions to have named arguments. Maybe one way to make it more convenient is to provide a wrapper for objective functions that do support named arguments, something like:

def objective(x=0, y=3., z=0.3):
  pass

def with_named_args(func, space):
  def f(*args):
    return func(point_asdict(*args, space))
  return f

gp_minimize(with_named_args(objective, space), space,...)

[pavelkomarov]

point_asdict should do what I want, I suppose. I was aware of it but didn't connect the dots. So so long as I keep the original search space in my Run, I should be able to reconstruct the dictionary form without resorting to anything hackier than native skopt code.

I'm not saying you should require X_minimize to take the dictionary form of the space, but it would be nice if it were supported so I didn't have to worry about taking all the values out of the space to pass as a parameter and then re-joining points from that space with keys later.

[betatim]

I am not sure what you mean with "dictionary form of the space". I thought what you wanted to do is have *_minimize call the objective function with named arguments (like objective(a=2, b=3.1)) instead of the current convention (objective(2, 3.1)).

What interface do you propose for switching the calling convention of the *_minimize functions? My best idea is what I posted here.

[pavelkomarov]

The dictionary form of the space is as defined here as param_grid and here as param_distributions (minus the actual distributions part).

Right now the objective just gets some point defined with a list of values. I would rather it recieve that point as a dictionary of parameter names mapping to values, because the keys are useful and important. Sure, you can work around not having them as you suggested, but it's annoying and unnecessary. All you have to do is extend base_minimizer to treat dictionaries as well as lists, and then you can keep the calling convention/interface the same.

I have implemented a version that can take dictionaries, but I wasn't worrying about also handling list-form spaces at the time, so it's not backwards compatible and all that. But when I find the time I'll have to combine them and submit a pull request so you can see it.

[betatim]

Ok, so if I understand what you want to do: you want to specify the search space differently not change how the objective function is called? This is what http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html is about.

Right now the objective just gets some point defined with a list of values. I would rather it recieve that point as a dictionary of parameter names mapping to values

If you do want to change how the objective function is called then we need a way for base_minimize to know whether the particular objective it is calling expects individual arguments or a dictionary.

[betatim]

The example I gave in #583 (comment) is actually wrong, but we could still build a adaptor similar to what I sketched that passes a single argument to the objective which looks like dict(lr=0.01, n_trees=100, foo=2)

[pavelkomarov]

Right, the objective still just gets the one parameter, but that parameter could be a dictionary or a list, depending on what the user passes as dimensions to *_minimize(). Really there are just a few guts in base_minimize that need to be changed, some conditionals that treat dimensions differently if it's a dictionary, whereas right now it errors if the space is not a list (line 179)

[betatim]

If you make a PR we can discuss a concrete proposal. I'd be -0 on adding a new way of specifying a search space that relies on inferring what the user meant. We support naming dimensions so we can generate a dictionary for calling the objective that way. The bit to work out is a user friendly way to then decide how to call the objective.

scikit-optimize/skopt/optimizer/base.py

Lines 179 to 180 in 6deff29

	if not isinstance(x0, list):
	raise ValueError("`x0` should be a list, but got %s" % type(x0))

is there because internally we use an instance of skopt.Space which requires a list of Dimension objects.

[betatim]

#590 is a proposal for allowing objective functions to receive named arguments.

[pavelkomarov]

I followed #590 through to #592 and added a comment at the bottom stating why I do not believe that PR solves this issue.

[betatim]

Let's close this for now and keep #514 which is about the same topic. This issue has a lot of back-and-forth in it till we worked out what we were both talking about, #514 is cleaner -> easier to work on.