New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature request: Optimizing hyperparameters under unknown constraints #884
Comments
@mlindauer we spoke about this briefly at the AutoML fall school, and you said it might be straightforward to add. If so, I could try to contribute this feature, but it would be helpful if someone familiar with the codebase could point me in the right direction. |
Hi,
Does that help you? Best, PS: I point you directly to the new code base since it would be wasted effort to do it in the old code base now. |
Thanks @mlindauer for the pointers! These were very helpful. I was able to put together an example that seems to work well on some toy target functions with unknown constraints: https://gist.github.com/sethaxen/331402e156537a933121133fe9965573 . Major thanks to @KEggensperger who helped me navigate SMAC. I'm planning to deploy this shortly on an expensive model, so if someone has ideas for improvement, I'd love to hear them. Also, if you think this would be a useful example to include in the docs, I'm happy to contribute it. |
Thanks for posting your outcome, @sethaxen . We will have a look at it during our meeting next week! |
@sethaxen Thanks for the updates! It looks quite promising to me and we would like to integrate this as an example for our model. We would appreciate it if you could provide a PR for us (ideally for development 2.0 branch) (you can also come to us if you need any support with the codebase). |
Sure, I'd be happy to! This would mean adding a new section to https://github.com/automl/SMAC3/tree/development/examples, right?
I'm new to AutoML, so I I can't say for certain. I came across examples where a classifier was incorporated into the acquisition function in more complicated ways (e.g. https://arxiv.org/abs/1004.4027), and the 2 papers I read that took the approach used here both used EI. Naively, it seems intuitive that this could work with other acquisition functions, so I could generalize |
Thanks! Exactly, the interface is overall the same, so it should not be so hard to transform to the development branch
I think this might only work for improvement-based acquisition functions (EI and PI). If you have an LCB as an acquisition function, then its acquisition function value might become negative, multiplying this value with |
Description
The ability to optimize along with a univariate hyperparameter an uknown/hidden constraint for the parameter, where invalid parameter values are identified by failed function evaluations.
This may be related to #403.
Specific application
For the simplest case I have, we can assume:
An initial idea
The KI Campus MOOC on AutoML mentioned that the Expected Constrained Improvement acquisition function, given in Eq.4 of https://engineering.ucsc.edu/sites/default/files/technical-reports/UCSC-SOE-10-10.pdf as
$$x' = \arg\max_{x \in \mathcal{X}} \mathbb{E}[I(x)] h(x)$$ $I$ is an improvement statistic, and $h(x)$ is the predicted probability that the parameter $x$ is valid, for which they use a random forest classifier.
is one approach that could work for something like this, where
I suppose a more general approach would allow a user to provide
The text was updated successfully, but these errors were encountered: