Limitation in number of Hyperparameters with Multi-Fidelity

[Mshz2]

question

Description

Hi, I would like to use Multi-Fidelity option of SMAC to find optimum arrangement defined by about 2752 Hyperparameters within ConfigurationSpace.

I was wondering if there is any performance limitation in number of hyperparameters for Optimization by using your implemetation?
these 2752 Hyperparameters can be all categorial or all integers.

Thanks for your support and best regards

[mlindauer]

Hi,

In principle, you can run SMAC on a space with 2752 hyperparameters. You only need to use the RF as the model (which is the default for the multi-fidelity facade), not the GP.
I think the largest one we ran once was also around 2k.

However, please note that if the effective dimensionality is also in the same order (i.e., nearly all hyperparameters are important) and the hyperparameters interact with each other, I doubt that SMAC will deliver a satisfying result. Nevertheless, it is often the case that only a few hyperparameters matter, and SMAC is doing a reasonable job of focusing on these.

Best,
Marius

[Mshz2]

Hi,

In principle, you can run SMAC on a space with 2752 hyperparameters. You only need to use the RF as the model (which is the default for the multi-fidelity facade), not the GP. I think the largest one we ran once was also around 2k.

However, please note that if the effective dimensionality is also in the same order (i.e., nearly all hyperparameters are important) and the hyperparameters interact with each other, I doubt that SMAC will deliver a satisfying result. Nevertheless, it is often the case that only a few hyperparameters matter, and SMAC is doing a reasonable job of focusing on these.

Best, Marius

Thank you so much and I really appreciate your reply. If it is the case, would you suggest a Random Search strategy instead of Surrogate Model-based Optimization or any Bayesian optimization methodology?

[mlindauer]

No, random search might even perform poorer than Bayesian Optimization. This won't help.
You could give it a try with local Bayesian Optimization methods such as TuRBO, but they might not have any multi-fidelity included -- I haven't checked that.