Bug Fix and conditional weight decay

[ArlindKadra]

This PR contains a bug fix for the search space updates for the OptimizerSelector Node.
Also takes in user input in consideration for when weight decay is not desired.

[LMZimmer]

Hey Arlind,
thank you for the PR. I am not sure if we want to add another hyperparameter in use_weight_decay as the search space is already quite large. Do you think there is a scenario where that would be better than having BOHB learning to choose very low weight decay?

[ArlindKadra]

Hey Lucas,

Hmm, in general I am not sure, it depends. If users want to use certain regularization method/methods and those combined with weight decay perform badly it would be up to BOHB to learn to push the weight decay value to 0.
At the worst case, the network has about 35-37 hyperparameters with nearly everything turned off and this is already a lot and it might be difficult for BOHB to learn to push the weight decay value towards 0.
At the best case, one might see that in most configurations, the ones that have the best performance might have the weight decay value close to 0. Then, you might decide to turn it off, to narrow down the search space of the good performing hp configurations. This would be perfect if BOHB could recognize constans in the configspace, or if you overwrite the threshold by yourself.
The same holds for batch normalization.

The crucial part of the pr is the fix for setting the hyperparameters for the optimizers. It was not working before.

[ravinkohli]

@ArlindKadra can we close this PR?

[ArlindKadra]

You definitely can, this is outdated since the old code will not be continued anymore.