feat(tuner): allow adjustment of optimizer #128
Conversation
github-actions
bot
added
size/m
area/core
area/entrypoint
area/testing
Yes I agree. However, as you can see, the 3 frameworks sometimes actually have different default values. And I guess it is in our interest to ensure that training is the same for all 3 frameworks. As this already involves creating some default params, we might as well make them available to the users, if they wants to use them. |
| ) | ||
| elif optimizer == 'sgd': | ||
| return keras.optimizers.SGD(learning_rate=learning_rate, **optimizer_kwargs) | ||
|
|
There was a problem hiding this comment.
raise an value error when optimizer name is not recognised
There was a problem hiding this comment.
This is already done in _get_optimizer_kwargs
There was a problem hiding this comment.
The reason for this is that I try to factor out as much code as possible to the base class, so that it is not repeated in each framework
| @@ -39,6 +44,32 @@ def _get_data_loader(self, inputs, batch_size: int, shuffle: bool): | |||
| shuffle=shuffle, | |||
| ) | |||
|
|
|||
| def _get_optimizer( | |||
There was a problem hiding this comment.
personally i'm not a big fan of get_something, in Java or other OOP world, it feels like a getter or setter, if it is a getter, then in Python I would turn it into a property, otherwise I'll rename it how do you think?
There was a problem hiding this comment.
It's a factory function, not a getter or setter. And we already have this in the current base class (see _get_data_loader), so I would stick with this convention for now. You can raise a separate PR to change this if you want
There was a problem hiding this comment.
Btw, speaking of get naming and Java: https://www.informit.com/articles/article.aspx?p=1216151#:~:text=getType%E2%80%94Like%20getInstance%2C%20but%20used%20when%20the%20factory%20method%20is%20in%20a%20different%20class.%20Type%20indicates%20the%20type%20of%20object%20returned%20by%20the%20factory%20method.
…er into feat-configure-optimizer
tadejsv
changed the title
Adds 3 different optimizers
they are customizable (I had to find intersection of options allowed under all 3 frameworks).
Missing docstrings/documentation, but let's add that once we have everything (currently there are no docstrings in place, this PR does not feel like the right place to add them)
Also, you can now adjust learning rate. The new deafult is now
1e-3, a magintude lowe, as @maximilianwerk observed problems with high learning rate (and in general in applications the lr is low, even lower than new default)