FTML optimizer #48
FTML optimizer #48
Conversation
| * @endcode | ||
| * | ||
| * For FTML to work, a DecomposableFunctionType template parameter is | ||
| * required. This class must implement the following function: |
There was a problem hiding this comment.
We have this documentation across a lot of optimizers, but the addition of function_types.md probably means we can reduce and centralize a lot of it. Do you think that we could replace this with something like... FTML requires a separable differentiable function to optimize (see <url>). The only problem with that is that it's not clear what URL to use there. I suppose ideally we'd like to point a user at the website documentation, but, the URL there could change. So I am not sure what the best choice is.
There was a problem hiding this comment.
Actually for this one I wonder if it would be best to open another issue instead of handling it in this PR.
include/ensmallen_bits/ftml/ftml.hpp
Outdated
| const size_t batchSize = 32, | ||
| const double beta1 = 0.9, | ||
| const double beta2 = 0.999, | ||
| const double eps = 1e-8, |
There was a problem hiding this comment.
Like for Padam, do you think we should adjust this documentation to point out that it's to avoid divisions by zero? (Or something like that.)
Also for consistency here I guess we should call it eps since there is a method Epsilon().
|
I'll update |
Implementation of "Follow the Moving Leader in Deep Learning" by Shuai Zheng and James T. Kwok.