You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04 (but irrelevant)
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No
TensorFlow installed from (source or binary): binary (but irrelevant)
TensorFlow version (use command below): 2.3, 2.7.0, current master
Python version: 3.7
Bazel version (if compiling from source): -
GCC/Compiler version (if compiling from source): -
CUDA/cuDNN version: -
GPU model and memory: -
Describe the current behavior
Nadam as implemented in the kernel via the use_nesterov option (tensorflow/tensorflow/core/kernels/training_ops_gpu.cu.cc) behaves different to tf.keras.optimizers.Nadam.
We observed this problem (here: rwth-i6/returnn#766 (comment)) where the model converged in TF 1 with the old Nadam and does not anymore with TF 2 with the new Nadam.
There are differences is the calculation of m_t and v_t. While mathematically the same, the variant in the kernel is probably numerically more stable. The difference is
m_i = m_i + one_minus_beta1 * (g_i - m_i);
vs
m_i = beta1 * m_i + one_minus_beta1 * g_i;
I think the first variant is better here. I think this should be adopted in the Keras code.
Another difference is mul_factor vs lr, and how var[i] (var_t) is calculated.
Describe the expected behavior
I would expect that Nadam as implemented in the kernel via the use_nesterov option (tensorflow/tensorflow/core/kernels/training_ops_gpu.cu.cc, available since TF 1, also still in current master), which could be used via tensorflow.contrib.opt.NadamOptimizer in TF 1, is conceptually the same as tf.keras.optimizers.Nadam.
Also, TF provides the ApplyAdamOp with Nesterov support. I would expect that Keras Nadam makes use of that anyway, as it should also be a bit more efficient.
The text was updated successfully, but these errors were encountered:
This is copied and adopted from here: tensorflow/tensorflow#53204
System information
Describe the current behavior
Nadam as implemented in the kernel via the
use_nesterov
option (tensorflow/tensorflow/core/kernels/training_ops_gpu.cu.cc
) behaves different totf.keras.optimizers.Nadam
.We observed this problem (here: rwth-i6/returnn#766 (comment)) where the model converged in TF 1 with the old Nadam and does not anymore with TF 2 with the new Nadam.
Specifically, in the kernel, you have this code:
And in the Keras Nadam optimizer, we have this code:
There are differences is the calculation of
m_t
andv_t
. While mathematically the same, the variant in the kernel is probably numerically more stable. The difference isvs
I think the first variant is better here. I think this should be adopted in the Keras code.
Another difference is
mul_factor
vslr
, and howvar[i]
(var_t
) is calculated.Describe the expected behavior
I would expect that Nadam as implemented in the kernel via the
use_nesterov
option (tensorflow/tensorflow/core/kernels/training_ops_gpu.cu.cc
, available since TF 1, also still in current master), which could be used viatensorflow.contrib.opt.NadamOptimizer
in TF 1, is conceptually the same astf.keras.optimizers.Nadam
.Also, TF provides the
ApplyAdamOp
with Nesterov support. I would expect that Keras Nadam makes use of that anyway, as it should also be a bit more efficient.The text was updated successfully, but these errors were encountered: