Cannot use AdagradOptimizer with MirroredStrategy #19551
Assignees
Labels
Comments
|
I got the same problem when using the def model_fn(features, labels, mode):
layer = tf.layers.Dense(1)
logits = layer(features)
if mode == tf.estimator.ModeKeys.PREDICT:
predictions = {"logits": logits}
return tf.estimator.EstimatorSpec(mode, predictions=predictions)
loss = tf.losses.mean_squared_error(
labels=labels, predictions=tf.reshape(logits, []))
if mode == tf.estimator.ModeKeys.EVAL:
return tf.estimator.EstimatorSpec(mode, loss=loss)
if mode == tf.estimator.ModeKeys.TRAIN:
optimizer = tf.train.GradientDescentOptimizer(0.02)
optimizer = tf.contrib.opt.MovingAverageOptimizer(optimizer, 0.9)
train_op = optimizer.minimize(loss)
return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=train_op)
def input_fn():
features = tf.data.Dataset.from_tensors([[1.]]).repeat(100)
labels = tf.data.Dataset.from_tensors(1.).repeat(100)
return tf.data.Dataset.zip((features, labels))
distribution = tf.contrib.distribute.MirroredStrategy(num_gpus=2)
config = tf.estimator.RunConfig(train_distribute=distribution)
classifier = tf.estimator.Estimator(model_fn=model_fn, config=config)
classifier.train(input_fn=input_fn)The error with some context |
rohan100jain
added
stat:awaiting tensorflower
|
Thanks for bringing this to our attention. We will look into why AdagadOptimizer and MovingAverageOptimizer don't work. |
tensorflowbutler
removed
the
stat:awaiting tensorflower
|
Hi All, I'm getting this same Error with Adagrad (the default for pre-canned Here's my definitions: Below is the ERROR message: |
|
Nagging Assignee @anj-s: It has been 15 days with no activity and this issue has an assignee. Please update the label and/or status accordingly. |
Above two GradientDescentOptimizers do not work either. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
System information
python train_model.pyDescribe the problem
It took me a while to work out what was going on, but it seems that tf.train.AdagradOptimizer has some specific implementation detail that causes an error when used with MirroredStrategy. I did a spot check with GradientDescentOptimizer and RMSPropOptimizer and they both appear to work in my environment. I'm happy to use a different optimizer as a workaround but I thought at the very least this might save others some time hunting down the cause of the error!
Source code / logs
This is almost exactly copied from the example at https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/distribute (except for the choice of optimizer)
Log output:
The text was updated successfully, but these errors were encountered: