New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should use '_train_distribution' because Estimator doesn't have attribute '_distribution' #23423

Open
wants to merge 1 commit into
base: r1.11
from

Conversation

Projects
None yet
4 participants
@RobinDong

RobinDong commented Nov 1, 2018

When I use tf.contrib.distribute.MirroredStrategy and training_hooks for one Estimator, like:

distribution = tf.contrib.distribute.MirroredStrategy(
      ["/device:GPU:0", "/device:GPU:1"])
config = tf.estimator.RunConfig(train_distribute=distribution,
                                  eval_distribute=distribution)
estimator = tf.estimator.Estimator(
      model_fn=build_model_fn_optimizer(), config=config)
estimator.train(input_fn=input_fn, steps=10)

and

    logging_hook = tf.train.LoggingTensorHook({'logits' : logits})
    return tf.estimator.EstimatorSpec(mode, loss=loss_fn(), train_op=train_op, training_hooks = [logging_hook])

It will report error:

  File "/usr/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 356, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1179, in _train_model
    return self._train_model_distributed(input_fn, hooks, saving_listeners)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1309, in _train_model_distributed
    grouped_estimator_spec.training_hooks)
  File "/usr/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1305, in get_hooks_from_the_first_device
    for per_device_hook in per_device_hooks
  File "/usr/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1305, in <listcomp>
    for per_device_hook in per_device_hooks
AttributeError: 'Estimator' object has no attribute '_distribution'

The solution is using '_train_distribution' instead of '_distribution'

@googlebot googlebot added the cla: yes label Nov 1, 2018

@akshaym akshaym requested a review from case540 Nov 1, 2018

@akshaym

This comment has been minimized.

Member

akshaym commented Nov 1, 2018

Just as a headsup, this looks like it is trying to fix tensorflow r1.11 (it seems fixed at head).

@tensorflowbutler

This comment has been minimized.

Member

tensorflowbutler commented Dec 4, 2018

Nagging Reviewer @case540: You have been added as a reviewer to this pull request. Please add your review or reassign. It has been 14 days with no activity and the awaiting review label has been applied.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment