Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tf.contrib.estimator.replicate_model_fn fails when a trainable variable doesn't have gradient #16829

Closed
x10000year opened this issue Feb 7, 2018 · 4 comments
Closed

tf.contrib.estimator.replicate_model_fn fails when a trainable variable doesn't have gradient #16829

x10000year opened this issue Feb 7, 2018 · 4 comments
Assignees
@isaprykin
Labels
type:bug Bug

Comments

@x10000year
Copy link

tf.contrib.estimator.replicate_model_fn fails when the gradient of a trainable variable is None. The error messages are:

  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 302, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 711, in _train_model
    features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 694, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/estimator/python/estimator/replicate_model_fn.py", line 235, in replicated_model_fn
    local_ps_devices=ps_devices)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/estimator/python/estimator/replicate_model_fn.py", line 558, in _get_loss_towers
    **optional_params)
  File "model-60m-1280-2gpus-16-32-64-128-bn50000/net.py", line 38, in model_fn
    train_op = optimizer.minimize(model.total_loss, global_step)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/optimizer.py", line 353, in minimize
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/estimator/python/estimator/replicate_model_fn.py", line 317, in apply_gradients
    with ops_lib.control_dependencies(_extract_tensors(grads_and_vars)):
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 4304, in control_dependencies
    return get_default_graph().control_dependencies(control_inputs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 4017, in control_dependencies
    c = self.as_graph_element(c)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3035, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3124, in _as_graph_element_locked
    types_str))
TypeError: Can not convert a NoneType into a Tensor or Operation.
@tensorflowbutler tensorflowbutler added the stat:awaiting response Status - Awaiting response from author label Feb 7, 2018
@tensorflowbutler
Copy link
Member

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

@x10000year
Copy link
Author

@isaprykin

@isaprykin isaprykin self-assigned this Feb 12, 2018
@isaprykin isaprykin added type:bug Bug stat:awaiting response Status - Awaiting response from author and removed stat:awaiting response Status - Awaiting response from author labels Feb 12, 2018
@isaprykin
Copy link
Contributor

@x10000year Hi! I fixed this yesterday and the fix is coming. Thanks for reporting.
I'll link the commit when it's available and then close the issue. I hope you can re-built or take the nightly build to take advantage of the fix.

@x10000year
Copy link
Author

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@isaprykin isaprykin

Labels
type:bug Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@isaprykin @x10000year @tensorflowbutler