Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

InvalidArgumentError when used with Estimator #1

Open
bwulff opened this issue Mar 17, 2018 · 1 comment
Open

InvalidArgumentError when used with Estimator #1

bwulff opened this issue Mar 17, 2018 · 1 comment
Labels
bug Something isn't working

Comments

@bwulff
Copy link
Contributor

bwulff commented Mar 17, 2018

InvalidArgumentError when using the Optimizer with tf.Estimator:

2018-03-17 12:01:15.805973: W tensorflow/core/kernels/queue_base.cc:285] _1_Evaluator/Evaluator/N_Eval/enqueue_input/random_shuffle_queue: Skipping cancelled dequeue attempt with queue not closed
2018-03-17 12:01:15.806118: W tensorflow/core/kernels/queue_base.cc:285] _2_Evaluator/Evaluator/P_Eval/enqueue_input/random_shuffle_queue: Skipping cancelled dequeue attempt with queue not closed
Traceback (most recent call last):
  File "cnn_mnist.py", line 162, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "cnn_mnist.py", line 149, in main
    hooks=[logging_hook])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 352, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 891, in _train_model
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 546, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1022, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1113, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1098, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1170, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 950, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 905, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1137, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1355, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1374, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Duplicate tag enqueue_input/queue/enqueue_input/random_shuffle_queuefraction_over_250_of_750_full found in summary inputs
	 [[Node: Merge/MergeSummary = MergeSummary[N=4, _device="/job:localhost/replica:0/task:0/device:CPU:0"](enqueue_input/queue/enqueue_input/random_shuffle_queuefraction_over_250_of_750_full, Evaluator/Evaluator/N_Eval/enqueue_input/queue/enqueue_input/random_shuffle_queuefraction_over_250_of_750_full, Evaluator/Evaluator/P_Eval/enqueue_input/queue/enqueue_input/random_shuffle_queuefraction_over_250_of_750_full, loss)]]

Caused by op u'Merge/MergeSummary', defined at:
  File "cnn_mnist.py", line 162, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "cnn_mnist.py", line 149, in main
    hooks=[logging_hook])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 352, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 888, in _train_model
    log_step_count_steps=self._config.log_step_count_steps) as mon_sess:
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 384, in MonitoredTrainingSession
    stop_grace_period_secs=stop_grace_period_secs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 795, in __init__
    stop_grace_period_secs=stop_grace_period_secs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 518, in __init__
    self._sess = _RecoverableSession(self._coordinated_creator)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 981, in __init__
    _WrappedSession.__init__(self, self._create_session())
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 986, in _create_session
    return self._sess_creator.create_session()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 675, in create_session
    self.tf_sess = self._session_creator.create_session()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 437, in create_session
    self._scaffold.finalize()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 209, in finalize
    summary.merge_all)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 264, in get_or_default
    op = default_constructor()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/summary/summary.py", line 318, in merge_all
    return merge(summary_ops)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/summary/summary.py", line 284, in merge
    val = _gen_logging_ops._merge_summary(inputs=inputs, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 314, in _merge_summary
    "MergeSummary", inputs=inputs, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1650, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Duplicate tag enqueue_input/queue/enqueue_input/random_shuffle_queuefraction_over_250_of_750_full found in summary inputs
	 [[Node: Merge/MergeSummary = MergeSummary[N=4, _device="/job:localhost/replica:0/task:0/device:CPU:0"](enqueue_input/queue/enqueue_input/random_shuffle_queuefraction_over_250_of_750_full, Evaluator/Evaluator/N_Eval/enqueue_input/queue/enqueue_input/random_shuffle_queuefraction_over_250_of_750_full, Evaluator/Evaluator/P_Eval/enqueue_input/queue/enqueue_input/random_shuffle_queuefraction_over_250_of_750_full, loss)]]
@bwulff
Copy link
Contributor Author

bwulff commented Mar 17, 2018 •
edited
Loading

Seems to have to do with the way that MonitoredTrainingSession is created and that the original graph is cloned twice.

@bwulff bwulff added the bug Something isn't working label Mar 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bwulff