NOTE: Issues that are not bugs or feature requests will be closed. Please ask usage questions on StackOverflow.

You must complete this information or else your issue will be closed

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow)?: Yes
• TensorFlow installed from (source or binary)?: binary
• TensorFlow version: 1.0.0 CPU / 1.0.1 (CPU and GPU enabled) / 1.1.0rc1 (CPU)
• Bazel version (if compiling from source):
• CUDA/cuDNN version: N/A
• GPU Model and Memory: N/A
• Exact command to reproduce: cf below.

This problem has been reproduced on both Linux and various Mac OS machines.

Describe the problem clearly

We seem to experience issues when using both queues + tf.train.Server. When executed in a simple python 3.5.3 console, the following script hangs:

import tensorflow as tf
import time

cluster = tf.train.ClusterSpec({"cpu1" : ['localhost:2222']})
server = tf.train.Server(cluster, job_name="cpu1", task_index=0)

with tf.Graph().as_default() as graph:
    # Queue
    input_queue = tf.train.input_producer(tf.constant([0.], dtype=tf.float32))

    # Useless variable
    variable = tf.Variable(1., dtype=tf.float32, trainable=False, name="variable")

    # Session and queue runners
    session = tf.Session(target=server.target)
    session.run(tf.global_variables_initializer())
    tf.train.start_queue_runners(session)

print(session.run(variable))  # this works
print(session.run(tf.assign(variable, 2)))  # this also works, but only if called directly

# any pause between creating and running the session breaks it
time.sleep(1)

print(session.run(variable))  # retrieving a variable still works, but...
print(session.run(tf.assign(variable, 3)))  # ... assigning a variable will make the program hang.

It outputs:

1
2
2

and then hangs forever. The problem vanishes when either commenting the input_queue=... line, or when writing session = tf.Session() instead of passing the server.target.

The problems seems to happen not only with variable assignments, but also saving the model using tf.train.Saver().save(session, 'my_model') for instance (and possibly other operations). Note that reading a variable works fine.

In the example script, the time.sleepcommand simulates a pause between creating the session and running it to set a variable. The same effect is achieved, for example, when splitting session creation and running code across two Jupyter notebook cells. When executing the whole code in one cell, it works fine.

Source Code / Logs

The source code to reproduce the problem is displayed above. I have attached a traceback using gdb, which shows that the program is hanging while trying to acquire a lock.

tf-issue-gdb-bt.txt
tf-issue-gdb-stack-threads.txt