control_dependencies throw Frame error when used in while_loop

[gongbudaizhe]

Hello,

Problem description

I need to put InceptionV3 in a while loop to save both GPU and CPU memory usage since I am handling videos, each contains hundreds of images. The problem is, InceptionV3 uses control_dependencies for BatchNorm and TensorFlow throws Frame Error if control_dependencies function is in the while_loop. It can run without errors if control_dependencies is removed though.

Below is a minimal snippet that reproduces the error:

sess = tf.Session()

with tf.variable_scope('state'):
    x = tf.get_variable('x', shape=(), 
                             initializer=tf.constant_initializer(1), 
                             dtype=tf.float32)
    update_x = tf.assign(x, x+1)

def iter_fun(i, y):
    # comment the line below, the program will run without any error
    # but I need control_dependencies, or at least some way to replace it...
    with tf.control_dependencies([update_x]): 
        y = y + x
    return (i+1, y)

with tf.variable_scope('iteration'):
    num_iterations = 5   
    initial_i = tf.constant(0, dtype=tf.int32)
    initial_y = tf.constant(0, dtype=tf.float32)
    _, result = tf.while_loop(
        cond=lambda i, *_: i < num_iterations,
        body=iter_fun,
        loop_vars=(initial_i, initial_y))

init_op = tf.global_variables_initializer()
sess.run(init_op)
sess.run(result)  

The stack trace of the error:

Traceback (most recent call last):
  File "demo.py", line 28, in <module>
    sess.run(result)
  File "/workspace/bily/anoaconda2/envs/tensorflow0.12/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 766, in run
    run_metadata_ptr)
  File "/workspace/bily/anoaconda2/envs/tensorflow0.12/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 964, in _run
    feed_dict_string, options, run_metadata)
  File "/workspace/bily/anoaconda2/envs/tensorflow0.12/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run
    target_list, options, run_metadata)
  File "/workspace/bily/anoaconda2/envs/tensorflow0.12/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: The node 'iteration/while/add' has inputs from different frames. The input 'iteration/while/add/Enter' is in frame 'iteration/while/iteration/while/'. The input 'state/Assign' is in frame ''.

Environments

• CentOS Linux release 7.2.1511
• TensorFlow 0.12 built from source
• Python 2.7.12
• CUDA 7.5 and CUDNN v5.1

Related issues

1. This issue in tflearn seems to be related to my problem but removing control_denpendencies isn't a solution for me.
2. Frame of the Variable #4478 and Backpropagation through the while-loop doesn't work if external tensors are used inside #3114 are issues about frame errors but these errors are caused by variables instead of control_dependencies.

Any help will be appreciated : )

[yuanbyu]

Thanks for the report. I have a fix that should show up here in a few days.

[gongbudaizhe]

@yuanbyu Looking forward to it 😃

[gongbudaizhe]

@yuanbyu how is it going now ?

[natlachaman]

I run into a similar error too, involving a while loop
InvalidArgumentError: The node 'map_15/while/map/while/Variable/Assign' has inputs from different frames. The input 'map_15/while/map/while/Const' is in frame 'map_15/while/map/while/map_15/while/map/while/'. The input 'map_15/while/map/while/Variable' is in frame ''.

What is the state of this issue?

[Pastromhaug]

I would also like to know the state of this issue. Getting tensorflow.python.framework.errors_impl.InvalidArgumentError: The node 'RNN/DizzyRNNCellOptHacky/Linear/Matrix/Assign' has inputs from different frames. The input 'RNN/while/DizzyRNNCellOptHacky/Linear/random_normal' is in frame 'RNN/while/RNN/while/'. The input 'RNN/DizzyRNNCellOptHacky/Linear/Matrix' is in frame ''.

[ffmpbgrnn]

@yuanbyu , it seems slim.batch_norm still cannot be used inside while_loop. Here is the snippet to reproduce the error: tensorflow.python.framework.errors_impl.InvalidArgumentError: The node 'control_dependency' has inputs from different frames. The input 'update_barrier' is in frame 'iteration/while/iteration/while/'. The input 'Mean' is in frame ''., I am using commit 2255bc2c1ada8a300adfc8a9a1e68d59c5df4296.

import tensorflow as tf
from tensorflow.python.framework import ops
from tensorflow.python.ops import control_flow_ops

slim = tf.contrib.slim

sess = tf.Session()

with tf.variable_scope('state'):
    x = tf.get_variable('x', shape=(),
                             initializer=tf.constant_initializer(1),
                             dtype=tf.float32)
    update_x = tf.assign(x, x+1)

def iter_fun(i, y):
    # comment the line below, the program will run without any error
    # but I need control_dependencies, or at least some way to replace it...
    # with tf.control_dependencies([update_x]):
        # y = y + x
    y = slim.batch_norm(y)
    return (i+1, y)

with tf.variable_scope('iteration'):
    num_iterations = 5
    initial_i = tf.constant(0, dtype=tf.int32)
    initial_y = tf.constant((1, 2, 3, 4), dtype=tf.float32)
    _, result = tf.while_loop(
        cond=lambda i, *_: i < num_iterations,
        body=iter_fun,
        loop_vars=(initial_i, initial_y))

total_loss = tf.reduce_mean(result)
batchnorm_updates = set(tf.get_collection(ops.GraphKeys.UPDATE_OPS))
if batchnorm_updates:
    with tf.control_dependencies(batchnorm_updates):
        barrier = tf.no_op(name='update_barrier')
    total_loss = control_flow_ops.with_dependencies([barrier], total_loss)

opt = tf.train.AdamOptimizer()
gradients = opt.compute_gradients(total_loss)
apply_gradient_op = opt.apply_gradients(gradients)

init_op = tf.global_variables_initializer()
sess.run(init_op)
print(sess.run(apply_gradient_op))
print(sess.run(total_loss))

[sid027]

@yuanbyu Is the fix done?

[yuanbyu]

@gongbudaizhe, I think I have long submitted a fix to your problem. Let me know if you still see a problem.

[yuanbyu]

@ffmpbgrnn, the problem with your program is that slim.batch_norm() is created in the loop body so the update_ops of batch_norm can't be used outside of the loop. (We don't allow a control edge from an op inside the loop body to an op outside the loop.)

Those update ops are completed by the time that result is completed. So all the control dependency logic for them is probably not needed.

[gongbudaizhe]

@yuanbyu Sorry, I was working on something else after coming across this bug. I just checked the snippet reproducing the bug and now it works smoothly in TensorFlow 1.0.

Thank you!

[naveenariva]

@yuanbyu @ffmpbgrnn I'm having a similar problem and am getting tensorflow.python.framework.errors_impl.InvalidArgumentError: The node 'update_barrier' has inputs from different frames. The input 'while/BatchNorm/AssignMovingAvg_1' is in frame 'while/while/'. The input 'fc1/BatchNorm/AssignMovingAvg_1' is in frame ''. Is there a way to use batch-norm inside tf.while_loop?

[naveenariva]

Setting updates_collections=None in BatchNorm seems to resolve the issue since this ensures that updates are done within a frame.