Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Repeated "Already exists: Resource" errors when I increase my model size #23780

Closed
aidanw95 opened this issue Nov 15, 2018 · 39 comments
Closed

Repeated "Already exists: Resource" errors when I increase my model size #23780

aidanw95 opened this issue Nov 15, 2018 · 39 comments

Comments

@aidanw95
Copy link

@aidanw95 aidanw95 commented Nov 15, 2018

System information

  • Custom model built by creating convolutional network tf graph with SGD optimizer and running
  • Windows 10
  • TensorFlow installed from pip
  • TensorFlow version (use command below): 1.12
  • Python version: 3.6
  • Running on cpu

I've created a convolutional network by making a custom graph for feed forward and using a tensorflow optimizer to train it (I've tried with SGD and Adam).

`sess = tf.Session()
optimizer = tf.train.GradientDescentOptimizer(learning_rate=.01)

input_size = 16
input_matrix = tf.placeholder(tf.float32, shape=[1,input_size,input_size])
label_ph = tf.placeholder(tf.float32, shape=[28*28])
model_out = Model.feed_forward(input_matrix)

init = tf.global_variables_initializer()
sess.run(init)

loss = tf.losses.mean_squared_error([tf.convert_to_tensor(label_ph, tf.float32)], [model_out])
train = optimizer.minimize(loss)
sess.graph.finalize()`

This is my graph definition. I know Model.feed_forward is opaque but it's basically a series of functions that get variable weights and feed input through layers. Just to give you an idea here's an example of a convolutional feed forward:

`def feed_input(input_tensors, filters, biases, stride, zero_pad=True):
        if zero_pad:
            pad_size = int((int(filters[0].shape[0]) + 1)/2 - 1)
            input_tensors = tf.map_fn(lambda inp:
                tf.pad(inp, [[pad_size, pad_size], [pad_size, pad_size]]), input_tensors)

        _, shape_x, _ = input_tensors.shape

        unflattened_output = tf.map_fn(lambda input_tensor: tf.convert_to_tensor([[[tf.add(tf.tensordot(curr_filter,
            ConvLayer.__get_input_slice__(input_tensor, x, y, tf.shape(curr_filter)[0]), 2), biases[filter_ind])
         for x in range(0, shape_x - curr_filter.shape[0] + 1, stride)]
         for y in range(0, shape_x - curr_filter.shape[0] + 1, stride)]
         for filter_ind, curr_filter in enumerate(filters)], dtype=tf.float32), input_tensors)

        shape_1, shape_2, shape_3, shape_4 = unflattened_output.shape

        return tf.reshape(unflattened_output, [shape_1 * shape_2, shape_3, shape_4])`

After defining my graph I am running a loop that gets some input and runs sess.run(train, feed_dict={input_matrix:img, label_ph:label_image}). If I make input_size smaller (e.g. 4) or if I reduce the number of layers in my network, everything runs fine and it does exactly what I expect it to do. However, when my network grows in the ways I just mentioned it consistently spits out these errors during sess.run(train, feed_dict={input_matrix:img, label_ph:label_image}):

2018-11-15 00:46:31.069913: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.069914: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.069914: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.069927: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.070138: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.078467: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.080434: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.081575: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.083125: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.085370: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.089281: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.090932: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.091080: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.091398: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.091916: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.092735: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.098503: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.094920: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.099036: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.101236: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.101988: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.101966: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.103264: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.102945: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.106092: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.106858: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.108494: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.109495: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.109987: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.112326: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.112525: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.113595: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.113831: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.115342: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.115418: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.120685: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.122569: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.122708: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.123904: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.124832: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.126926: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.127336: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.129575: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.128498: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.115477: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.130180: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.130601: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.131469: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.135938: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.132834: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.139111: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.139728: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.140827: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.141429: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.142202: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.140004: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.142621: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.144039: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.147757: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.145043: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.149502: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.146723: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.148971: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.145861: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.149850: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.151642: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.152947: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.154227: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.155666: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.157249: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.157877: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.159295: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.160204: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.163481: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.163750: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.164518: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.166533: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.172400: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.174768: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.174793: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
Traceback (most recent call last):
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
[[{{node gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}} = TemporaryVariabledtype=DT_FLOAT, shape=[5,5], var_name="gradients/...dd/tmp_var", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/Aidan/Documents/PycharmProjects/SketchIt/src/network/FeedImages.py", line 57, in
sess.run(train, feed_dict={input_matrix:img, label_ph:label_image})
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
run_metadata_ptr)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
run_metadata)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
[[{{node gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}} = TemporaryVariabledtype=DT_FLOAT, shape=[5,5], var_name="gradients/...dd/tmp_var", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Anyone know what might be the cause? Maybe some concurrency issue since its telling me things that shouldn't exist have already been created? Help would be much appreciated, I'm close to making this thing work! Thanks for your time.

@alextp
Copy link
Contributor

@alextp alextp commented Nov 27, 2018

What is "AddOpsRewrite_Add/tmp_var/struct" in your graph? Why are you using temporary variables? They are not compatible with while loops, which is what you're seeing in the error message.

(I thought temporary variables had been removed as part of tf's API a long time ago)

@aidanw95 aidanw95 closed this Nov 27, 2018
@aidanw95
Copy link
Author

@aidanw95 aidanw95 commented Nov 27, 2018

Thanks for the response.

Those are created by sess.run(train) where train is GradientDescentOptimizer.minimize. So tensorflow is generating them, not me.

Also I should update that I rolled back to tensorflow 1.8 (haven't tried 9, 10, or 11 just due to lack of time) and the error no longer comes up.

@aidanw95 aidanw95 reopened this Nov 27, 2018
@aidanw95
Copy link
Author

@aidanw95 aidanw95 commented Nov 27, 2018

Issue close was accidental

@alextp
Copy link
Contributor

@alextp alextp commented Nov 27, 2018

What is ArithmeticOptimizer in your code? These ops don't look like they were added by TF's backward pass.

@alextp
Copy link
Contributor

@alextp alextp commented Nov 27, 2018

Hm this might come from grappler.

@alextp
Copy link
Contributor

@alextp alextp commented Nov 27, 2018

@rmlarsen I see that the memory optimizer adds temporaryvariable. This is not safe to do inside a while loop because the temporaryvariable op has weird issues with the resource manager.

@aidanw95
Copy link
Author

@aidanw95 aidanw95 commented Nov 28, 2018

ArithmeticOptimizer comes from running GradientDescentOptimizer.minimize as well. I'm running train in a few nested python for loops. I figured that's super typical, but should I be making a list of inputs and feeding them in with one call instead?

If anyone's interested you can see exactly what I'm running here. I already uploaded the network on a different account. https://github.com/awoods12/Sketch_It

@matanagasauce
Copy link

@matanagasauce matanagasauce commented Dec 3, 2018

Could it be an issue about tf.map_fn rather than optimizer?
I met the same errors and the core part of my code was like

y = tf.map_fn(some_function, x)

I found it worked when x had shape [1], but got errors otherwise.
And then I found another way without map_fn and everything works well now.

@alextp
Copy link
Contributor

@alextp alextp commented Dec 3, 2018

@artsobolev
Copy link
Contributor

@artsobolev artsobolev commented Feb 7, 2019

I have the same problem on TF 1.12 + Ubuntu 16.04 and my custom code. Weirdly enough, even though I share the vact_scope code in different networks, the problem does not happen for all of them (It might be due to calling tf.nn.dynamic_rnn in the same scope twice with scope.reuse_variables() between the calls)

@ezhulenev, any news on the problem? Any workarounds? Maybe there's a way to disable the ArithmeticOptimizer? (or whole Grappler? What is the expected performance hit then?)

2019-02-07 17:26:05.004296: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.030579: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.031806: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.048779: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.052690: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.054059: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.056958: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.062997: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.074335: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.093691: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.106734: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.128811: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.145847: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.147453: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.164530: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.168118: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.174878: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.181530: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.242049: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.287269: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.375503: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.737421: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.737421: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.756781: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.757632: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.915909: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE

Weirdly, I see strange nested scopes (__per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE), which I do not create in the code (the GLOBAL_VARIABLES collection does not contain any variables with vact_iteration/vact_iteration in the name)

@ezhulenev
Copy link
Contributor

@ezhulenev ezhulenev commented Feb 7, 2019

You can disable all optimizations in arithmetic optimizer with something like:

    from google3.third_party.tensorflow.core.protobuf import rewriter_config_pb2
    
    config_proto = tf.ConfigProto()

    off = rewriter_config_pb2.RewriterConfig.OFF
    config_proto.graph_options.rewrite_options.arithmetic_optimization = off
    
    session = tf.Session(config=config_proto)

another example in

def _no_rewrite_session_config(self):
rewriter_config = rewriter_config_pb2.RewriterConfig(
dependency_optimization=rewriter_config_pb2.RewriterConfig.OFF,
pin_to_host_optimization=rewriter_config_pb2.RewriterConfig.OFF,
min_graph_nodes=-1)
graph_options = config_pb2.GraphOptions(rewrite_options=rewriter_config)
return config_pb2.ConfigProto(graph_options=graph_options)

But I suspect that the real problem is in memory optimizer, you can disable it with this option:

MemOptType memory_optimization = 4;

Can you try if disabling one of them helps.

@artsobolev
Copy link
Contributor

@artsobolev artsobolev commented Feb 7, 2019

Either arithmetic_optimization = off or memory_optimization = off does the job, thanks!

UPD 8/02: my problem was also caused by the model size, disregard the "calling tf.nn.dynamic_rnn in the same scope twice" in my previous comment.

@alextp alextp closed this Feb 22, 2019
@saforem2
Copy link

@saforem2 saforem2 commented Mar 1, 2019

@artsobolev Out of curiosity, what was the issue with the model size?

I was running into the same issue as described but was able to 'fix' it by disabling arithmetic optimizations via

    from google3.third_party.tensorflow.core.protobuf import rewriter_config_pb2
    
    config_proto = tf.ConfigProto()

    off = rewriter_config_pb2.RewriterConfig.OFF
    config_proto.graph_options.rewrite_options.arithmetic_optimization = off
    
    session = tf.Session(config=config_proto)

without having to disable memory optimization.

Also, I was wondering if anyone might have any additional insight into this issue and/or any ideas about possibly fixing it instead of just disabling arithmetic optimizations?

@ezhulenev
Copy link
Contributor

@ezhulenev ezhulenev commented Mar 1, 2019

Do you have minimal reproducible example? I have a few theories, but small example would help.

@artsobolev
Copy link
Contributor

@artsobolev artsobolev commented Mar 1, 2019

@saforem2, disabling optimizations worked for me as well, but so did decreasing the hidden layer size in the LSTM from 512 to 128. Apparently, the optimizer behaves differently in these cases.

@ezhulenev, unfortunately, I do not have such example.

@saforem2
Copy link

@saforem2 saforem2 commented Mar 7, 2019

@ezhulenev Unfortunately I don't have a minimal reproducible example either.

If you really think it would help, I can work on writing up a gist or walking through the steps to reproduce the error, but I think it would require cloning a (moderately sized) repository of mine.

@ezhulenev
Copy link
Contributor

@ezhulenev ezhulenev commented Mar 7, 2019

I'll try to work on a fix, I'll let you know when it will be in master, so you could test it.

@ghazi-f
Copy link

@ghazi-f ghazi-f commented Apr 11, 2019

Hello,
I am facing a similar issue but I can't seem to find this google3 library to import the off constant mentioned by the comments above.

Can anyone tell me where to get google3 ?
Thank you.

@weixsong
Copy link

@weixsong weixsong commented May 6, 2019

I also meet the same issue with tf.nn.conv1d, by using

    config_proto = tf.ConfigProto(log_device_placement=False, allow_soft_placement=True)
    off = rewriter_config_pb2.RewriterConfig.OFF
    config_proto.graph_options.rewrite_options.arithmetic_optimization = off

    sess = tf.Session(config=config_proto)

the issue is fixed. So i think maybe tf1.13 has some inner bug.

@abdo96
Copy link

@abdo96 abdo96 commented May 8, 2019

I had the same problem about import rewriter_config_pb2 from google3

from tensorflow.core.protobuf import rewriter_config_pb2

config_proto = tf.ConfigProto(log_device_placement=False, allow_soft_placement=True)

off = rewriter_config_pb2.RewriterConfig.OFF

config_proto.graph_options.rewrite_options.arithmetic_optimization = off

session = tf.Session(config=config_proto)


@alextp
Copy link
Contributor

@alextp alextp commented May 9, 2019

Just remove "google3" from the example and it should work.

@abdo96
Copy link

@abdo96 abdo96 commented May 9, 2019

But I had another problem I used python3.6 and I countered this problem "AlreadyExists error"

@abdo96
Copy link

@abdo96 abdo96 commented May 10, 2019

I tried to disable the arithmetic optimizer but I don't know where does it ? in version python3.7, TF1.13.1
I access the arithmetic optimizer.h and disabled all variables. But it hasn't effect on the error.


2019-05-10 23:23:48.280599: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.280882: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.281480: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.281866: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.282864: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.283515: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.284000: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.284436: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.284976: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.285390: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.285765: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.286219: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.286526: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.287017: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.287346: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.287845: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.288143: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.288607: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.289050: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.289331: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.289795: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.290269: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.290532: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.291102: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.292775: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.293485: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.293981: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.294523: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.295292: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.296044: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.296854: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
	 [[{{node training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Generator2.py", line 232, in <module>
    estimator_model.train(input_fn=customTrain, steps=100)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 358, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1124, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model_default
    saving_listeners)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1407, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 676, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1171, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1270, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1255, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1327, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1091, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
	 [[{{node training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}}]]
'''

@183amir
Copy link

@183amir 183amir commented May 12, 2019

If you are using the estimators API, try something like this:

import tensorflow as tf
from tensorflow.core.protobuf import rewriter_config_pb2
session_config = tf.ConfigProto()
off = rewriter_config_pb2.RewriterConfig.OFF
session_config.graph_options.rewrite_options.arithmetic_optimization = off
run_config = tf.estimator.RunConfig(session_config=session_config)
...
estimator = tf.estimator.Estimator(model_fn, model_dir, run_config)

Note to developers: this issue is not fixed. Theses are just workarounds as far as I can understand. Could you please re-open the issue?

@abdo96
Copy link

@abdo96 abdo96 commented May 12, 2019

Thank so much. I will try then If it's still existing. I will re-open issue @183amir

@livinglife2x
Copy link

@livinglife2x livinglife2x commented May 26, 2019

Perfect solution. I ran my model on non GPU machine and ended up with this error, on GPU machine it works totally fine.. Does presence/absence of GPU anything to do with arithmetic optimization?

@bignamehyp
Copy link
Member

@bignamehyp bignamehyp commented Jun 2, 2019

Same problem here. Workaround is not a fix.
Reopen it.

@bignamehyp bignamehyp reopened this Jun 2, 2019
@rmlarsen
Copy link
Member

@rmlarsen rmlarsen commented Jun 3, 2019

Thank you for the report. We will fix the corresponding rewrite in Grappler. It looks like it generates non-unique node names.

@pekaalto
Copy link

@pekaalto pekaalto commented Jun 14, 2019

I have this exact same issue in tf 2.0 beta0. Is there a workaround for that?

I tried:

tf.config.optimizer.set_experimental_options({'arithmetic_optimization': False})

and

tf.config.optimizer.set_experimental_options({'arithmetic_optimization': rewriter_config_pb2.RewriterConfig.OFF})

but it didn't help.

E: For me issue happens only with AdamOptimizer. RMSProp is fine

@timakro
Copy link

@timakro timakro commented Jun 19, 2019

I'm having the same issue and I can't get the workaround to work. Do I do something with the tf.Session instance after creating it or will it automatically be used?

For now I just try different sequence lengths in my model, some work, some don't.

EDIT: Figured it out: To make keras use the model call tensorflow.keras.backend.set_session with the model instance.

@sivakarthikgade
Copy link

@sivakarthikgade sivakarthikgade commented Jul 14, 2019

tensorflow.keras.backend.set_session worked. using TF 1.13

@Rassibassi
Copy link

@Rassibassi Rassibassi commented Aug 9, 2019

For me, either reducing the graph size or memory_optimization = off helped. TF 1.14

As follows:

from tensorflow.core.protobuf import rewriter_config_pb2

config_proto = tf.ConfigProto()
off = rewriter_config_pb2.RewriterConfig.OFFrite_options.arithmetic_optimization = off
config_proto.graph_options.rewrite_options.memory_optimization  = off

@danijar
Copy link
Member

@danijar danijar commented Dec 30, 2019

Here is the corrected code example:

from tensorflow.core.protobuf import rewriter_config_pb2
config = tf.ConfigProto()
off = rewriter_config_pb2.RewriterConfig.OFF
config.graph_options.rewrite_options.memory_optimization  = off
return tf.Session(config=config)

@mdev111
Copy link

@mdev111 mdev111 commented Feb 27, 2020

For me making the second parameter of Embedding layer smaller (LSTM) worked.

@sushreebarsa
Copy link
Contributor

@sushreebarsa sushreebarsa commented Sep 20, 2021

@aidanw95
It looks like you are using an older Version of Tensorflow. Many bugs have been fixed in the latest version. Can you please execute your code using Latest Version (2.6.0) and let us know if the issue still persists? Thanks!

@google-ml-butler
Copy link

@google-ml-butler google-ml-butler bot commented Sep 27, 2021

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

@google-ml-butler
Copy link

@google-ml-butler google-ml-butler bot commented Oct 4, 2021

Closing as stale. Please reopen if you'd like to work on this further.

@google-ml-butler google-ml-butler bot closed this Oct 4, 2021
@google-ml-butler
Copy link

@google-ml-butler google-ml-butler bot commented Oct 4, 2021

Are you satisfied with the resolution of your issue?
Yes
No

@sairajm
Copy link

@sairajm sairajm commented Oct 15, 2021

This is definitely an issue in Tensorflow V2.6.0.
I'm using python 3.9.
Tensorflow installed using binaries(pip).
Code runs on a CPU.

Like the comments above suggest, turning off arithmetic optimization was also the fix workaround for me. But their code is not compatible with V2, so I'll post a snippet with what worked. I used the docs on graph optimization for reference: https://www.tensorflow.org/guide/graph_optimization.

import contextlib
@contextlib.contextmanager
def options(options):
  old_opts = tf.config.optimizer.get_experimental_options()
  tf.config.optimizer.set_experimental_options(options)
  try:
    yield
  finally:
    tf.config.optimizer.set_experimental_options(old_opts)

def test_function(body):
  with options({'arithmetic_optimization': False}):
          tf_tensor = tf.constant([body])
          if model:
              tensor_object = model(tf_tensor)
.....

I'm not sure if this is an issue because I'm running this on a CPU but it seems like if the code isn't able to detect a GPU then these should be sensibly turned off.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet