Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Repeated "Already exists: Resource" errors when I increase my model size #23780

Open
aidanw95 opened this issue Nov 15, 2018 · 32 comments
Assignees
Labels

Comments

@aidanw95
Copy link

@aidanw95 aidanw95 commented Nov 15, 2018

System information

  • Custom model built by creating convolutional network tf graph with SGD optimizer and running
  • Windows 10
  • TensorFlow installed from pip
  • TensorFlow version (use command below): 1.12
  • Python version: 3.6
  • Running on cpu

I've created a convolutional network by making a custom graph for feed forward and using a tensorflow optimizer to train it (I've tried with SGD and Adam).

`sess = tf.Session()
optimizer = tf.train.GradientDescentOptimizer(learning_rate=.01)

input_size = 16
input_matrix = tf.placeholder(tf.float32, shape=[1,input_size,input_size])
label_ph = tf.placeholder(tf.float32, shape=[28*28])
model_out = Model.feed_forward(input_matrix)

init = tf.global_variables_initializer()
sess.run(init)

loss = tf.losses.mean_squared_error([tf.convert_to_tensor(label_ph, tf.float32)], [model_out])
train = optimizer.minimize(loss)
sess.graph.finalize()`

This is my graph definition. I know Model.feed_forward is opaque but it's basically a series of functions that get variable weights and feed input through layers. Just to give you an idea here's an example of a convolutional feed forward:

`def feed_input(input_tensors, filters, biases, stride, zero_pad=True):
        if zero_pad:
            pad_size = int((int(filters[0].shape[0]) + 1)/2 - 1)
            input_tensors = tf.map_fn(lambda inp:
                tf.pad(inp, [[pad_size, pad_size], [pad_size, pad_size]]), input_tensors)

        _, shape_x, _ = input_tensors.shape

        unflattened_output = tf.map_fn(lambda input_tensor: tf.convert_to_tensor([[[tf.add(tf.tensordot(curr_filter,
            ConvLayer.__get_input_slice__(input_tensor, x, y, tf.shape(curr_filter)[0]), 2), biases[filter_ind])
         for x in range(0, shape_x - curr_filter.shape[0] + 1, stride)]
         for y in range(0, shape_x - curr_filter.shape[0] + 1, stride)]
         for filter_ind, curr_filter in enumerate(filters)], dtype=tf.float32), input_tensors)

        shape_1, shape_2, shape_3, shape_4 = unflattened_output.shape

        return tf.reshape(unflattened_output, [shape_1 * shape_2, shape_3, shape_4])`

After defining my graph I am running a loop that gets some input and runs sess.run(train, feed_dict={input_matrix:img, label_ph:label_image}). If I make input_size smaller (e.g. 4) or if I reduce the number of layers in my network, everything runs fine and it does exactly what I expect it to do. However, when my network grows in the ways I just mentioned it consistently spits out these errors during sess.run(train, feed_dict={input_matrix:img, label_ph:label_image}):

2018-11-15 00:46:31.069913: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.069914: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.069914: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.069927: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.070138: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.078467: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.080434: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.081575: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.083125: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.085370: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.089281: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.090932: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.091080: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.091398: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.091916: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.092735: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.098503: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.094920: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.099036: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.101236: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.101988: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.101966: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.103264: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.102945: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.106092: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.106858: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.108494: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.109495: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.109987: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.112326: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.112525: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.113595: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.113831: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.115342: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.115418: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.120685: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.122569: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.122708: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.123904: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.124832: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.126926: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.127336: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.129575: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.128498: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.115477: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.130180: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.130601: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.131469: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.135938: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.132834: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.139111: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.139728: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.140827: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.141429: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.142202: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.140004: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.142621: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.144039: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.147757: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.145043: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.149502: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.146723: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.148971: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.145861: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.149850: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.151642: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.152947: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.154227: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.155666: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.157249: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.157877: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.159295: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.160204: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.163481: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.163750: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.164518: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.166533: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.172400: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.174768: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.174793: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
Traceback (most recent call last):
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
[[{{node gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}} = TemporaryVariabledtype=DT_FLOAT, shape=[5,5], var_name="gradients/...dd/tmp_var", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/Aidan/Documents/PycharmProjects/SketchIt/src/network/FeedImages.py", line 57, in
sess.run(train, feed_dict={input_matrix:img, label_ph:label_image})
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
run_metadata_ptr)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
run_metadata)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
[[{{node gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}} = TemporaryVariabledtype=DT_FLOAT, shape=[5,5], var_name="gradients/...dd/tmp_var", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Anyone know what might be the cause? Maybe some concurrency issue since its telling me things that shouldn't exist have already been created? Help would be much appreciated, I'm close to making this thing work! Thanks for your time.

@alextp

This comment has been minimized.

Copy link
Member

@alextp alextp commented Nov 27, 2018

What is "AddOpsRewrite_Add/tmp_var/struct" in your graph? Why are you using temporary variables? They are not compatible with while loops, which is what you're seeing in the error message.

(I thought temporary variables had been removed as part of tf's API a long time ago)

@aidanw95 aidanw95 closed this Nov 27, 2018
@aidanw95

This comment has been minimized.

Copy link
Author

@aidanw95 aidanw95 commented Nov 27, 2018

Thanks for the response.

Those are created by sess.run(train) where train is GradientDescentOptimizer.minimize. So tensorflow is generating them, not me.

Also I should update that I rolled back to tensorflow 1.8 (haven't tried 9, 10, or 11 just due to lack of time) and the error no longer comes up.

@aidanw95 aidanw95 reopened this Nov 27, 2018
@aidanw95

This comment has been minimized.

Copy link
Author

@aidanw95 aidanw95 commented Nov 27, 2018

Issue close was accidental

@alextp

This comment has been minimized.

Copy link
Member

@alextp alextp commented Nov 27, 2018

What is ArithmeticOptimizer in your code? These ops don't look like they were added by TF's backward pass.

@alextp

This comment has been minimized.

Copy link
Member

@alextp alextp commented Nov 27, 2018

Hm this might come from grappler.

@alextp

This comment has been minimized.

Copy link
Member

@alextp alextp commented Nov 27, 2018

@rmlarsen I see that the memory optimizer adds temporaryvariable. This is not safe to do inside a while loop because the temporaryvariable op has weird issues with the resource manager.

@aidanw95

This comment has been minimized.

Copy link
Author

@aidanw95 aidanw95 commented Nov 28, 2018

ArithmeticOptimizer comes from running GradientDescentOptimizer.minimize as well. I'm running train in a few nested python for loops. I figured that's super typical, but should I be making a list of inputs and feeding them in with one call instead?

If anyone's interested you can see exactly what I'm running here. I already uploaded the network on a different account. https://github.com/awoods12/Sketch_It

@yangyangxcf

This comment has been minimized.

Copy link

@yangyangxcf yangyangxcf commented Dec 3, 2018

Could it be an issue about tf.map_fn rather than optimizer?
I met the same errors and the core part of my code was like

y = tf.map_fn(some_function, x)

I found it worked when x had shape [1], but got errors otherwise.
And then I found another way without map_fn and everything works well now.

@alextp

This comment has been minimized.

Copy link
Member

@alextp alextp commented Dec 3, 2018

@artsobolev

This comment has been minimized.

Copy link
Contributor

@artsobolev artsobolev commented Feb 7, 2019

I have the same problem on TF 1.12 + Ubuntu 16.04 and my custom code. Weirdly enough, even though I share the vact_scope code in different networks, the problem does not happen for all of them (It might be due to calling tf.nn.dynamic_rnn in the same scope twice with scope.reuse_variables() between the calls)

@ezhulenev, any news on the problem? Any workarounds? Maybe there's a way to disable the ArithmeticOptimizer? (or whole Grappler? What is the expected performance hit then?)

2019-02-07 17:26:05.004296: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.030579: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.031806: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.048779: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.052690: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.054059: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.056958: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.062997: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.074335: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.093691: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.106734: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.128811: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.145847: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.147453: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.164530: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.168118: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.174878: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.181530: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.242049: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.287269: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.375503: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.737421: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.737421: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.756781: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.757632: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.915909: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE

Weirdly, I see strange nested scopes (__per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE), which I do not create in the code (the GLOBAL_VARIABLES collection does not contain any variables with vact_iteration/vact_iteration in the name)

@ezhulenev

This comment has been minimized.

Copy link
Contributor

@ezhulenev ezhulenev commented Feb 7, 2019

You can disable all optimizations in arithmetic optimizer with something like:

    from google3.third_party.tensorflow.core.protobuf import rewriter_config_pb2
    
    config_proto = tf.ConfigProto()

    off = rewriter_config_pb2.RewriterConfig.OFF
    config_proto.graph_options.rewrite_options.arithmetic_optimization = off
    
    session = tf.Session(config=config_proto)

another example in

def _no_rewrite_session_config(self):
rewriter_config = rewriter_config_pb2.RewriterConfig(
dependency_optimization=rewriter_config_pb2.RewriterConfig.OFF,
pin_to_host_optimization=rewriter_config_pb2.RewriterConfig.OFF,
min_graph_nodes=-1)
graph_options = config_pb2.GraphOptions(rewrite_options=rewriter_config)
return config_pb2.ConfigProto(graph_options=graph_options)

But I suspect that the real problem is in memory optimizer, you can disable it with this option:

MemOptType memory_optimization = 4;

Can you try if disabling one of them helps.

@artsobolev

This comment has been minimized.

Copy link
Contributor

@artsobolev artsobolev commented Feb 7, 2019

Either arithmetic_optimization = off or memory_optimization = off does the job, thanks!

UPD 8/02: my problem was also caused by the model size, disregard the "calling tf.nn.dynamic_rnn in the same scope twice" in my previous comment.

@alextp alextp closed this Feb 22, 2019
@saforem2

This comment has been minimized.

Copy link

@saforem2 saforem2 commented Mar 1, 2019

@artsobolev Out of curiosity, what was the issue with the model size?

I was running into the same issue as described but was able to 'fix' it by disabling arithmetic optimizations via

    from google3.third_party.tensorflow.core.protobuf import rewriter_config_pb2
    
    config_proto = tf.ConfigProto()

    off = rewriter_config_pb2.RewriterConfig.OFF
    config_proto.graph_options.rewrite_options.arithmetic_optimization = off
    
    session = tf.Session(config=config_proto)

without having to disable memory optimization.

Also, I was wondering if anyone might have any additional insight into this issue and/or any ideas about possibly fixing it instead of just disabling arithmetic optimizations?

@ezhulenev

This comment has been minimized.

Copy link
Contributor

@ezhulenev ezhulenev commented Mar 1, 2019

Do you have minimal reproducible example? I have a few theories, but small example would help.

@artsobolev

This comment has been minimized.

Copy link
Contributor

@artsobolev artsobolev commented Mar 1, 2019

@saforem2, disabling optimizations worked for me as well, but so did decreasing the hidden layer size in the LSTM from 512 to 128. Apparently, the optimizer behaves differently in these cases.

@ezhulenev, unfortunately, I do not have such example.

@saforem2

This comment has been minimized.

Copy link

@saforem2 saforem2 commented Mar 7, 2019

@ezhulenev Unfortunately I don't have a minimal reproducible example either.

If you really think it would help, I can work on writing up a gist or walking through the steps to reproduce the error, but I think it would require cloning a (moderately sized) repository of mine.

@ezhulenev

This comment has been minimized.

Copy link
Contributor

@ezhulenev ezhulenev commented Mar 7, 2019

I'll try to work on a fix, I'll let you know when it will be in master, so you could test it.

@Sora77

This comment has been minimized.

Copy link

@Sora77 Sora77 commented Apr 11, 2019

Hello,
I am facing a similar issue but I can't seem to find this google3 library to import the off constant mentioned by the comments above.

Can anyone tell me where to get google3 ?
Thank you.

@weixsong

This comment has been minimized.

Copy link

@weixsong weixsong commented May 6, 2019

I also meet the same issue with tf.nn.conv1d, by using

    config_proto = tf.ConfigProto(log_device_placement=False, allow_soft_placement=True)
    off = rewriter_config_pb2.RewriterConfig.OFF
    config_proto.graph_options.rewrite_options.arithmetic_optimization = off

    sess = tf.Session(config=config_proto)

the issue is fixed. So i think maybe tf1.13 has some inner bug.

@abdo96

This comment has been minimized.

Copy link

@abdo96 abdo96 commented May 8, 2019

I had the same problem about import rewriter_config_pb2 from google3

from tensorflow.core.protobuf import rewriter_config_pb2

config_proto = tf.ConfigProto(log_device_placement=False, allow_soft_placement=True)

off = rewriter_config_pb2.RewriterConfig.OFF

config_proto.graph_options.rewrite_options.arithmetic_optimization = off

session = tf.Session(config=config_proto)


@alextp

This comment has been minimized.

Copy link
Member

@alextp alextp commented May 9, 2019

Just remove "google3" from the example and it should work.

@abdo96

This comment has been minimized.

Copy link

@abdo96 abdo96 commented May 9, 2019

But I had another problem I used python3.6 and I countered this problem "AlreadyExists error"

@abdo96

This comment has been minimized.

Copy link

@abdo96 abdo96 commented May 10, 2019

I tried to disable the arithmetic optimizer but I don't know where does it ? in version python3.7, TF1.13.1
I access the arithmetic optimizer.h and disabled all variables. But it hasn't effect on the error.


2019-05-10 23:23:48.280599: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.280882: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.281480: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.281866: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.282864: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.283515: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.284000: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.284436: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.284976: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.285390: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.285765: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.286219: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.286526: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.287017: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.287346: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.287845: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.288143: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.288607: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.289050: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.289331: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.289795: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.290269: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.290532: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.291102: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.292775: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.293485: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.293981: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.294523: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.295292: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.296044: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.296854: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
	 [[{{node training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Generator2.py", line 232, in <module>
    estimator_model.train(input_fn=customTrain, steps=100)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 358, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1124, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model_default
    saving_listeners)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1407, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 676, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1171, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1270, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1255, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1327, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1091, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
	 [[{{node training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}}]]
'''
@183amir

This comment has been minimized.

Copy link

@183amir 183amir commented May 12, 2019

If you are using the estimators API, try something like this:

import tensorflow as tf
from tensorflow.core.protobuf import rewriter_config_pb2
session_config = tf.ConfigProto()
off = rewriter_config_pb2.RewriterConfig.OFF
session_config.graph_options.rewrite_options.arithmetic_optimization = off
run_config = tf.estimator.RunConfig(session_config=session_config)
...
estimator = tf.estimator.Estimator(model_fn, model_dir, run_config)

Note to developers: this issue is not fixed. Theses are just workarounds as far as I can understand. Could you please re-open the issue?

@abdo96

This comment has been minimized.

Copy link

@abdo96 abdo96 commented May 12, 2019

Thank so much. I will try then If it's still existing. I will re-open issue @183amir

@sonukick

This comment has been minimized.

Copy link

@sonukick sonukick commented May 26, 2019

Perfect solution. I ran my model on non GPU machine and ended up with this error, on GPU machine it works totally fine.. Does presence/absence of GPU anything to do with arithmetic optimization?

@bignamehyp

This comment has been minimized.

Copy link
Member

@bignamehyp bignamehyp commented Jun 2, 2019

Same problem here. Workaround is not a fix.
Reopen it.

@bignamehyp bignamehyp reopened this Jun 2, 2019
@rmlarsen

This comment has been minimized.

Copy link
Member

@rmlarsen rmlarsen commented Jun 3, 2019

Thank you for the report. We will fix the corresponding rewrite in Grappler. It looks like it generates non-unique node names.

@pekaalto

This comment has been minimized.

Copy link

@pekaalto pekaalto commented Jun 14, 2019

I have this exact same issue in tf 2.0 beta0. Is there a workaround for that?

I tried:

tf.config.optimizer.set_experimental_options({'arithmetic_optimization': False})

and

tf.config.optimizer.set_experimental_options({'arithmetic_optimization': rewriter_config_pb2.RewriterConfig.OFF})

but it didn't help.

E: For me issue happens only with AdamOptimizer. RMSProp is fine

@timakro

This comment has been minimized.

Copy link

@timakro timakro commented Jun 19, 2019

I'm having the same issue and I can't get the workaround to work. Do I do something with the tf.Session instance after creating it or will it automatically be used?

For now I just try different sequence lengths in my model, some work, some don't.

EDIT: Figured it out: To make keras use the model call tensorflow.keras.backend.set_session with the model instance.

@sivakarthikgade

This comment has been minimized.

Copy link

@sivakarthikgade sivakarthikgade commented Jul 14, 2019

tensorflow.keras.backend.set_session worked. using TF 1.13

@Rassibassi

This comment has been minimized.

Copy link

@Rassibassi Rassibassi commented Aug 9, 2019

For me, either reducing the graph size or memory_optimization = off helped. TF 1.14

As follows:

from tensorflow.core.protobuf import rewriter_config_pb2

config_proto = tf.ConfigProto()
off = rewriter_config_pb2.RewriterConfig.OFFrite_options.arithmetic_optimization = off
config_proto.graph_options.rewrite_options.memory_optimization  = off
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.