Repeated "Already exists: Resource" errors when I increase my model size

[aidanw95]

System information

• Custom model built by creating convolutional network tf graph with SGD optimizer and running
• Windows 10
• TensorFlow installed from pip
• TensorFlow version (use command below): 1.12
• Python version: 3.6
• Running on cpu

I've created a convolutional network by making a custom graph for feed forward and using a tensorflow optimizer to train it (I've tried with SGD and Adam).

`sess = tf.Session()
optimizer = tf.train.GradientDescentOptimizer(learning_rate=.01)

input_size = 16
input_matrix = tf.placeholder(tf.float32, shape=[1,input_size,input_size])
label_ph = tf.placeholder(tf.float32, shape=[28*28])
model_out = Model.feed_forward(input_matrix)

init = tf.global_variables_initializer()
sess.run(init)

loss = tf.losses.mean_squared_error([tf.convert_to_tensor(label_ph, tf.float32)], [model_out])
train = optimizer.minimize(loss)
sess.graph.finalize()`

This is my graph definition. I know Model.feed_forward is opaque but it's basically a series of functions that get variable weights and feed input through layers. Just to give you an idea here's an example of a convolutional feed forward:

`def feed_input(input_tensors, filters, biases, stride, zero_pad=True):
        if zero_pad:
            pad_size = int((int(filters[0].shape[0]) + 1)/2 - 1)
            input_tensors = tf.map_fn(lambda inp:
                tf.pad(inp, [[pad_size, pad_size], [pad_size, pad_size]]), input_tensors)

        _, shape_x, _ = input_tensors.shape

        unflattened_output = tf.map_fn(lambda input_tensor: tf.convert_to_tensor([[[tf.add(tf.tensordot(curr_filter,
            ConvLayer.__get_input_slice__(input_tensor, x, y, tf.shape(curr_filter)[0]), 2), biases[filter_ind])
         for x in range(0, shape_x - curr_filter.shape[0] + 1, stride)]
         for y in range(0, shape_x - curr_filter.shape[0] + 1, stride)]
         for filter_ind, curr_filter in enumerate(filters)], dtype=tf.float32), input_tensors)

        shape_1, shape_2, shape_3, shape_4 = unflattened_output.shape

        return tf.reshape(unflattened_output, [shape_1 * shape_2, shape_3, shape_4])`

After defining my graph I am running a loop that gets some input and runs sess.run(train, feed_dict={input_matrix:img, label_ph:label_image}). If I make input_size smaller (e.g. 4) or if I reduce the number of layers in my network, everything runs fine and it does exactly what I expect it to do. However, when my network grows in the ways I just mentioned it consistently spits out these errors during sess.run(train, feed_dict={input_matrix:img, label_ph:label_image}):

2018-11-15 00:46:31.069913: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.069914: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.069914: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.069927: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.070138: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.078467: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.080434: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.081575: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.083125: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.085370: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.089281: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.090932: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.091080: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.091398: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.091916: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.092735: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.098503: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.094920: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.099036: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.101236: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.101988: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.101966: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.103264: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.102945: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.106092: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.106858: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.108494: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.109495: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.109987: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.112326: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.112525: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.113595: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.113831: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.115342: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.115418: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.120685: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.122569: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.122708: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.123904: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.124832: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.126926: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.127336: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.129575: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.128498: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.115477: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.130180: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.130601: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.131469: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.135938: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.132834: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.139111: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.139728: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.140827: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.141429: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.142202: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.140004: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.142621: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.144039: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.147757: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.145043: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.149502: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.146723: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.148971: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.145861: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.149850: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.151642: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.152947: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.154227: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.155666: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.157249: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.157877: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.159295: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.160204: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.163481: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.163750: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.164518: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.166533: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.172400: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.174768: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.174793: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
Traceback (most recent call last):
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
[[{{node gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}} = TemporaryVariabledtype=DT_FLOAT, shape=[5,5], var_name="gradients/...dd/tmp_var", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/Aidan/Documents/PycharmProjects/SketchIt/src/network/FeedImages.py", line 57, in
sess.run(train, feed_dict={input_matrix:img, label_ph:label_image})
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
run_metadata_ptr)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
run_metadata)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
[[{{node gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}} = TemporaryVariabledtype=DT_FLOAT, shape=[5,5], var_name="gradients/...dd/tmp_var", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Anyone know what might be the cause? Maybe some concurrency issue since its telling me things that shouldn't exist have already been created? Help would be much appreciated, I'm close to making this thing work! Thanks for your time.

[alextp]

What is "AddOpsRewrite_Add/tmp_var/struct" in your graph? Why are you using temporary variables? They are not compatible with while loops, which is what you're seeing in the error message.

(I thought temporary variables had been removed as part of tf's API a long time ago)

[aidanw95]

Thanks for the response.

Those are created by sess.run(train) where train is GradientDescentOptimizer.minimize. So tensorflow is generating them, not me.

Also I should update that I rolled back to tensorflow 1.8 (haven't tried 9, 10, or 11 just due to lack of time) and the error no longer comes up.

[aidanw95]

Issue close was accidental

[alextp]

What is ArithmeticOptimizer in your code? These ops don't look like they were added by TF's backward pass.

[alextp]

Hm this might come from grappler.

[alextp]

@rmlarsen I see that the memory optimizer adds temporaryvariable. This is not safe to do inside a while loop because the temporaryvariable op has weird issues with the resource manager.

[aidanw95]

ArithmeticOptimizer comes from running GradientDescentOptimizer.minimize as well. I'm running train in a few nested python for loops. I figured that's super typical, but should I be making a list of inputs and feeding them in with one call instead?

If anyone's interested you can see exactly what I'm running here. I already uploaded the network on a different account. https://github.com/awoods12/Sketch_It