Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AggregationMethod.EXPERIMENTAL_ACCUMULATE_N gives Already exists error. #3220

Closed
chasep255 opened this issue Jul 7, 2016 · 2 comments
Closed

AggregationMethod.EXPERIMENTAL_ACCUMULATE_N gives Already exists error. #3220

chasep255 opened this issue Jul 7, 2016 · 2 comments

Comments

@chasep255
Copy link

I have used EXPERIMENTAL_ACCUMULATE_N in all my past tensorflow projects since it lets me use larger batch sizes. I have never had any issues with it until now. Now I get the following error when I run the train step.

W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_4/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_2/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_3/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
     [[Node: gradients/AccumulateN_5/TemporaryVariable = TemporaryVariable[dtype=DT_FLOAT, shape=[2,1024], var_name="", _device="/job:localhost/replica:0/task:0/gpu:0"](^gradients/Sub_1)]]
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
     [[Node: gradients/AccumulateN_5/TemporaryVariable = TemporaryVariable[dtype=DT_FLOAT, shape=[2,1024], var_name="", _device="/job:localhost/replica:0/task:0/gpu:0"](^gradients/Sub_1)]]
W tensorflow/core/framework/op_kernel.cc:936] Already exists: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
     [[Node: gradients/AccumulateN_5/TemporaryVariable = TemporaryVariable[dtype=DT_FLOAT, shape=[2,1024], var_name="", _device="/job:localhost/replica:0/task:0/gpu:0"](^gradients/Sub_1)]]
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 14880 get requests, put_count=3563 evicted_count=1000 eviction_rate=0.280662 and unsatisfied allocation rate=0.834476
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 100 to 110
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=10010 evicted_count=10000 eviction_rate=0.999001 and unsatisfied allocation rate=0
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 730, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 712, in _run_fn
    status, run_metadata)
  File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors.py", line 450, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors.AlreadyExistsError: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
     [[Node: gradients/AccumulateN_5/TemporaryVariable = TemporaryVariable[dtype=DT_FLOAT, shape=[2,1024], var_name="", _device="/job:localhost/replica:0/task:0/gpu:0"](^gradients/Sub_1)]]
     [[Node: Mean/_21 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_3391_Mean", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/chase/workspace/Translator/translator.py", line 106, in <module>
    c, _ = sess.run([cost, train_step], feed_dict = {input_tensor: x, expected_output: y})
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 382, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 655, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 723, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 743, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.AlreadyExistsError: Resource tmp_var/gradients/AccumulateN_5/TemporaryVariable/N10tensorflow19TemporaryVariableOp6TmpVarE
     [[Node: gradients/AccumulateN_5/TemporaryVariable = TemporaryVariable[dtype=DT_FLOAT, shape=[2,1024], var_name="", _device="/job:localhost/replica:0/task:0/gpu:0"](^gradients/Sub_1)]]
     [[Node: Mean/_21 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_3391_Mean", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Caused by op 'gradients/AccumulateN_5/TemporaryVariable', defined at:
  File "/home/chase/workspace/Translator/translator.py", line 79, in <module>
    train_step = tf.train.AdamOptimizer(1e-4).minimize(cost, aggregation_method = tf.AggregationMethod.EXPERIMENTAL_ACCUMULATE_N)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/optimizer.py", line 193, in minimize
    grad_loss=grad_loss)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/optimizer.py", line 250, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients.py", line 432, in gradients
    out_grads = _AggregatedGrads(grads, op, loop_state, aggregation_method)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients.py", line 692, in _AggregatedGrads
    out_grads[i] = math_ops.accumulate_n(out_grad)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py", line 1497, in accumulate_n
    var = gen_state_ops._temporary_variable(shape=shape, dtype=tensor_dtype)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 365, in _temporary_variable
    var_name=var_name, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2297, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1231, in __init__
    self._traceback = _extract_stack()

This error goes away if I change the aggregation method of ADD_N. Here is my code. I am trying to implement a sequence to sequence model to translate english to german.

import unidecode
import tensorflow as tf
rnn = tf.nn.rnn_cell
import numpy as np
import string
import sys
import random
import multiprocessing

TIME_STEPS = 128
BATCH_SIZE = 2

def ctov(c):
    v = np.zeros(128)
    v[ord(c)] = 1
    return v

def vtoc(v):
    return chr(np.argmax(v))

def stov(s, vlen):
    null_vec = ctov('\0')
    v = np.tile(null_vec, (vlen, 1))
    if len(s) > vlen: s = s[:vlen]
    for i, c in enumerate(s):
        v[i] = ctov(c)
    return v

def vtos(v):
    s = ''
    for cv in v:
        c = vtoc(cv)
        if c == '\0':
            break
        s += c
    return s

def clean_text(txt):
    txt = unidecode.unidecode(txt)
    txt = ''.join([c for c in txt if c in CHARS])
    return txt

def load_training_data():
    print('Loading English...')
    en = [e.strip() for e in open('/home/chase/Desktop/de-en/english.txt')]
    print('Loading German...')
    de = [d.strip() for d in open('/home/chase/Desktop/de-en/german.txt')]
    print('Processing...')
    translations = [(unidecode.unidecode(e), unidecode.unidecode(d)) for e, d in zip(en, de) if len(e) and len(d) and len(e) <= TIME_STEPS and len(d) <= TIME_STEPS]
    return translations


input_tensor = tf.placeholder(tf.float32, (BATCH_SIZE, TIME_STEPS, 128), 'input_tensor')
expected_output = tf.placeholder(tf.float32, (BATCH_SIZE, TIME_STEPS, 128), 'expected_output')

y = input_tensor
with tf.variable_scope('encoder'):
    rnn_cell = rnn.MultiRNNCell([rnn.GRUCell(1024) for _ in range(3)])
    y = tf.nn.dynamic_rnn(rnn_cell, y, dtype = tf.float32)[0]

with tf.variable_scope('attn'):
    W = tf.get_variable('W_attn', shape = (1, TIME_STEPS, TIME_STEPS), initializer = tf.truncated_normal_initializer(0.0, 1 / np.sqrt(TIME_STEPS)))
    W = tf.tile(W, (BATCH_SIZE, 1, 1))
    y = tf.batch_matmul(W, y)

with tf.variable_scope('decoder'):
    rnn_cell = rnn.MultiRNNCell([rnn.GRUCell(1024) for _ in range(3)])
    y = tf.nn.dynamic_rnn(rnn_cell, y, dtype = tf.float32)[0]

with tf.variable_scope('output'):
    y = tf.split(1, TIME_STEPS, y)
    y = [tf.reshape(x, (BATCH_SIZE, 1024)) for x in y]
    W = tf.get_variable('W', shape = (1024, 128), initializer = tf.truncated_normal_initializer(0.0, 1 / np.sqrt(1024)))
    b = tf.get_variable('b', shape = (128,), initializer = tf.truncated_normal_initializer(0.0, 0.01))
    y = [tf.nn.softmax(tf.matmul(x, W) + b) for x in y]
    output = tf.pack(y, 1)

cost = tf.reduce_mean(-tf.reduce_sum(expected_output * tf.log(output), reduction_indices = 2))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cost, aggregation_method = tf.AggregationMethod.EXPERIMENTAL_ACCUMULATE_N)

config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.8
sess = tf.Session(config = config)
sess.run(tf.initialize_all_variables())

if 'train' in sys.argv or True:
    translations = load_training_data()
    validation = translations[:10000]
    translations = translations[10000:]

    def training_data_generator(data):
        x = np.zeros((BATCH_SIZE, TIME_STEPS, 128))
        y = np.zeros_like(x)
        random.shuffle(data)
        for i in range(0, len(data) - BATCH_SIZE, BATCH_SIZE):        
            for b in range(BATCH_SIZE):
                e, d = data[i + b]
                x[b] = stov(e, TIME_STEPS)
                y[b] = stov(d, TIME_STEPS)

            yield (x, y)

    for e in range(50):
        for b, v in enumerate(training_data_generator(translations)):
            x, y = v
            c, _ = sess.run([cost, train_step], feed_dict = {input_tensor: x, expected_output: y})
            print('Epoch %-6d  Batch %-6d  Cost %-6e' % (e, b, c), end = '\r')

I am using Python 3.5 on Ubuntu 16.04. I have the latest tensorflow version 0.9.
@chasep255
Copy link
Author

When I replaced the dynamic_rnn in the encoder scope with something I unrolled my self the issue goes away.

with tf.variable_scope('encoder') as scope:
    rnn_cell = rnn.MultiRNNCell([rnn.GRUCell(1024) for _ in range(3)])
    state = tf.zeros((BATCH_SIZE, rnn_cell.state_size))
    output = [None] * TIME_STEPS
    for t in range(TIME_STEPS):
        y_t = tf.reshape(y[:, t, :], (BATCH_SIZE, -1))
        output[t], state = rnn_cell(y_t, state)
        scope.reuse_variables()
    y = tf.pack(output, 1)

@aselle
Copy link
Contributor

aselle commented Jul 9, 2016

It is a known issue with using EXPERIMENTAL_ACCUMULATE_N with dynamic_rnn. But it really shouldn't be needed for dynamic_rnn's already accumulates internally. @yuanbyu or @ebrevdo can elaborate. Closing for now since it is an experimental feature that is technically unsupported.

@aselle aselle closed this as completed Jul 9, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aselle @chasep255