ValueError: Attempt to reuse RNNCell with a different variable scope than its first use.

[doncat99]

I am not sure if I am the first who met the following error:

ValueError: Attempt to reuse RNNCell <tensorflow.contrib.rnn.python.ops.core_rnn_cell_impl.BasicLSTMCell object at 0x10210d5c0> with a different variable scope than its first use. First use of cell was with scope 'rnn/multi_rnn_cell/cell_0/basic_lstm_cell', this attempt is with scope 'rnn/multi_rnn_cell/cell_1/basic_lstm_cell'. Please create a new instance of the cell if you would like it to use a different set of weights. If before you were using: MultiRNNCell([BasicLSTMCell(...)] * num_layers), change to: MultiRNNCell([BasicLSTMCell(...) for _ in range(num_layers)]). If before you were using the same cell instance as both the forward and reverse cell of a bidirectional RNN, simply create two instances (one for forward, one for reverse). In May 2017, we will start transitioning this cell's behavior to use existing stored weights, if any, when it is called with scope=None (which can lead to silent model degradation, so this error will remain until then.)

with the code fragment:

  import tensorflow as tf
  from tensorflow.contrib import rnn

  hidden_size = 100
  batch_size  = 100
  num_steps   = 100
  num_layers  = 100
  is_training = True
  keep_prob   = 0.4

  input_data = tf.placeholder(tf.float32, [batch_size, num_steps])
  lstm_cell = rnn.BasicLSTMCell(hidden_size, forget_bias=0.0, state_is_tuple=True)

  if is_training and keep_prob < 1:
      lstm_cell = rnn.DropoutWrapper(lstm_cell)
  cell = rnn.MultiRNNCell([lstm_cell for _ in range(num_layers)], state_is_tuple=True)

  _initial_state = cell.zero_state(batch_size, tf.float32)

  iw = tf.get_variable("input_w", [1, hidden_size])
  ib = tf.get_variable("input_b", [hidden_size])
  inputs = [tf.nn.xw_plus_b(i_, iw, ib) for i_ in tf.split(input_data, num_steps, 1)]

  if is_training and keep_prob < 1:
      inputs = [tf.nn.dropout(input_, keep_prob) for input_ in inputs]

  outputs, states = rnn.static_rnn(cell, inputs, initial_state=_initial_state)

I had googled around with no luck, can anyone show me a way out?

I am getting the same error when trying to run the translate example (even when doing the small self test) which can be found here: https://github.com/tensorflow/models/tree/master/tutorials/rnn/translate

[tongda]

I met the same issue. If you are all using compiled version on master branch, I believe that we are the same issue caused by the recent commit. As the commit message says:

Make all RNNCells in tf.contrib.rnn act like tf.layers Layers, but with stricter semantics for no
w:

1. Upon first use of call, the used scope is stored in the cell. The RNNCell tries to create weights in that scope but if some are already set, an error is raised unless the RNNCell was constructed with argument reuse=True.

2. A subsequent use of call of the same cell instance must be in the same scope.
   If it is not, an error is raised.

From my case, which is running the ptb tutorial, the solution is just to add a parameter named with reuse like this at line 112:

def lstm_cell():
  return tf.contrib.rnn.BasicLSTMCell(
      size, forget_bias=0.0, state_is_tuple=True, reuse=tf.get_variable_scope().reuse)

Then it works.

[prb12]

@ebrevdo Could you please take a look at this?

[tomwanzek]

The issue replicates for me when using the Windows/GPU build 105 on the Shakespeare RNN Repo.

When running the code with the Win 1.0.0/GPU Release, there is no issue.

[ebrevdo]

That repo looks like it's targeted at tf 1.0, not intermediate releases.

…

On Mar 8, 2017 3:56 PM, "Tom Wanzek" ***@***.***> wrote: The issue replicates for me when using the Windows/GPU build 105 on the Shakespeare RNN Repo <https://github.com/martin-gorner/tensorflow-rnn-shakespeare>. When running the code with the Win 1.0.0/GPU Release, there is no issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8191 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABtim5ansaL1KN51T4nCaqLnqw2QHN4Wks5rj0BBgaJpZM4MWl4f> .

[doncat99]

@tongda , I am using the Release Version of Tensorflow 1.0, working on MacOS in cpu mode. I will switch to the master branch to see if it work by adding the "reuse" parameter, thanks.

[ebrevdo]

doncat99: if you do, please ensure your code queries the tensorflow version and raises a flag if the version is lower than the master branch version. you may need to check against: from tensorflow.core import versions versions.GIT_VERSION

…

On Wed, Mar 8, 2017 at 6:58 PM, doncat99 ***@***.***> wrote: @tongda <https://github.com/tongda> , I am using the Release Version of Tensorflow 1.0, working on MacOS in cpu mode. I will switch to the master branch to see if it work by adding the "reuse" parameter, thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8191 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABtim66cU9e16lgD-2D0QLGcQCiHbV0zks5rj2rbgaJpZM4MWl4f> .

[tomwanzek]

@ebrevdo So what would be the suggested changes to the Shakepeare RNN to allow it to work with the intermediate stable release?

Here is the key architectural section of the code, which now fails with build#105:

#
# the model (see FAQ in README.md)
#
lr = tf.placeholder(tf.float32, name='lr')  # learning rate
pkeep = tf.placeholder(tf.float32, name='pkeep')  # dropout parameter
batchsize = tf.placeholder(tf.int32, name='batchsize')

# inputs
X = tf.placeholder(tf.uint8, [None, None], name='X')    # [ BATCHSIZE, SEQLEN ]
Xo = tf.one_hot(X, ALPHASIZE, 1.0, 0.0)                 # [ BATCHSIZE, SEQLEN, ALPHASIZE ]
# expected outputs = same sequence shifted by 1 since we are trying to predict the next character
Y_ = tf.placeholder(tf.uint8, [None, None], name='Y_')  # [ BATCHSIZE, SEQLEN ]
Yo_ = tf.one_hot(Y_, ALPHASIZE, 1.0, 0.0)               # [ BATCHSIZE, SEQLEN, ALPHASIZE ]
# input state
Hin = tf.placeholder(tf.float32, [None, INTERNALSIZE*NLAYERS], name='Hin')  # [ BATCHSIZE, INTERNALSIZE * NLAYERS]

# using a NLAYERS=3 layers of GRU cells, unrolled SEQLEN=30 times
# dynamic_rnn infers SEQLEN from the size of the inputs Xo

onecell = rnn.GRUCell(INTERNALSIZE)
dropcell = rnn.DropoutWrapper(onecell, input_keep_prob=pkeep)
multicell = rnn.MultiRNNCell([dropcell for _ in range(NLAYERS)], state_is_tuple=False)
multicell = rnn.DropoutWrapper(multicell, output_keep_prob=pkeep)
Yr, H = tf.nn.dynamic_rnn(multicell, Xo, dtype=tf.float32, initial_state=Hin)
# Yr: [ BATCHSIZE, SEQLEN, INTERNALSIZE ]
# H:  [ BATCHSIZE, INTERNALSIZE*NLAYERS ] # this is the last state in the sequence

I do not seem to find any documentation regarding a reuse flag?

Thanks in advance.

[ebrevdo]

Use: multicell = rnn.MultiRNNCell([rnn.DropoutWrapper(rnn.GRUCell(INTERNALSIZE), input_keep_prob=pkeep) for _ in range(NLAYERS)], state_is_tuple=False) Which creates a separate grucell object for each layer.

…

On Mar 10, 2017 7:44 AM, "Tom Wanzek" ***@***.***> wrote: @ebrevdo <https://github.com/ebrevdo> So what would be the suggested changes to the Shakepeare RNN to allow it to work with the intermediate stable release? Here is the key architectural section of the code, which now fails with build#105: ## the model (see FAQ in README.md)# lr = tf.placeholder(tf.float32, name='lr') # learning rate pkeep = tf.placeholder(tf.float32, name='pkeep') # dropout parameter batchsize = tf.placeholder(tf.int32, name='batchsize') # inputs X = tf.placeholder(tf.uint8, [None, None], name='X') # [ BATCHSIZE, SEQLEN ] Xo = tf.one_hot(X, ALPHASIZE, 1.0, 0.0) # [ BATCHSIZE, SEQLEN, ALPHASIZE ]# expected outputs = same sequence shifted by 1 since we are trying to predict the next character Y_ = tf.placeholder(tf.uint8, [None, None], name='Y_') # [ BATCHSIZE, SEQLEN ] Yo_ = tf.one_hot(Y_, ALPHASIZE, 1.0, 0.0) # [ BATCHSIZE, SEQLEN, ALPHASIZE ]# input state Hin = tf.placeholder(tf.float32, [None, INTERNALSIZE*NLAYERS], name='Hin') # [ BATCHSIZE, INTERNALSIZE * NLAYERS] # using a NLAYERS=3 layers of GRU cells, unrolled SEQLEN=30 times# dynamic_rnn infers SEQLEN from the size of the inputs Xo onecell = rnn.GRUCell(INTERNALSIZE) dropcell = rnn.DropoutWrapper(onecell, input_keep_prob=pkeep) multicell = rnn.MultiRNNCell([dropcell for _ in range(NLAYERS)], state_is_tuple=False) multicell = rnn.DropoutWrapper(multicell, output_keep_prob=pkeep) Yr, H = tf.nn.dynamic_rnn(multicell, Xo, dtype=tf.float32, initial_state=Hin)# Yr: [ BATCHSIZE, SEQLEN, INTERNALSIZE ]# H: [ BATCHSIZE, INTERNALSIZE*NLAYERS ] # this is the last state in the sequence I do not seem to find any documentation regarding a reuse flag? Thanks in advance. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8191 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABtim6MOOCbx3RJEJe8PQBDXGVIXTGPmks5rkW_jgaJpZM4MWl4f> .

[BSVogler]

I don't understand why I am getting this error with the seq2seq tutorial model:

cell = tf.contrib.rnn.MultiRNNCell([single_cell() for _ in range(num_layers)])

Source

where the cell is created with

def single_cell():
    return tf.contrib.rnn.GRUCell(size)

[tomwanzek]

@ebrevdo Thanks for getting back to this issue. Unfortunately, the suggested change leaves matters as they are, with the aforementioned error. Given the above comment regarding the seq2seq tutorial, I suspect we are all in the same boat?

[ebrevdo]

Are you sure it's the exact same error? Please copy and paste it here.

[tomwanzek]

My bad, I just went through the change process to the relevant code again (from scratch) and re-ran it as proposed. The error has indeed been removed and the Old Bard is hallucinating just fine now 👍

So, thx, not sure where I went wrong yesterday, but it was clearly on me.