Throws error when running with rnn_dropout_embedding: true parameter

[rajatbhatnagar94]

When I try to give dropout options for in rnn model, it doesn't run and throws the following error.

Traceback (most recent call last):
  File "/projects/rabh7066/software/anaconda/envs/nematus_5/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 854, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/projects/rabh7066/software/anaconda/envs/nematus_5/lib/python3.7/site-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper 
    raise e.ag_error_metadata.to_exception(e)
tensorflow.python.framework.errors_impl.OperatorNotAllowedInGraphError: in converted code:
    relative to /projects/rabh7066/software/anaconda/envs/nematus_5/lib/python3.7/site-packages/tensorflow_core/python:

    layers/core.py:226 call
        return super(Dropout, self).call(inputs, training=training)
    keras/layers/core.py:166 call
        lambda: array_ops.identity(inputs))
    keras/utils/tf_utils.py:59 smart_cond
        pred, true_fn=true_fn, false_fn=false_fn, name=name)
    framework/smart_cond.py:59 smart_cond
        name=name)
    util/deprecation.py:507 new_func
        return func(*args, **kwargs)
    ops/control_flow_ops.py:1224 cond
        orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
    ops/control_flow_ops.py:1061 BuildCondBranch
        original_result = fn()
    keras/layers/core.py:160 dropped_inputs
        noise_shape=self._get_noise_shape(inputs),
    keras/layers/core.py:149 _get_noise_shape
        for i, value in enumerate(self.noise_shape):
    framework/ops.py:547 __iter__
        self._disallow_iteration()
    framework/ops.py:543 _disallow_iteration
        self._disallow_in_graph_mode("iterating over `tf.Tensor`")
    framework/ops.py:523 _disallow_in_graph_mode
        " this function with @tf.function.".format(task))

    OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "prerequisite_scripts/nematus/nematus/train.py", line 522, in <module>
    train(config, sess)
  File "prerequisite_scripts/nematus/nematus/train.py", line 134, in train
    model = rnn_model.RNNModel(config)
  File "/projects/rabh7066/repos/low-resource-mt-survey/prerequisite_scripts/nematus/nematus/rnn_model.py", line 70, in __init__
    dropout_embedding, dropout_hidden)
  File "/projects/rabh7066/repos/low-resource-mt-survey/prerequisite_scripts/nematus/nematus/rnn_model.py", line 418, in __init__
    first_residual_output=1)
  File "/projects/rabh7066/repos/low-resource-mt-survey/prerequisite_scripts/nematus/nematus/layers.py", line 439, in __init__
    transition_depth=transition_depth))
  File "/projects/rabh7066/repos/low-resource-mt-survey/prerequisite_scripts/nematus/nematus/layers.py", line 380, in __init__
    dropout_state=dropout_state)
  File "/projects/rabh7066/repos/low-resource-mt-survey/prerequisite_scripts/nematus/nematus/layers.py", line 239, in __init__
    self.dropout_mask_input_to_gates = dropout_input(ones)
  File "/projects/rabh7066/repos/low-resource-mt-survey/prerequisite_scripts/nematus/nematus/rnn_model.py", line 59, in dropout_embedding
    training=self.inputs.training)
  File "/projects/rabh7066/software/anaconda/envs/nematus_5/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/projects/rabh7066/software/anaconda/envs/nematus_5/lib/python3.7/site-packages/tensorflow_core/python/layers/core.py", line 271, in dropout
    return layer.apply(inputs, training=training)
  File "/projects/rabh7066/software/anaconda/envs/nematus_5/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/projects/rabh7066/software/anaconda/envs/nematus_5/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 1700, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "/projects/rabh7066/software/anaconda/envs/nematus_5/lib/python3.7/site-packages/tensorflow_core/python/layers/base.py", line 548, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "/projects/rabh7066/software/anaconda/envs/nematus_5/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 861, in __call__
    str(e) + '\n"""')
TypeError: You are attempting to use Python control flow in a layer that was not declared to be dynamic. Pass `dynamic=True` to the class constructor.
Encountered error:
"""
in converted code:
    relative to /projects/rabh7066/software/anaconda/envs/nematus_5/lib/python3.7/site-packages/tensorflow_core/python:

    layers/core.py:226 call
        return super(Dropout, self).call(inputs, training=training)
    keras/layers/core.py:166 call
        lambda: array_ops.identity(inputs))
    keras/utils/tf_utils.py:59 smart_cond
        pred, true_fn=true_fn, false_fn=false_fn, name=name)
    framework/smart_cond.py:59 smart_cond
        name=name)
    util/deprecation.py:507 new_func
        return func(*args, **kwargs)
    ops/control_flow_ops.py:1224 cond
        orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
    ops/control_flow_ops.py:1061 BuildCondBranch
        original_result = fn()
    keras/layers/core.py:160 dropped_inputs
        noise_shape=self._get_noise_shape(inputs),
    keras/layers/core.py:149 _get_noise_shape
        for i, value in enumerate(self.noise_shape):
    framework/ops.py:547 __iter__
        self._disallow_iteration()
    framework/ops.py:543 _disallow_iteration
        self._disallow_in_graph_mode("iterating over `tf.Tensor`")
    framework/ops.py:523 _disallow_in_graph_mode
        " this function with @tf.function.".format(task))

    OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.

My tensorflow version is 1.15
python - 3.7.6

I think the error is in line 57, in the following code -

return tf.layers.dropout(e, noise_shape=tf.shape(e),
                                         rate=config.rnn_dropout_embedding,
                                         training=self.inputs.training)

It is because of the noise_shape. If I remove it the code runs fine.

Please fix this ASAP. I have to train a model urgently with those parameters.

[ibakarr]

Hi @rajatbhatnagar94 i'm experiencing the same issue. Were you able to find a way to use dropout?

[rajatbhatnagar94]

Hey @ibakarr
I did the following

In the dropout_hidden and dropout_embedding function, I replaced tf.shape(e) with (tf.shape(e)[0], tf.shape(e)[1]). You can use https://github.com/rajatbhatnagar94/nematus/blob/master/nematus/rnn_model.py#L57 for reference

You can make these changes in the nematus code or use my fork of nematus! https://github.com/rajatbhatnagar94/nematus/

Thanks!

Hope it helps

[ibakarr]

Yeah i used your fork and works well. Thanks!

[rsennrich]

thanks for reporting this; I fixed this in the Tensorflow 2 branch, which has now also been merged into master.

[sven1977]

Awesome! I had this error in a different repo, but also related to tf.shape.

For me, it helped replacing something like this (trying to fold time dim back into batch dim [B, T, ...] -> [B+T, ...]):

[-1] + list(tf.shape(model_out)[2:])

with

tf.concat([[-1], tf.shape(model_out)[2:]], axis=0))