No gradients provided for any variable

[arthurflor23]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Arch linux + Gnome
• TensorFlow version: tf-nightly==2.2.0.dev20200504 from pip install tf-nightly
• Python version: 3.8.2

Describe the current behavior
When running the training on the localhost the error occurs:

Epoch 1/1000
Traceback (most recent call last):
  File "main.py", line 155, in <module>
    h = model.fit(x=dtgen.next_train_batch(),
  File "/home/arthur/Code/handwritten-text-recognition/src/network/model.py", line 166, in fit
    out = self.model.fit(x=x, y=y, batch_size=batch_size, epochs=epochs, verbose=verbose,
  File "/home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 72, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 921, in fit
    tmp_logs = train_function(iterator)
  File "/home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 695, in __call__
    result = self._call(*args, **kwds)
  File "/home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 737, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 616, in _initialize
    self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
  File "/home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2902, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 3232, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 3111, in _create_graph_function
    func_graph_module.func_graph_from_py_func(
  File "/home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py", line 981, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 528, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py", line 968, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:

    /home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:631 train_function  *
        return step_function(self, iterator)
    /home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:621 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    /home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:952 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:2292 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:2651 _call_for_each_replica
        return fn(*args, **kwargs)
    /home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:614 run_step  **
        outputs = model.train_step(data)
    /home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:581 train_step
        _minimize(self.distribute_strategy, tape, self.optimizer, loss,
    /home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1946 _minimize
        gradients = optimizer._aggregate_gradients(zip(gradients,  # pylint: disable=protected-access
    /home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:554 _aggregate_gradients
        filtered_grads_and_vars = _filter_grads(grads_and_vars)
    /home/arthur/Code/handwritten-text-recognition/.venv/lib/python3.8/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:1251 _filter_grads
        raise ValueError("No gradients provided for any variable: %s." %

    ValueError: No gradients provided for any variable: ['conv2d/kernel:0', 'conv2d/bias:0', 'conv2d_1/kernel:0', 'conv2d_1/bias:0', 'gated_conv2d/kernel:0', 'gated_conv2d/bias:0', 'conv2d_2/kernel:0', 'conv2d_2/bias:0', 'gated_conv2d_1/kernel:0', 'gated_conv2d_1/bias:0', 'conv2d_3/kernel:0', 'conv2d_3/bias:0', 'gated_conv2d_2/kernel:0', 'gated_conv2d_2/bias:0', 'conv2d_4/kernel:0', 'conv2d_4/bias:0', 'bidirectional/forward_lstm/lstm_cell_1/kernel:0', 'bidirectional/forward_lstm/lstm_cell_1/recurrent_kernel:0', 'bidirectional/forward_lstm/lstm_cell_1/bias:0', 'bidirectional/backward_lstm/lstm_cell_2/kernel:0', 'bidirectional/backward_lstm/lstm_cell_2/recurrent_kernel:0', 'bidirectional/backward_lstm/lstm_cell_2/bias:0', 'dense/kernel:0', 'dense/bias:0', 'bidirectional_1/forward_lstm_1/lstm_cell_4/kernel:0', 'bidirectional_1/forward_lstm_1/lstm_cell_4/recurrent_kernel:0', 'bidirectional_1/forward_lstm_1/lstm_cell_4/bias:0', 'bidirectional_1/backward_lstm_1/lstm_cell_5/kernel:0', 'bidirectional_1/backward_lstm_1/lstm_cell_5/recurrent_kernel:0', 'bidirectional_1/backward_lstm_1/lstm_cell_5/bias:0', 'dense_1/kernel:0', 'dense_1/bias:0'].

Describe the expected behavior
This issue didn't happen in previous versions, but I'm having to use python 3.8, thus the tf-nightly version. I also noticed that in the collab the code works, with the TF 2.1 version and python 3.7.

In addition, I found issues suggesting using:

with tf.GradientTape as tape:
[...]

However, the examples I saw use a customized training function (train_step()). In my case, I only use the standard fit() function, which is enough for the context.

Maybe I'm not sure how to use this in new version, or maybe it's an issue. Anyway, if anyone can help, thank you very much.

Standalone code to reproduce the issue
Project code and model class

[arthurflor23]

Data generator error. fixed

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No