Tensorflow 2.1.0: Error occurred when finalizing GeneratorDataset iterator

[muellerdo]

Occurrence of the warning "Error occurred when finalizing GeneratorDataset iterator" when running training on Tensorflow 2.1.0.

Reason is assumed to be a memory leak in Keras.
We have to wait for the new Tensorflow update.

Warning seems to not have any influence on fitting process.
Also bug seems to be related to number of workers != 1.
Nevertheless, I recommend using only a single worker until the new Tensorflow version fixes this issue. Just to be sure.

150/150 [==============================] - 301s 2s/step - loss: 2.8790 - tversky_loss: 2.2741 - dice_soft: 0.4315 - dice_crossentropy: 1.1734 - val_loss: 17.1148 - val_tversky_loss: 3.7358 - val_dice_soft: 0.0698 - val_dice_crossentropy: 14.3242
Epoch 2/500
2020-05-09 11:52:43.955314: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
 10/150 [=>............................] - ETA: 4:37 - loss: 2.4723 - tversky_loss: 2.0354 - dice_soft: 0.4912 - dice_crossentropy: 0.9458

Sources:
tensorflow/tensorflow#35100
tensorflow/tensorflow#37515

[muellerdo]

Update: Errors only occurs during evaluation/validation

[muellerdo]

Update:
Tensorflow 2.2.0 does not support Keras Data Generators for validation, anymore.

I'm probably have to rework the complete Data Generator of MIScnn into TF datasets...
Any suggestions are welcome.

Source: https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit

[muellerdo]

TF 2.2.0 fixes Data Generators bug.

Contrary to the TF documentation, Keras Data Generators seem to be still functional with TF 2.2.0.

Either Data Generators were silently declared deprecated or it is an error in the new documentation.
TF 2.2.0 changelog also did not mention any changes in Keras Data Generator support.

So, let's rest this for a start and have an eye on it in the future.