UnknownError in eager_few_shot_od_training_tf2_colab.ipynb

[yumemio]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/blob/master/research/object_detection/colab_tutorials/eager_few_shot_od_training_tf2_colab.ipynb

2. Describe the bug

Executing the notebook in Colaboratory (with a GPU accelerator) raises UnknownError at detection_model.predict(image, shapes):

---------------------------------------------------------------------------

UnknownError                              Traceback (most recent call last)

<ipython-input-15-5906ad4ff27f> in <module>()
     37 # Run model through a dummy image so that variables are created
     38 image, shapes = detection_model.preprocess(tf.zeros([1, 640, 640, 3]))
---> 39 prediction_dict = detection_model.predict(image, shapes)
     40 _ = detection_model.postprocess(prediction_dict, shapes)
     41 print('Weights restored!')

3 frames

/usr/local/lib/python3.7/dist-packages/object_detection/meta_architectures/ssd_meta_arch.py in predict(self, preprocessed_inputs, true_image_shapes)
    568       batchnorm_updates_collections = tf.GraphKeys.UPDATE_OPS
    569     if self._feature_extractor.is_keras_model:
--> 570       feature_maps = self._feature_extractor(preprocessed_inputs)
    571     else:
    572       with slim.arg_scope([slim.batch_norm],

/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
     65     except Exception as e:  # pylint: disable=broad-except
     66       filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67       raise e.with_traceback(filtered_tb) from None
     68     finally:
     69       del filtered_tb

/usr/local/lib/python3.7/dist-packages/object_detection/meta_architectures/ssd_meta_arch.py in call(self, inputs, **kwargs)
    249   # method.
    250   def call(self, inputs, **kwargs):
--> 251     return self._extract_features(inputs)
    252 
    253 

/usr/local/lib/python3.7/dist-packages/object_detection/models/ssd_resnet_v1_fpn_keras_feature_extractor.py in _extract_features(self, preprocessed_inputs)
    223 
    224     image_features = self.classification_backbone(
--> 225         ops.pad_to_multiple(preprocessed_inputs, self._pad_to_multiple))
    226 
    227     feature_block_list = []

UnknownError: Exception encountered when calling layer "conv1_conv" (type Conv2D).

Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [Op:Conv2D]

Call arguments received:
  • inputs=tf.Tensor(shape=(1, 646, 646, 3), dtype=float32)

3. Steps to reproduce

1. Open the notebook in Colab.
2. (optional) Uncomment the ready-made gt_boxes (8th code cell) and comment out colab_utils.annotate() (7th).
3. Execute all cells.
4. The above error is raised.

4. Expected behavior

The error should not appear.

5. Additional context

Gist: https://gist.github.com/yumemio/9924c11121fa9d2e4ca32c3ab0070987

Looks like a shape mismatch (image has shape 640x640x3 vs. the model expects 646x646x3), but resizing the image to 646x646x3 with tf.keras.layers.Resizing beforehand did not help (the input tensor shape changed to 652x652x3).

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Colaboratory (Linux Ubuntu 18.04.5)
• Mobile device name if the issue happens on a mobile device:
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 2.7.0
• Python version: 3.7.12
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: CUDA 11.2 / cuDNN 7.6.5
• GPU model and memory: Tesla K80 / 12GB

[kumariko]

@yumemio Could you please have a look on the link1 , link2 and let us know if it helps? Thanks!

[yumemio]

As per @kumariko's suggestion, I re-ran the code without !pip install -U --pre tensorflow=="2.2.0" (the first cell), and voilà, all went well¹. Turned out it was a problem with TF version.
I am filing a PR to fix this.

@kumariko, thanks for the reply, you helped me a lot!

Footnotes

1. Gist: https://gist.github.com/yumemio/389b7d3fa47ebae24e88080195486458 . Note that the last cell is not executed because the output is too huge (in terms of file size; it will plot a video clip). ↩

[kumariko]

@yumemio Thanks for the update. If it is resolved then please feel free to move this issue to close status ?Thanks!

[Yokanater]

I am facing the same error for the same notebook. I removed the !pip install -U --pre tensorflow=="2.2.0 which worked for @yumemio but the error is still being thrown

[yumemio]

@Endless-Debugger My gist was run on a Colab VM, which has TensorFlow==2.7.0 preinstalled.
Are you running the notebook in Colab? If not, installing the latest version of TensorFlow might help.