Object Detection API with TensorFlow 2 Not Training On GPU or Recognizing it

[hadikoub]

Prerequisites

Please answer the following question for yourself before submitting an issue.

• I checked to make sure that this issue has not been filed already.

1. The entire URL of the documentation with the issue

https://github.com/tensorflow/models/tree/master/research/object_detection/g3doc/tf2.md

2. Describe the issue

I tried to build the docker image of the tf2 object detection API on my system running ubuntu 16.04 and
the output of

nvidia-smi

Tue Feb  9 21:50:50 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.118.02   Driver Version: 440.118.02   CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:03:00.0 Off |                  N/A |
|  0%   48C    P8    13W / 280W |    145MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:04:00.0 Off |                  N/A |
| 49%   53C    P8    15W / 280W |    145MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 108...  Off  | 00000000:05:00.0 Off |                  N/A |
| 46%   51C    P8    13W / 280W |    145MiB / 11176MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      4978      C   python3                                      135MiB |
|    1      4978      C   python3                                      135MiB |
|    2      4978      C   python3                                      135MiB |
+-----------------------------------------------------------------------------+

when I run the docker image
docker build -f research/object_detection/dockerfiles/tf2/Dockerfile -t od .
docker run --gpus all -it od
and then I tried to check if the TensorFlow is using GPU with python3

import tensorflow as tf 
assert tf.test.is_gpu_available()

I get this error
AssertionError
and when I tried
tf.config.list_physical_devices('GPU')
I got
[]

So apparently the od tf2 is not using the gpu and I don't know the error

when I tried to run the tensorflow/tensorflow:2.2.0-gpu docker image and tried the same python code It is reading the gpu and using it

[hadikoub]

Update:

after trying to
execute this command
python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
I'm getting this response

2021-02-10 13:50:23.999177: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2021-02-10 13:50:23.999224: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/__init__.py", line 436, in <module>
    _ll.load_library(_main_dir)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 153, in load_library
    py_tf.TF_LoadLibrary(lib)
tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.6/dist-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so: undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb

[AdityaNikhil]

Update:

after trying to
execute this command
python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
I'm getting this response

2021-02-10 13:50:23.999177: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2021-02-10 13:50:23.999224: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/__init__.py", line 436, in <module>
    _ll.load_library(_main_dir)
  File "/home/tensorflow/.local/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 153, in load_library
    py_tf.TF_LoadLibrary(lib)
tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.6/dist-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so: undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb

I see that the TensorFlow is clearly not loading the dynamic library libcudart.so.11.0. Please check if you configured GPU support for TensorFlow properly in your device.

[phiresky]

I had the same issue with

tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.6/dist-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so: undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb

seems to be caused by this repo actually depending on tensorflow 2.4.1 but the dockerfile building with 2.2.0, and fixed with

diff --git a/research/object_detection/dockerfiles/tf2/Dockerfile b/research/object_detection/dockerfiles/tf2/Dockerfile
index c4dfc6b2..9f67ffe0 100644
--- a/research/object_detection/dockerfiles/tf2/Dockerfile
+++ b/research/object_detection/dockerfiles/tf2/Dockerfile
@@ -1,4 +1,4 @@
-FROM tensorflow/tensorflow:2.2.0-gpu
+FROM tensorflow/tensorflow:2.4.1-gpu

[AdityaNikhil]

@phiresky Yeah probably. As per this list, the TensorFlow 2.4.1 supports Cuda 11.0. So upgrading the CUDA and CUdnn as per the list should fix this issue. (If you're thinking to use 2.4.0 version)

[hadikoub]

Yes, I think that's the issue is that TensorFlow version provided with the docker is incompatible with the requirements of TF2 Object detection.
When I tried to install the TensorFlow 2.3 pip install -U tensorflow==2.3.0 the issue is solved.
These are similar threads of the issue I found on Github
tensorflow/text#385
tensorflow/tensorflow#44255