Edge TPU lifespan resulted in Failed to load delegate

[smart-face]

Description

I have used Dev board SoM continuously for 2 years in the following two cases:
case (1) EdgeTPU lasted about 8 months for continuous object detection (without motion sensing) on the mobilenetSSD model and now I got the following error,
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "face_run.py", line 129, in
experimental_delegates=[load_delegate('libedgetpu.so.1.0')])
File "/usr/local/lib/python3.7/dist-packages/tflite_runtime/interpreter.py", line 163, in load_delegate
library, str(e)))
ValueError: Failed to load delegate from libedgetpu.so.1.0

case (2) A model inference is only performed when a motion is detected in the camera scene, and this device lasted about 1.5 years, and again I got the same error as above.
Then, I performed and device query so there are no available TPU devices.

Any luck on extending the lifespan of EdgeTPU or solving these errors?

Click to expand!

Issue Type

Performance

Operating System

Mendel Linux

Coral Device

Dev Board

Other Devices

No response

Programming Language

Python 3.7

Relevant Log Output

No response

[hjonnala]

Hello @smart-face can you please update/reflash the mendel software and share the output of following commands.

mendel@undefined-apple:~$ uname -r
4.14.98-imx
mendel@undefined-apple:~$ cat /etc/mendel_version
5.3

Python 3.9.7 (default, Sep  3 2021, 06:18:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tflite_runtime as tflite
>>> import pycoral
>>> from pycoral.utils.edgetpu import get_runtime_version
>>> get_runtime_version()
'BuildLabel(COMPILER=6.3.0 20170516,DATE=redacted,TIME=redacted), RuntimeVersion(14)'
>>> tflite.__version__
'2.5.0.post1'
>>> pycoral.__version__
'2.0.0'
>>> from pycoral.pybind._pywrap_coral import ListEdgeTpus as list_edge_tpus
>>> list_edge_tpus()
[{'type': 'pci', 'path': '/dev/apex_0'}]

[sharoseali]

Hi, I am using py coral on raspberry PI4 with a USB edge TPU accelerator. I get the same error when I run the example code provided in pycoral. I also download edge you runtime repo from here https://github.com/google-coral/libedgetpu/releases/download/release-grouper/edgetpu_runtime_20210726.zip and run install.sh to generate libedgetpu.so library for my armv7l device and then I change the path to newly generated .so file. But still I get the same error.
My Error logs are printed below:

python3 make_inference_pycoral.py 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 162, in load_delegate
    delegate = Delegate(library, options)
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 119, in __init__
    raise ValueError(capture.message)
ValueError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "make_inference_pycoral.py", line 17, in <module>
    interpreter = make_interpreter(model_file)
  File "/usr/local/lib/python3.7/dist-packages/pycoral/utils/edgetpu.py", line 87, in make_interpreter
    delegates = [load_edgetpu_delegate({'device': device} if device else {})]
  File "/usr/local/lib/python3.7/dist-packages/pycoral/utils/edgetpu.py", line 52, in load_edgetpu_delegate
    return tflite.load_delegate(_EDGETPU_SHARED_LIB, options or {})
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 166, in load_delegate
    library, str(e)))
ValueError: Failed to load delegate from libedgetpu.so.1

After generating new libedgetpu.so and changing path in "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py" I get again the same issue:

 $ python3 make_inference_pycoral.py 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 162, in load_delegate
    delegate = Delegate(library, options)
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 119, in __init__
    raise ValueError(capture.message)
ValueError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "make_inference_pycoral.py", line 17, in <module>
    interpreter = make_interpreter(model_file)
  File "/usr/local/lib/python3.7/dist-packages/pycoral/utils/edgetpu.py", line 87, in make_interpreter
    delegates = [load_edgetpu_delegate({'device': device} if device else {})]
  File "/usr/local/lib/python3.7/dist-packages/pycoral/utils/edgetpu.py", line 52, in load_edgetpu_delegate
    return tflite.load_delegate(_EDGETPU_SHARED_LIB, options or {})
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 166, in load_delegate
    library, str(e)))
ValueError: Failed to load delegate from /usr/lib/arm-linux-gnueabihf/libedgetpu.so.1.0

any help or suggestion. Thanks

[hjonnala]

@sharoseali can you please share the output of below commands:

• ls -al /usr/lib/arm-linux-gnueabihf/libedgetpu*
• python3 -c 'from pycoral.utils.edgetpu import get_runtime_version; print(get_runtime_version())'

[sharoseali]

@sharoseali can you please share the output of below commands:

• ls -al /usr/lib/arm-linux-gnueabihf/libedgetpu*
• python3 -c 'from pycoral.utils.edgetpu import get_runtime_version; print(get_runtime_version())'

Here are the results

ls -al /usr/lib/arm-linux-gnueabihf/libedgetpu*
lrwxrwxrwx 1 root root     17 Jul  9  2021 /usr/lib/arm-linux-gnueabihf/libedgetpu.so.1 -> libedgetpu.so.1.0
-rw-r--r-- 1 root root 780032 May 23 11:29 /usr/lib/arm-linux-gnueabihf/libedgetpu.so.1.0

python3 -c 'from pycoral.utils.edgetpu import get_runtime_version; print(get_runtime_version())'
BuildLabel(COMPILER=6.3.0 20170516,DATE=redacted,TIME=redacted), RuntimeVersion(14)

Good News !! I managed to make it work by using libedgetpu.so.1 from usr/lib/arm-linux-gnueabihf/ by running python3 with sudo and it worked. Without sudo even if you point usr/lib/arm-linux-gnueabihf/libedgtpu.so.1 it didn't work. Thanks for your reply and sorry for my late reply.

[hjonnala]

@sharoseali can you please try changing the permissions so that you don't need to run with sudo.

sudo chmod +rwx /usr/lib/arm-linux-gnueabihf/libedgetpu.so.1.0
sudo chmod +rwx /usr/lib/arm-linux-gnueabihf/libedgetpu.so.1

[sharoseali]

Thanks @hjonnala i tried but it gives the same message without sudo

python3 inference_TFLite.py --video ./test.mp4 --model best-int8_edgetpu.tflite

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 162, in load_delegate
    delegate = Delegate(library, options)
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 119, in __init__
    raise ValueError(capture.message)
ValueError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "inference_TFLite_YOLO_with_nms.py", line 266, in <module>
    model_path=model, experimental_delegates=[tflite.load_delegate(_EDGETPU_SHARED_LIB,  {})])
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 166, in load_delegate
    library, str(e)))
ValueError: Failed to load delegate from /usr/lib/arm-linux-gnueabihf/libedgetpu.so.1

[smart-face]

Hello @smart-face can you please update/reflash the mendel software and share the output of following commands.

mendel@undefined-apple:~$ uname -r
4.14.98-imx
mendel@undefined-apple:~$ cat /etc/mendel_version
5.3

Python 3.9.7 (default, Sep  3 2021, 06:18:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tflite_runtime as tflite
>>> import pycoral
>>> from pycoral.utils.edgetpu import get_runtime_version
>>> get_runtime_version()
'BuildLabel(COMPILER=6.3.0 20170516,DATE=redacted,TIME=redacted), RuntimeVersion(14)'
>>> tflite.__version__
'2.5.0.post1'
>>> pycoral.__version__
'2.0.0'
>>> from pycoral.pybind._pywrap_coral import ListEdgeTpus as list_edge_tpus
>>> list_edge_tpus()
[{'type': 'pci', 'path': '/dev/apex_0'}]

Hello, sorry for the delay, the first device with continuous inference was not showing any of the PCI devices, and later it started heating up more so I dumped it. I tried flashing a new installation.

For a second case, I figured out the issue by reinstalling a new Mendel. There was a problem with a kernel module that handles the socket connections. BTW thanks for the replay.

[hjonnala]

@smart-face Thanks for response. Can you please confirm that issue got resolved with new Mendel software for the second case. Thanks!

[smart-face]

Yes, the story behind this issue is as follows, I use this device as a tiny HTTP server that processes the requests in a FIFO fashion. During testing, I have performed multiple sockets queries where the kernel module failed and all TCP requests got stuck, resulting in the entire PCI line getting down. In imx8, a Wi-Fi uses pcie1 data lines and pcie2 for EdgeTPU. The issue was there until I re-flashed the SoM. I have tried finding the PCI kernel driver but I chose to reinstall everything for time constraints. Maybe you guys can try this test and figure out the problem. Yes I confirm that my issue is resolved. Thanks

…

Sent from my Galaxy

-------- Original message -------- From: Hemanth-Jonnala ***@***.***> Date: 5/26/22 10:03 PM (GMT+09:00) To: google-coral/edgetpu ***@***.***> Cc: Abhijeet ***@***.***>, Mention ***@***.***> Subject: Re: [google-coral/edgetpu] Edge TPU lifespan resulted in Failed to load delegate (Issue #589) @smart-face<https://github.com/smart-face> Thanks for the response. Can you please confirm that issue got resolved with new Mendel software for the second case? Thanks! — Reply to this email directly, view it on GitHub<#589 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AUGPGWKYYZIQPYSTTSZWDY3VL5ZCLANCNFSM5VQFNVOA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[google-coral-bot]

Are you satisfied with the resolution of your issue?
Yes
No