Dual edge TPU hangs when running detect_image.py

[truncs]

classify_image.py runs well and the temprature seems to be stable. But when I run detect_image.py the temprature goes negative and the device is throttled with HIB errors. Please see below for logs

(base) aditya@aditya-desktop:~/workspace/coral/pycoral$ python3.6 examples/classify_image.py --model test_data/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite --labels test_data/inat_bird_labels.txt --input test_data/parrot.jpg
----INFERENCE TIME----
Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory.
12.1ms
2.7ms
2.7ms
2.7ms
2.7ms
-------RESULTS--------
Ara macao (Scarlet Macaw): 0.76953

base) aditya@aditya-desktop:~$ for (( ; ; )); do  sleep 1; cat /sys/class/apex/apex_0/temp; done
43050
43300
43550
43300
43550
43300
43550
43550
43300
43550
43550
43550
43300
43300
43300
43300
43550
43050
43300
43300
43300
43050
43550
43050
43300
43550
43550
43300
43550
43300
43550
43300
43050
43300
43300
43300
43300
43050
43550
43050
43550
43300
43300
43300
43050
43300
43300
43050
43050
43300
43300
43300
43050
43300
43050
43050
43300
43050
43300
43050
43300
43550
43300
43050
43300
43300
43050
43300
43050
43300
43300
43300
43300
43300
43300
43300

(base) aditya@aditya-desktop:~/workspace/coral/pycoral$ python3.6 examples/detect_image.py   --model test_data/ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite   --labels test_data/coco_labels.txt   --input test_data/grace_hopper.bmp   --output ${HOME}/grace_hopper_processed.bmp
----INFERENCE TIME----
Note: The first inference is slow because it includes loading the model into Edge TPU memory.
E driver/mmio_driver.cc:254] HIB Error. hib_error_status = ffffffffffffffff, hib_first_error_status = ffffffffffffffff

(base) aditya@aditya-desktop:~$ for (( ; ; )); do  sleep 1; cat /sys/class/apex/apex_0/temp; done
43300
43550
43800
43800
43800
43550
43550
43800
43550
43800
43800
43050
43300
43300
43300
43550
43550
43300
43550
43550
43550
43550
-89700
-89700
-89700
-89700
-89700
-89700
-89700
-89700
-89700
-89700
-89700
-89700
-89700
-89700
-89700
-89700

(base) aditya@aditya-desktop:~$ uname -r
4.15.0-153-generic

[hjonnala]

@truncs sorry for the delay. Are you still facing the issue? Can you please try with other object detentions models and see if Dual Edge TPU hangs or not.

[truncs]

Yeah same problem.

[hjonnala]

@truncs can you please share the software and hardware details.

[hjonnala]

can you please try two model inference.py with dual edge tpu and share the output of below snippet:

hemanth@hemanth-glaptop:~$ python3
Python 3.9.7 (default, Sep  3 2021, 06:18:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pycoral.pybind._pywrap_coral import ListEdgeTpus as list_edge_tpus
>>> list_edge_tpus()
[{'type': 'usb', 'path': '/sys/bus/usb/devices/2-1'}]
>>> 

[truncs]

This is what I see

list_edge_tpus()
[{'type': 'pci', 'path': '/dev/apex_0'}]

[manoj7410]

@truncs Can you paste the output of the command:

lspci -vvv

[truncs]

The relevant lspci -vvv

/apex                                                                                                                                                                                                                                                                                                                         
...skipping                                                                                                                                                                                                                                                                                                                   
        Kernel driver in use: apex                                                                                                                                                                                                                                                                                            
        Kernel modules: apex

[hjonnala]

do you have any other operating system and/or hardware to test to know whether the issue with software or hardware?

[truncs]

I don't have a hardware with an E key slot, unless I can plug it into a Pi.

[hjonnala]

Okay, can you please try with different operating system as other user faced HIB error issue with windows but it got resolved with ubuntu. google-coral/edgetpu#484

[truncs]

That is interesting since I did test on Ubuntu without any adapters. Do you guys have any recommendation on what adapter to use?

Distributor ID: Ubuntu
Description:    Ubuntu 18.04.6 LTS
Release:        18.04
Codename:       bionic

[hjonnala]

we have tested ASUS Coral Card: https://iot.asus.com/products/AI-accelerator/AI-Accelerator-PCIe-Card/. But, with this card you won't be able to use the dual edge TPU as it likely doesn't provide 2 PCIe buses per M.2 (but a single TPU card would work).

Here is an example for the ASUS PCIe card that uses 8 TPUs in parallel: https://github.com/google-coral/demo-multi-video-stream

[google-coral-bot]

Are you satisfied with the resolution of your issue?
Yes
No