coral edge TPU issues on RPI4 #46
Comments
|
That's a new one for me. Do you think it's possible the USB flash drive is bad? Maybe it starts swapping and causing the issue? I have a RPi4 with microsd and never had an issue. I thought I saw some comment about power... did you delete that? Could the power be sagging? I thought I saw the RPi4 needs a 3A power supply... |
|
Nah, I don't think it's an issue with the USB drive -- they're 2.5" western digital SSDs attached to UASP/USB3 to SATA3 interface adapters and I've got the exact same setup running on 4x RPI 4Bs -- all of which have been up and stable for months running various workloads. I think a relevant question is whether its something with the Firmware of the Coral device or the RPI4B. It's a bit interesting the device IDs change once loaded or probed via the app or whatever is happening there. The PIs all have 3.5A power supplies. I did write a thing about power earlier. I had installed the |
|
I'm running these workloads as part of a Hashi Nomad, Consul, Vault cluster, with noobs having an affinity to land on a particular node that has the Coral Edge TPU. What I can do, maybe, is move that workload to the one RPI 3b+ I have that's got an SD card as it's boot/primary volume to see if it's able to do its thing. I'll try doing that later today. |
|
The PI3B+ doesn't have the same issues. It looks like we're doing requests pretty quick with the TPU: Logs like: ... and a load average close to 1. The same request stream using the non-TPU model is significantly slower. log entries: ...with a CPU load average closer to 3 proving that the TPU is about 2x as fast. Which is better than I thought it'd be considering it's USB 2. I'm using this example to send 1080p h264 streams to doods. There might be a penalty downsampling the video frames too. Perhaps I should switch to lower res. |
|
Man these look good.... shame you can't buy them in Australia! Hope you get it going! |
|
Currently looking at a lead here thinking that it might related to power safe issues with the firmware. |
|
Hmmm.. That's interesting. Sorry, I don't have a lot of other suggestions. Most problems I have seen with EdgeTPU generally seem to relate to power or the USB. It either works perfect or has random crashes on a particular hardware platform. I have a raspberry Pi 4 and I don't have issues with it though. There is another issue with resizing causing it to be really slow, I am working on that one. It should be even faster soon. |
|
@snowzach Is working but sometimes processing delays are longer. How is the progress with issue with resizing going? |
|
@mmatesic01 I fixed the issue with resizing making it slow, that shaved off a good 500ms or so. That is def fixed. Make sure for the models using the EdgeTPU you have numThreads set to 1 and numConcurrent also set to 1. It might be trying to send more than one detection at a time through the TPU and causing issues. I think I also upgraded the edgeTPU library as well. I'll have to check that. Been a while since I looked. Are you def on the latest version? If so, I can try checking for an EdgeTPU upgrade. |
|
@snowzach |
|
Hi,
In your opinion which of two parallel camera source formats should I use 1280x720@15FPS or 640x360@7FPS in better processing result and lower load on CORAL/RPi CPU?
Or it does not matter?
Scan interval is 2 s.
…________________________________
From: Zach <notifications@github.com>
Sent: Saturday, December 26, 2020 22:01
To: snowzach/doods <doods@noreply.github.com>
Cc: mmatesic01 <mmatesic@live.com>; Mention <mention@noreply.github.com>
Subject: Re: [snowzach/doods] coral edge TPU issues on RPI4 (#46)
@mmatesic01<https://github.com/mmatesic01> I fixed the issue with resizing making it slow, that shaved off a good 500ms or so. That is def fixed. Make sure for the models using the EdgeTPU you have numThreads set to 1 and numConcurrent also set to 1. It might be trying to send more than one detection at a time through the TPU and causing issues. I think I also upgraded the edgeTPU library as well. I'll have to check that. Been a while since I looked. Are you def on the latest version? If so, I can try checking for an EdgeTPU upgrade.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#46 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIZLSD6FPSRH4MXJJTT5QPDSWZFLBANCNFSM4T7DDLZQ>.
|
|
@mmatesic01 it doesn't matter which one you use... It will be resized to 300x300 because that's what the model requires. That's the latest version as well. Curious if those changes to the config help? |
|
Changes you sugested did not make much of an effect on silencing the errors in HA. But in general it does not effect the expected functionality. |
|
I'm seeing a very similar issue to @joshdurbin here. Running RPI4 off of a SSD with a Coral Edge TPU. The Edge TPU is plugged into a USB hub that is not powered. HOWEVER, After the first run, so far my system seems stable. for whatever reason, the first time i start doods i get stuck, the error: Then after that, all is well and I haven't seen any crashes. Will see if the stability continues. Maybe if doods tried to initialize a few times, It would work without hitting this the first run? EDIT: as I've run out of spots on my current USB hub, i'm going to grab a larger one that is powered separately. perhaps that'll help resolve this for me. EDIT 2: Even after using a powered USB hub, i still see the same issues. |
|
@sebirdman |
|
@mmatesic01 I've got this RPi 4 running a whole bunch of things, including Coral USB/Doods. I used to run a NUT server for my UPS on it, but noticed that the UPS usb was frequently disconnecting from my RPi. After removing that device, so far - all of my issues have went away. Even the strange error on first start of the Doods server. Thanks for the suggestion, I've got another RPi4 that is currently unused. If i continue to have issues i'll report them here and will try switching to a dedicated RPi4 for Coral USB/Doods. Edit: Also, Do you have your dedicated RPi running wireless or wired internet? I'm curious how the network connection would effect the processing time from HA to Doods. I've got a few RPi's already setup around my home doing other things that i could try as well, but they're all wifi. |
|
Please try with new Python DOODS if this is still and issue: https://github.com/snowzach/doods2 |
|
Hello,
Did the instalation and ran docker image with additiional » -- device /dev/bus/usb «. You have a typo in your instructions abou this parameter.
Then I changed config.yaml in my container and copied model and label file for Edge Coral.
doods:
detectors:
- name: default
type: tflite
modelFile: models/ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite
labelFile: models/coco_labels_0.txt
hwAccel: true
2. http://192.168.xxx.xxx:8080/detectors shows:
{"detectors":[{"name":"default","type":"tensorflow2","model":"models/ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite","labels":["person","bicycle","car","motorcycle","airplane","bus","train","truck","boat","traffic light","fire hydrant","n/a","stop sign","parking meter","bench","bird","cat","dog","horse","sheep","cow","elephant","bear","zebra","giraffe","n/a","backpack","umbrella","n/a","n/a","handbag","tie","suitcase","frisbee","skis","snowboard","sports ball","kite","baseball bat","baseball glove","skateboard","surfboard","tennis racket","bottle","n/a","wine glass","cup","fork","knife","spoon","bowl","banana","apple","sandwich","orange","broccoli","carrot","hot dog","pizza","donut","cake","chair","couch","potted plant","bed","n/a","dining table","n/a","n/a","toilet","n/a","tv","laptop","mouse","remote","keyboard","cell phone","microwave","oven","toaster","sink","refrigerator","n/a","book","clock","vase","scissors","teddy bear","hair drier","toothbrush"],"width":0,"height":0}]}
3. Why is type different in my config and report (tflite- tensorflow2)?
I did try to test my stream address in your GUI, http://192.168.xxx.xxx:8080/, which is acessible.
***@***.*** But regarding stream (with my rtsp address) test all I got is empty blue web screen.
BR
MArko
Od: ***@***.***>
Poslano: nedelja, 02. januar 2022 15:43
Za: ***@***.***>
Kp: ***@***.***>; ***@***.***>
Zadeva: Re: [snowzach/doods] coral edge TPU issues on RPI4 (#46)
Please try with new Python DOODS if this is still and issue: https://github.com/snowzach/doods2
—
Reply to this email directly, view it on GitHub<#46 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIZLSD33NC4FAEW7UU67OWTUUBQAZANCNFSM4T7DDLZQ>.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Howdy. I'm not sure this is explicitly a thing with doods, but thought you might have some insight into what might be going on with doods and Coral. I'm running Coral attached to a RPI4B that has USB boot enabled and uses an external USB disk as its system drive (no SD card at all). The OS is debian/pios lite running arm64.
I'm trying to transition from CPU-based detection to the Coral tf lite models. To do so I used the script to download all the models then, with the following config:
... am running:
docker run -it --device /dev/bus/usb -e logger.level='debug' -v /home/pi/models/:/opt/doods/models -v /home/pi/asdf.config:/opt/doods/config.yaml -p 8080:8080 snowzach/doods:latest. The first time through it errors out, stating it can't initialize the edgetpu.Prior to this run the
lsusboutput returns:...after the run, though, the
Bus 002 Device 003changes to:A second pass at running the container yields the following:
and will remain up and online for about another 8-10 seconds before the host OS crashes. It crashes because the USB host controller seems to die and hang. Specifically:
and:
Full log:
This requires a hard reset of the PI4. Any idea what could be going on here?
I've tried the Coral device on the 2 other RPI4Bs that I have and it produces the same error/problem.
The text was updated successfully, but these errors were encountered: