[Detector Support]: Detection appears to be stuck. Restarting detection process... #16649
Describe the problem you are havingHi Frigate team, I have the USB Google Coral TPU and every few hours I get the INFO message in the logs "Detection appears to be stuck. Restarting detection process..." and then Frigate kills and restarts the detection process. After going through the discussion boards I can see a potential solution was to replace the USB cable, which I did with a new USB-IF Certified USB-C Cable and also swapped the ports. This unfortunately did not stop the INFO messages from appearing. I have seen people replacing the USB one with the dual TPU and Dual Edge TPU Adapter - PCIe. But before purchasing this I was wondering is there anything you could suggest I take a look before I go and purchase these ? I am running Frigate in docker on a Ubuntu 24.04.1 LTS server and the motherboard USB ports are USB 3.1 Gen2. Version0.15.0-cea210d Frigate config filemqtt:
enabled: false
auth:
reset_admin_password: false
ffmpeg:
hwaccel_args: preset-vaapi
detectors: # <---- add detectors
coral:
type: edgetpu
device: usbdocker-compose file or Docker CLI commandversion: "3.9"
services:
frigate:
container_name: frigate
privileged: true # this may not be necessary for all setups
restart: unless-stopped
image: ghcr.io/blakeblackshear/frigate:0.15.0
shm_size: "349.12mb" # update for your cameras based on calculation above
devices:
- /dev/bus/usb:/dev/bus/usb # Passes the USB Coral, needs to be modified for other versions
- /dev/dri/renderD128:/dev/dri/renderD128 # For intel hwaccel, needs to be updated for your
environment:
LIBVA_DRIVER_NAME: radeonsiRelevant Frigate log output2025-02-17 13:27:36.377200511 [INFO] Preparing Frigate...
2025-02-17 13:27:36.489030569 [INFO] Starting Frigate...
2025-02-17 13:27:38.889234844 [2025-02-17 13:27:38] frigate.util.config INFO : Checking if frigate config needs migration...
2025-02-17 13:27:38.918998718 [2025-02-17 13:27:38] frigate.util.config INFO : frigate config does not need migration...
2025-02-17 13:28:08.093282285 [2025-02-17 13:28:08] frigate.app INFO : Starting Frigate (0.15.0-cea210d)
2025-02-17 13:28:08.123823425 [2025-02-17 13:28:08] peewee_migrate.logs INFO : Starting migrations
2025-02-17 13:28:08.124084213 [2025-02-17 13:28:08] peewee_migrate.logs INFO : There is nothing to migrate
2025-02-17 13:28:08.144726197 [2025-02-17 13:28:08] frigate.app INFO : Recording process started: 436
2025-02-17 13:28:08.145433723 [2025-02-17 13:28:08] frigate.app INFO : Review process started: 438
2025-02-17 13:28:08.147680903 [2025-02-17 13:28:08] frigate.app INFO : go2rtc process pid: 98
2025-02-17 13:28:08.163123064 [2025-02-17 13:28:08] detector.coral INFO : Starting detection process: 452
2025-02-17 13:28:10.817466806 [2025-02-17 13:28:08] frigate.app INFO : Output process started: 470
2025-02-17 13:28:10.817696756 [2025-02-17 13:28:08] frigate.app INFO : Camera processor started for side_passage: 484
2025-02-17 13:28:10.828786564 [2025-02-17 13:28:08] frigate.detectors.plugins.edgetpu_tfl INFO : Attempting to load TPU as usb
2025-02-17 13:28:10.828995445 [2025-02-17 13:28:10] frigate.detectors.plugins.edgetpu_tfl INFO : TPU found
2025-02-17 13:28:10.829704383 [2025-02-17 13:28:08] frigate.app INFO : Camera processor started for driveway: 487
2025-02-17 13:28:10.829829778 [2025-02-17 13:28:08] frigate.app INFO : Camera processor started for backdoor: 490
2025-02-17 13:28:10.829986352 [2025-02-17 13:28:08] frigate.app INFO : Camera processor started for reolink_doorbell: 500
2025-02-17 13:28:10.830061803 [2025-02-17 13:28:08] frigate.app INFO : Capture process started for side_passage: 510
2025-02-17 13:28:10.830190835 [2025-02-17 13:28:08] frigate.app INFO : Capture process started for driveway: 520
2025-02-17 13:28:10.830293788 [2025-02-17 13:28:08] frigate.app INFO : Capture process started for backdoor: 525
2025-02-17 13:28:10.830434060 [2025-02-17 13:28:08] frigate.app INFO : Capture process started for reolink_doorbell: 535
2025-02-17 13:28:10.830584402 [2025-02-17 13:28:08] frigate.api.fastapi_app INFO : Starting FastAPI app
2025-02-17 13:28:10.830719014 [2025-02-17 13:28:08] frigate.api.fastapi_app INFO : FastAPI started
2025-02-17 16:39:18.815334865 [2025-02-17 16:39:18] frigate.watchdog INFO : Detection appears to be stuck. Restarting detection process...
2025-02-17 16:39:18.815489345 [2025-02-17 16:39:18] root INFO : Waiting for detection process to exit gracefully...
2025-02-17 16:39:48.845636058 [2025-02-17 16:39:48] root INFO : Detection process didn't exit. Force killing...
2025-02-17 16:39:48.872978128 [2025-02-17 16:39:48] root INFO : Detection process has exited...
2025-02-17 16:39:48.936493585 [2025-02-17 16:39:48] detector.coral INFO : Starting detection process: 84201
2025-02-17 16:39:51.594004326 [2025-02-17 16:39:48] frigate.detectors.plugins.edgetpu_tfl INFO : Attempting to load TPU as usb
2025-02-17 16:39:51.605355542 [2025-02-17 16:39:51] frigate.detectors.plugins.edgetpu_tfl INFO : TPU found
2025-02-17 17:06:38.976425305 [2025-02-17 17:06:38] frigate.watchdog INFO : Detection appears to be stuck. Restarting detection process...
2025-02-17 17:06:38.976568844 [2025-02-17 17:06:38] root INFO : Waiting for detection process to exit gracefully...
2025-02-17 17:07:09.006716531 [2025-02-17 17:07:09] root INFO : Detection process didn't exit. Force killing...
2025-02-17 17:07:09.051257667 [2025-02-17 17:07:09] root INFO : Detection process has exited...
2025-02-17 17:07:09.115473084 [2025-02-17 17:07:09] detector.coral INFO : Starting detection process: 94027
2025-02-17 17:07:11.768825571 [2025-02-17 17:07:09] frigate.detectors.plugins.edgetpu_tfl INFO : Attempting to load TPU as usb
2025-02-17 17:07:11.779661267 [2025-02-17 17:07:11] frigate.detectors.plugins.edgetpu_tfl INFO : TPU found
2025-02-17 19:24:49.304609352 [2025-02-17 19:24:49] frigate.watchdog INFO : Detection appears to be stuck. Restarting detection process...
2025-02-17 19:24:49.304692678 [2025-02-17 19:24:49] root INFO : Waiting for detection process to exit gracefully...
2025-02-17 19:25:19.335318298 [2025-02-17 19:25:19] root INFO : Detection process didn't exit. Force killing...
2025-02-17 19:25:19.393513299 [2025-02-17 19:25:19] root INFO : Detection process has exited...
2025-02-17 19:25:19.444791326 [2025-02-17 19:25:19] detector.coral INFO : Starting detection process: 140783
2025-02-17 19:25:22.103497793 [2025-02-17 19:25:19] frigate.detectors.plugins.edgetpu_tfl INFO : Attempting to load TPU as usb
2025-02-17 19:25:22.108987715 [2025-02-17 19:25:22] frigate.detectors.plugins.edgetpu_tfl INFO : TPU foundInstall methodDocker Compose Object DetectorCoral Screenshots of the Frigate UI's System metrics pages
Any other information that may be helpful
|
Replies: 6 comments 14 replies
|
You might also try plugging the Coral into a powered USB 3.0 hub. Some users have found success with that as well. |
|
Ok thanks appreciate the response! I will order one of those and give it a try. Will update this thread once I have tested it. |
|
Hi all, so I tested with a powered USB Hub and unfortunately this was even more unstable. Frigate for some reason could not see the USB Coral and kept throwing errors. I did see some other posts about using a USB 3 cable that was too long with the Coral so I bought a 0.3m cable matters and will test. Don't have the highest of hopes that will do the trick but will monitor none the less and might just purchase the PCIE adapter and dual TPU if that doesn't work. |
|
What CPU are you running? If you're on an Intel 6th gen or higher, you can use the OpenVINO detector. |
|
I'm running an AMD Ryzen 5 3400G Processor |
|
I had the same issue! I'm running unbuntu server 22.04 with frigate in docker passing the usb through: The issue I found is that my server usb 3.0 spec is the minimum 900 mA. im sure my used server built in 2017 is probably not putting this out for both internal usb ports on the same stack as it worked with 1 coral device just fine and issue started when I installed the second. Quote from coral usb specifications My solution was to use an external usb hub, this is what i bought Feel free to buy what ever ya like. Just sharing what I did. NOTE there are not affiliate links or any thing. I'm not getting a kick back if you buy the above. Installed the hub and corals inside the 1U server and fed the usb c power cable out the back. removed a micro pci plate cover to give me a 1in hole :) The usb c cable is for just supplying power. I use the stock coral cable plugged directly in to the usb 3.0 hub then the hub plugged directly into the internal usb 3.0 port. Now that I'm running both corals off of that one hub and my only lines in the log about the TPU coral device is:
oh if I had to do this again id get this charger of 5.1v instead as i have no issues now I'm going to leave it be. but the 5.1v external charge circuit issue out lined in link above might be why your having issues with your current powered hub @seanbrady180 |
|
Thanks for this clear writeup, @meeki007 |
|
@meeki007 Really appreciate the write up super informative! So I am using the motherboard I have tried both USB 3.1 Gen2 (Rear Type-A+C) with the Coral and a new USB-IF (1meter) cable directly plug in and but would still get the issue. Great idea about the placement of your coral I was just leaving my one lose on top of the server, does it require active cooling? Could my issue be potentially relating to overheating of the Coral? I'm based in Ireland so would have a different selection on amazon but I went for this powered hub, please excuse my ignorance but did I get the wrong type of powered hub? Now this might also be user error on my part but I plugged the coral into the powered hub and then plugged the hub into one of the USB 3.1 Gen2 ports on my server. I forgot to save the logs but my frigate container was not able to find the USB coral device when it was connected to the powered hub. Would anyone know if the the docker compose configuration needs to change for the usb passthrough when using the powered hub? |
"USB accelerator says it works in a reduced state at 35C and maximum at 25C" ---> https://www.reddit.com/r/homeassistant/comments/15z35w2/frigate_is_google_coral_usb_suppose_to_be/
It helps if you want to get the most out of it. It should clock its self down when temps get high see above link talking about it.
No. USB device should passthrough no issues. Mine did.
Thank you for this info ^^^ important for helping you out. when you first plug it in. You then have to use it once for it to change ID to Quote from github coral page "When Coral USB connected it would list as Global Unichip Corp and after first inference it would change to 'Google Inc'." see this link ---> google-coral/edgetpu#536 I've had this issue a docker stop then docker run did not fix it. I had to restart the server then it worked but if I stopped docker it would not work again and I had to do a full server reboot and then it's ID was Google Inc. @seanbrady180 I want to make sure you read the amazon review outlining the issue with using powered hubs ---> https://www.amazon.com/gp/customer-reviews/R36CEESKZGEKC9/ref=cm_cr_arp_d_rvw_ttl?ie=UTF8&ASIN=B0CN3F9Y1Z I don't know if you got the wrong hub. There are allot of issues with hubs and getting the correct one matters. Looking at your mother board it should have enough power on a usb port to run a coral without a powered hub. I'm wondering if you are having an over heating issue ... over 35c or close to it |
|
@meeki007 Thanks for that link about the coral initializing that explains a lot! With my ubuntu version I couldn't download the python libraries to test the Coral and initialize it, so I was always wondering when I restarted my server why it took me two attempts to start up the docker container and only the second time it would find the TPU. In other news I swapped out to the shorter 0.3m cable and and for the last few days still getting the random TPU restart log messages. I took a look at the powered hub amazon review linked above, a little concerning to be honest. I'm honestly now thinking of just going for the Dual TPU option seems like the safest and most reliable at this point. |
|
Hi all, I got the Google Coral Dual PCIE detector with the special adapter. It has been running now for quite a few days and I have not had one INFO message about the detector needing restart. On a side note if anyone is trying to install the PCIE version of the coral on a latest version of Ubuntu I found this post with workarounds really helpful: |
|
Good info. Update on my USB hub
It started crashing with 4 corals
I even tried cutting USB 5v power cable cord to isolate from the host, so
100% powered from external.
Still same issues.
I'm looking at PCIe USB 3.0 cards that will play nice with the server
…On Thu, Mar 6, 2025, 13:20 seanbrady180 ***@***.***> wrote:
Hi all, I got the Google Coral Dual PCIE detector with the special
adapter. It has been running now for quite a few days and have not had one
INFO message about the detector needing restart.
On a side note if anyone is trying to install the PCIE version of the
coral on a latest version of Ubuntu I found this post with workarounds
really helpful:
https://www.reddit.com/r/frigate_nvr/comments/1g3izfe/google_coral_project_appears_to_have_been/
—
Reply to this email directly, view it on GitHub
<#16649 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABNNLRGZ43SSEYNFGEQEWZD2TCGYRAVCNFSM6AAAAABXKI6ZUOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENBRG42TCOI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
com>
|
|
Are you in the position to get the dual coral one ? I got this adapter for it which allows you to access both Coral TPUs: https://www.makerfabs.com/dual-edge-tpu-adapter.html But obviously all depends on the budget, PCIE slot and OS you are using. I highly recommend it, currently working flawlessly for me the Detector Inference Speed went down by about 2 seconds, no restarts in the past 4 days and great future proof as it's a dual TPU so plenty of room to add more cameras. |
|
Price I can afford it.
The operating system must be Linux based.
My issue/ The issue with rack mounted servers is that they can be selective
about what hardware will work on them.
Thanks for the links! I might just buy it and see if it works.
…On Thu, Mar 6, 2025, 17:22 seanbrady180 ***@***.***> wrote:
Are you in the position to get the dual coral one
<https://coral.ai/products/m2-accelerator-dual-edgetpu/> ?
I got this adapter for it which allows you to access both Coral TPUs:
https://www.makerfabs.com/dual-edge-tpu-adapter.html
But obviously all depends on the budget, PCIE slot and OS you are using.
I highly recommend it, currently working flawlessly for me the Detector
Inference Speed went down by about 2 seconds, no restarts in the past 4
days and great future proof as it's a dual TPU so plenty of room to add
more cameras.
—
Reply to this email directly, view it on GitHub
<#16649 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABNNLRF6ZGLLWT2ZYCK3BJD2TDDCDAVCNFSM6AAAAABXKI6ZUOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENBRHE2DMMQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
com>
|
|
Hi, I encountered a similar issue with the Google Coral TPU and the "Detection appears to be stuck. Restarting detection process..." message appearing in the logs. After trying a few things, I initially swapped to a new 1-meter USB-IF Certified cable and also used a powered hub, but the issue persisted. Interestingly, I decided to try using a shorter 30cm USB-IF Certified cable, and it's been 4 days now without encountering the disconnect error (previously, it would happen at least once a day). I don’t fully understand why, but it seems to have worked. It would be great if anyone could explain if cable length might be a factor in this kind of issue. Best regards, |
|
@nizaga Would you mind linking to which cable worked for you? I would love to give this a try. |
Cable performance is intrinsically linked to cable length. This applies to simple situations with (say) 12V dc powering a light bulb, or complicated circumstances like high speed differential data (ethernet, USB3). In the simple example, the gauge of copper used (the diameter of the conductor) governs the voltage drop per unit length as a function of the current (drawn by the bulb in this case). In other words, a very long cable ends up delivering to the bulb a lower voltage than the supplied voltage and you end up with a dim bulb. In the complicated example, the same principles apply. Shorter cables exhibit lower total resistance for the same conductor diameter and hence the digital signals presented at one end, generated from the other, start to degrade in amplitude and the threshold for discriminating between a logical one and a logical zero becomes more blurred as the cable gets longer. This is complicated by higher speed signalling whereby aspects such as conductor skin effect (additional resistive loss as a function of signal frequency) and cross talk between conductors also increases with signalling frequency and cable length. Increasing frequency also increases phase shift per unit length, requiring twisted pairs of conductors, perhaps individually shielded per pair, to control the characteristic impedance of what is now a (data) transmission line. The idea is to shove a one and a zero in at one end, get the same out at the other end, and not have signals echoing back and forth between the two ends which is what happens with impedance mismatch. USB and ethernet bear protocols to help control aspects of all of this. Ergo: cheap USB cables, or worse still, cheap cables which are longer than necessary, are a bad idea. Certified cables are the route to follow if reliable performance is needed in demanding applications. Assuming a USB Coral TPU is a demanding application, this might explain why long (unqualified) cables don't perform reliably whereas a cable which is as short as possible and is from a qualified source is likely to be a better bet. |
|
Many thanks for the explanation. What I don’t quite understand is how a certified cable, being only 90 cm long, could still trigger the issue. Maybe the cable was defective — I’m not sure. In my case, I tested with a certified 1-meter cable, specifically this one: https://www.amazon.es/dp/B01GGKYR2O?ref_=ppx_hzsearch_conn_dt_b_fed_asin_title_31&th=1 (90 cm). I'm currently trying to find the exact reference of the 0.3-meter cable I'm using, as I haven't located it yet. As soon as I find it, I'll share the link. |
|
I’ve now found the exact reference of the 0.3-meter cable I mentioned — it’s this one : https://amzn.eu/d/0w4FHs8 |
|
Good work. Yes, 13 units is a lot of money for a short cable, when in fact you can buy a USB cable for 1 unit or even cheaper! But then you'll find the cheap cable is unreliable and not worth the pain. I have seen "USB2.0" cables which are four cores (+V, 0V, +data, -data) and no overall braiding/screening which means they'll work as DC power supply cables and not much else. Certainly not USB2.0! |
Hi all, I got the Google Coral Dual PCIE detector with the special adapter. It has been running now for quite a few days and I have not had one INFO message about the detector needing restart.
On a side note if anyone is trying to install the PCIE version of the coral on a latest version of Ubuntu I found this post with workarounds really helpful:
https://www.reddit.com/r/frigate_nvr/comments/1g3izfe/google_coral_project_appears_to_have_been/