Getting Coral TPU to work with Frigate via ESXi

[EM-Tom]

Hey all - I've just spent a couple of hours troubleshooting Coral USB passthrough from ESXi 6.7 to Ubuntu then my frigate container - It looks like a reasonably common problem and I couldn't find this solution on Google so I thought I'd post it here in the hopes it might help someone.

Inititally I plugged my new Coral into my ESXi host and checked it was available

[root@esxi:~] esxcli hardware usb passthrough device list
Bus  Dev  VendorId  ProductId  Enabled  Can Connect to VM  Name
---  ---  --------  ---------  -------  -----------------  
1    3    1a6e      89a           true  yes                Global Unichip Corp.

Adding that as a USB Device on my docker host (Ubuntu) then running lsusb didn't seem to work

root@docker01:/home/tom# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 0e0f:0003 VMware, Inc. Virtual Mouse
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

After some googling I discovered that I could disable the 'new' USB module in esxi so thought it was worth a try
esxcli system module set -m=vmkusb -e=FALSE
This requires a reboot

Low and behold

[root@esxi:~] esxcli hardware usb passthrough device list
Bus  Dev  VendorId  ProductId  Enabled  Can Connect to VM  Name
---  ---  --------  ---------  -------  -----------------  
2    6    18d1      9302          true  yes                Google Inc.

That's already looking better

Then lsusb on my docker host again

root@docker01:/home/tom# lsusb
Bus 002 Device 006: ID 18d1:9302 Google Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 0e0f:0003 VMware, Inc. Virtual Mouse
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

And ....

[2022-08-08 10:47:32] frigate.edgetpu                INFO    : Attempting to load TPU as usb
[2022-08-08 10:47:35] frigate.edgetpu                INFO    : TPU found

Also make sure you are running a USB3 USB controller if using a USB3 device!

Hope this helps anyone

[NickyM]

You sire, is a genius!

[atv2016]

would this work for using with VMware player as well ?

[EM-Tom]

Glad it helped!

Just to add to this

I learned that by design the Coral will change its device ID when the first inferance is attempted, so when I just rebooted my ESXi and it came back as the dreaded Global Unichip Corp, I wanted to test that, so I started up Frigate, the first search for the TPU failed - as expected.

[2022-08-10 10:52:38] frigate.edgetpu INFO : Attempting to load TPU as usb [2022-08-10 10:52:39] ws4py INFO : Using epoll Process detector:coral: [2022-08-10 10:53:12] frigate.edgetpu ERROR : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 160, in load_delegate delegate = Delegate(library, options) File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 119, in __init__ raise ValueError(capture.message) ValueError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/opt/frigate/frigate/edgetpu.py", line 135, in run_detector object_detector = LocalObjectDetector( File "/opt/frigate/frigate/edgetpu.py", line 43, in __init__ edge_tpu_delegate = load_delegate("libedgetpu.so.1.0", device_config) File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 162, in load_delegate raise ValueError('Failed to load delegate from {}\n{}'.format( ValueError: Failed to load delegate from libedgetpu.so.1.0 [2022-08-10 10:53:19] frigate.watchdog INFO : Detection appears to have stopped. Exiting frigate...

At this point the device ID changed Bus 002 Device 004: ID 18d1:9302 Google Inc.

And when Frigate auto restarted we're good again

[2022-08-10 10:53:24] frigate.edgetpu INFO : Attempting to load TPU as usb [2022-08-10 10:53:27] frigate.edgetpu INFO : TPU found

[NickyM]

Thanks for the details - do you run Frigate in docker?

[EM-Tom]

Yep

ESXi > Ubuntu running Docker > Frigate container (11 RC1) then I have the Frigate Home Asisstant proxy setup pointing to my URL

[NickyM]

Great, same here. No worries for powerloss then. ;)
I was so close yesterday, on using an old machine for my home automation/video surveillance, since EXSi wouldnt passthrough my onboard USB controller and I'm just about out of PCI-E slots. But this post saved my powerbill ;)

[husnainali09]

This post will be extremely useful if I can ever find a google coral, easier to buy a kidney atm.

My setup is Esxi 6.5 ---> Xpenology (Synology) ---> Docker on synology

[bm55]

Trying the ESXi OS with Google Coral but after disabling vmkusb, it doesn't show any passthrough devices. Any suggestion?

If I enable it again "esxcli system module set -m=vmkusb -e=TRUE" and reboot then it shows the coral

[bm55]

Replaced esxi version 7 with 6.7 and @EM-Tom's solution is working!

[ashleyw-gh]

Just to add some things here to help others. I've been running ESXi 7 on my home server for some time. I have Frigate running on an Ubuntu VM. The only way I could get the Coral USB to work reliably was to us a cheap USB3 PCIe X1 controller card and then to use PCI pass-through on that for the whole card on that VM (after having to set; VMKernel.Boot.disableACSCheck = true on the ESXi server to allow the card to be passed through).

What I did find as part of my testing was that you can't have a VM with multiple PCI devices passed through to the same VM otherwise it can cause the PCIe USB controller card to drop out, and you also can't have both USB PCI pass through and connected USB devices on the same VM - otherwise the USB bus scanning under Linux seems to get confused.

Another thing to be aware of... I had set Frigate to use an NFS store for the recordings... By default the internal database for Frigate was getting stored on the same NFS path.
This was troublesome as I had multiple cameras using HD feeds recording 24/7 and this lead to Frigate container restarting regularly due to timeouts but were showing as "Bus Errors" in the docker logs. So what I did was to keep the recordings going to NFS but putting the Frigate DB on a local SSD drive accessible from the VM - thereby preventing any latency caused by the NFS layer from impacting the performance/reliability of Frigate.
e.g. in my docker-compose.yml file I have;

volumes:
  - /home/ubuntu/frigate/db:/db
  - /mnt/SECURITYCAM/frigate:/media/frigate

and in my Frigate configuration.yml file I have;

database:
  path: /db/frigate.db

where /home/ubuntu/frigate/db is the local directory on SSD where the frigate DB is getting stored and /mnt/SECURITYCAM/frigate is the NFS directory where the recording are being stored.

It would be great if the documentation for Frigate could specifically mention this as it caused me many hours to trace as similar errors were memory related - but this was not the case in this situation.

On my VMware server, I'm getting consistently 7.6ms for the inference speed using the Coral USB stick and everything is working great! (I was previously getting about 40ms inference speed using the CPU detector and that was with an AMD 5950x).
I'm running 2 Reolink RLC-810a (on PoE) Cameras pushing detection streams of 640x360 into Frigate, but 24/7 recording in 2560x1440 as that's the highest resolution on H264 as the 3850x2160 is only available in H265. The detect stream size can not be changed on the Reolink cameras.

The hardest thing was actually getting hold of a Coral USB stick in the first place and figuring out the quirks under VMware!

[blakeblackshear]

Could you suggest a way to make this more clear? https://docs.frigate.video/installation#storage

[NickyM]

I have both Frigate db and recordings saved on the same NFS-share without any issues.

[ashleyw-gh]

thanks guys,
Just looking at the installation guide, things actually look quite clear. I think what threw me was that I must have overlooked moving the DB initially and I couldn't find any reference to the the dreaded "Fatal Python error: Bus error" being related to the a DB over an NFS mount.
But working perfectly now and rock solid.
Thanks again for creating a really useful and reliable AI driven NVR platform!

[cpnbnanamn]

Hey all - I've just spent a couple of hours troubleshooting Coral USB passthrough from ESXi 6.7 to Ubuntu then my frigate container - It looks like a reasonably common problem and I couldn't find this solution on Google so I thought I'd post it here in the hopes it might help someone.

Inititally I plugged my new Coral into my ESXi host and checked it was available

[root@esxi:~] esxcli hardware usb passthrough device list
Bus  Dev  VendorId  ProductId  Enabled  Can Connect to VM  Name
---  ---  --------  ---------  -------  -----------------  
1    3    1a6e      89a           true  yes                Global Unichip Corp.

Adding that as a USB Device on my docker host (Ubuntu) then running lsusb didn't seem to work

root@docker01:/home/tom# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 0e0f:0003 VMware, Inc. Virtual Mouse
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

After some googling I discovered that I could disable the 'new' USB module in esxi so thought it was worth a try esxcli system module set -m=vmkusb -e=FALSE This requires a reboot

Low and behold

[root@esxi:~] esxcli hardware usb passthrough device list
Bus  Dev  VendorId  ProductId  Enabled  Can Connect to VM  Name
---  ---  --------  ---------  -------  -----------------  
2    6    18d1      9302          true  yes                Google Inc.

That's already looking better

Then lsusb on my docker host again

root@docker01:/home/tom# lsusb
Bus 002 Device 006: ID 18d1:9302 Google Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 0e0f:0003 VMware, Inc. Virtual Mouse
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

And ....

[2022-08-08 10:47:32] frigate.edgetpu                INFO    : Attempting to load TPU as usb
[2022-08-08 10:47:35] frigate.edgetpu                INFO    : TPU found

Also make sure you are running a USB3 USB controller if using a USB3 device!

Hope this helps anyone

So.. I've been working to get my USB Coral to work on my R430 running ESXi 6.7.. Tried the above steps. The commands are accepted, but even if I reboot ESXI, I still end up with the same. Did I miss a step somewhere?

localhost:~] esxcli hardware usb passthrough device list
Bus Dev VendorId ProductId Enabled Can Connect to VM Name

---

2 3 1a6e 89a true yes Global Unichip Corp.
1 3 413c a101 false no (passthrough disabled) Dell Computer Corp.

Product: VMware ESXi
Version: 6.7.0
Build: Releasebuild-15160138
Update: 3
Patch: 89

[cpnbnanamn]

Using the cable that came with it. Not sure what port I'm using. It's one of the one's on the back of the server.

[NickM-27]

When I had a USB coral the cable it came with was low quality. I had less issues and much better inference times with a USB C to C cable

[cpnbnanamn]

I don't think my R430 has any USB-C ports.
I was trying not to go to ProxMox.. I tinker with ESXi for work

[cpnbnanamn]

sigh Tried every USB port on the server. No go. Now, I have one of two choices... Maybe 3.. 1) Go to Proxmox 2) Just live with it and let the CPU's do it. 3) do Frigate on another device that will accommodate the TPU's

[atv2016]

Does this work with VMware player as well? e.g. esxcli system module set -m=vmkusb -e=FALSE ? If not how do i get this to work there ?

[Rolzzz]

sadly this didn't work for me, ran " esxcli system module set -m=vmkusb -e=FALSE", rebooted but still see

[root@asuspn50:~] esxcli hardware usb passthrough device list
Bus Dev VendorId ProductId Enabled Can Connect to VM Name
3 2 1a6e 89a true yes Global Unichip Corp.

[dsm150]

I have the same problem. tried on ESXi 6.5 and 6.7

[martijndierckx]

Anyone tried and got it working on ESXi 8.0?

[FoUStep]

Tried it, didn’t change the outcome. I might start looking for to passthrough an USB controller with the coral attached to it.

[NickyM]

@FoUStep - I ended up doing the same on 6.7. The processes kept crashing every second day when using the USB device in passthrough. With a PCI-E USB-controller it has been stable for months now.

[knurz]

It works with a PCI-E Card as described here. I've used this one:
https://www.amazon.de/dp/B00B6ZCNGM/

The stats of my server are:
ESXi-8.0.0-20513097
Dell T40
Xeon E-2224G
64 GB RAM
Frigate VM with 1 vCPU and 3 GB RAM.
6 cameras

I run Go2RTC on another VM. Load for the Frigate VM is around 50-70%. I hope that I can get that down a bit with some optimization. The coral TPU must be one from the newest batch. I got mine 2 weeks ago from mouser.

[martijndierckx]

I bought the same. Works flawlessly!

[JohnClay]

I've been reading about issues w/ the PCIe Corals installed in ESXi systems - did you encounter any of the issues noted at the link below, or have they resolved the firmware issues and the Corals now follow the PCIe spec? google-coral/edgetpu#343 (comment)

[moskovskiy82]

Ditched the whole ESXi setup because of this.

[sanderlv]

I run esxi 8.0.x

Also running HassOS as a VM with 3 USB devices passed through (not controller, but devices). All runs fine.

Now I want to pass through Coral to another VM... BUMP.

When I run esxcli system module set -m=vmkusb -e=FALSE on the esxi host, would this also affect the existing passthoughs, will they remain working?

[lamw]

JFYI - For those interested in using the Coral USB w/latest ESXi releases, please see the following write-up which has all the details. Please do not disable the vmkusb module, while this may have seem to work with earlier releases of ESXi, its not an actual solution and will disable all USB functionality for newer releases of ESXi

https://williamlam.com/2023/05/google-coral-usb-edge-tpu-accelerator-on-esxi.html

[mateuszdrab]

JFYI - For those interested in using the Coral USB w/latest ESXi releases, please see the following write-up which has all the details. Please do not disable the vmkusb module, while this may have seem to work with earlier releases of ESXi, its not an actual solution and will disable all USB functionality for newer releases of ESXi

https://williamlam.com/2023/05/google-coral-usb-edge-tpu-accelerator-on-esxi.html

I must admit it's terrible how they made the device ID change on the USB board and the PCIe board doesn't comply to standards. I was thinking of buying the PCIe card until I saw your post and think I'll leave (sell) the accelerator. I got it nearly a year ago and it's been collecting dust so far.

[jakubsuchybio]

• Was figuring it out today for like 5 hours.
• Your tutorial with Windows is completely unusable, there is a lot of steps missing for windows.
• For esxi it doesn't need to be USB 3.1 controller, but USB 3.0 is enough. (Don't have 3.1, because of older esxi 6.7, which doesn't have it)
• Also it might be out of the scope of that article, but to get it working with Frigate in docker, docker-compose needs to be adjusted to have devices section like this

    devices:
      - /dev/bus/usb:/dev/bus/usb

But in the end your article helped quite a lot. Thank you for it.

[sanderlv]

William, cool! I will test asap and report back.

[YenLegion]

JFYI - For those interested in using the Coral USB w/latest ESXi releases, please see the following write-up which has all the details. Please do not disable the vmkusb module, while this may have seem to work with earlier releases of ESXi, its not an actual solution and will disable all USB functionality for newer releases of ESXi

williamlam.com/2023/05/google-coral-usb-edge-tpu-accelerator-on-esxi.html

Does this guide work for 6.7? The advanced USB quirks parameters? If not, are they needed in 6.7? Thanks.

[jakubsuchybio]

So... As I was saying that I figured it out and it works. Well. For a small amount of time yes. But after like a week the USB coral stopped to work. So like 70% of the steps needed to be redone again. And after another week the same. So basically it was not stable...
So I returned the USB, but bought the pciem variant, then I bought some PCIEM to PCIE adapter card and put it into the server.
I was able to passthrough it to the VM and the VM is detecting it in lspci, but that is as far as I got.
I tried to follow the guide for PCIEM, but got stuck on drivers.
More info here: google-coral/edgetpu#787
However!
When I was debugging this one thing occured to me. That all these issues could be avoided with the USB Coral, if there would be a USB controller in the PCIE passthrough to the VM directly, so that the USB Coral would be directly visible in the VM not through the USB arbitrator of the esxi.
However I already returned the USB coral, so I'm now stuck with that pciem driver issue :( And hopefully waiting for a response from the coral team

[lamw]

re: PCIe Coral - There's no easy fix after speaking w/Engr on our side and best option would be for Coral team to update/fix their firmware so that it doesn't violate the PCIe specification (as originally shared from one of our Principal Engr). I've even filed an issue google-coral/libedgetpu#48 but haven't had any response from the team. I would recommend folks here chime in there to see if we can get Coral team to consider this augmentation to their firmware by either hiding MSI-X (which might be needed to function, not sure honestly OR actually fix the firmware so that it conforms to PCIe spec and would allow ESXi to properly consume it)

re: USB Coral - I've heard of some folks running into instability issues but I know others who've not had any problems over duration of time. The only difference that I could tell is that ones not having issues were using recent systems including Intel NUC 11th Gen or later ... even for myself, I was using Intel NUC 12/13th Gen

[jakubsuchybio]

re: usb coral instability issues - i'm having amd 7 5700G with MSI MAG B550M MORTAR motherboard, but as I said, not sure if those instability issues would have been with the pcie usb controller passthrough to the vm directly, I had the setup with passing only the usb device to the VM

This is kinda sad from coral team...
Is there any other alternative for frigate pcie accelerator instead of coral that is verified that it works properly? Thx 😊

[Marcwa19197]

Sorry to bump this "old" post. Im using the USB Coral with ESXI 8 on a Intel NUC13.

Its working very fine, BUT after some time (few days, maybe one week) the Coral stops working, Frigate is stopped and during restart complaining about not finding the TPU.

ESXI shows:

[root@esxi:~] esxcli hardware usb passthrough device list
Bus  Dev  VendorId  ProductId  Enabled  Can Connect to VM  Name
---  ---  --------  ---------  -------  -----------------  ----
2    2    10c4      ea60          true  yes                Silicon Labs CP210x UART Bridge
2    4    8087      33            true  yes                Intel Corp.
2    5    18d1      9302          true  yes                Google Inc.

My Setup was done after reading https://williamlam.com/2023/05/google-coral-usb-edge-tpu-accelerator-on-esxi.html
So are there any news? Or ideas what i can do to get it stable? Are there other users with that instability?

My Setup is ESXI 8 > VM with HassIO > Frigate as HA Addon.

If this issue is happening i rerun the following on the ESXI-Host:

/etc/init.d/usbarbitrator stop && vmkload_mod -u vmkusb;vmkload_mod vmkusb && kill -SIGHUP $(ps -C | grep vmkde
vmgr | awk '{print $1}') && /etc/init.d/usbarbitrator start

But after that, the Google-Coral is completely gone:

[root@esxi:~] esxcli hardware usb passthrough device list
Bus  Dev  VendorId  ProductId  Enabled  Can Connect to VM  Name
---  ---  --------  ---------  -------  -----------------  ----
2    2    10c4      ea60          true  yes                Silicon Labs CP210x UART Bridge
2    4    8087      33            true  yes                Intel Corp.

So only way to go if this happens is walking to the NUC and reinserting the USB Coral, Mapping the Coral "GlobalChip" via ESXI to my Ubuntu VM, reruning the initialization. Then i need to restart the usbarbitator on ESXI Host, this also affects my Skyconnect USB Stick, so in that timefrime no ZigBee Device is working. And then Shutdown HA VM, remapping the Google-Coral as "Google Chip" to the HA VM and starting everything up again.... it is a mess for me... :-/ Someone fighting with the same problems?

[Marcwa19197]

CPU usage could be dropped significantly if a GPU is used for hwaccel, just to note

If i somehow get it working with my Setup:
ESXI > HassIO VM > Frigate as Addon (Full-Access Version). See also my other Thread: #10698 :)

So i guess if im able to use the iGPU its a better performance than using the normal -virtualized- CPU. But thats another issue... maybe i will open a new Thread for it....

[Marcwa19197]

So sorry to say that, but im deinstalling my ESXI NUC right now and installing my good old buddy... fedora server. Will use a "bare metal" virtualisation for my vm appliances.

For Frigate im not sure, also for home-assistant. Maybe i will set it up in a plain podman container on the host. The other appliances will be virtualised using KVM/qemu.

@lamw Hope you get some insights of the vm-ware Support bundle. Maybe sometime i will switch back to ESXI. But as you know, just for a "homelab" purpose ESXI is gone. And i will also not support the decision of Broadcom. Sorry to say that.

[YenLegion]

The only way I have gotten this to work for extended periods of time is by using the USB 2 controller. Inference time is higher, mine is between 50-150ms, but it runs for weeks at a time instead of hours at a time before crashing.

[Marcwa19197]

So right now it looks to be stable.
Im running Frigate directly on FedoraServer 39 within Podmain Container. One point was the mapping of the Device into a rootless container. But following https://gist.github.com/ofstudio/ab8001a21c257d67255c9f43451132c0 fixed it.
Google USB Coral is connected via a good USB-C to USB-C Cable on the backplane of NUC13.

Im also able to use hwaccel with Intel iGPU in Container.

Only point open (but this is just a cosmetic thing for me): Intel iGPU Stats are not shown. Its also something related to the rootless-container.

My Container-Deployment File:

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: false # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "512mb" # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb  # Passes the USB Coral, needs to be modified for other versions
      - /dev/dri/renderD128:/dev/dri/renderD128 # For intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /vmdatastore/container/frigate/config:/config:Z
      - /frigate-media:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "verysecret123!"

Also note the uppercase "Z" for the correct relabelling of the mapped Volume on SELinux. Also one rule to set for SELinux: sudo setsebool container_use_devices=true

[Marcwa19197]

One week later - USB Coral still stable. Dont know if it is related to the new USB-C to USB-C Cable or to the change of the infrastructure from ESXI VM running HomeAssistant and Frigate as Addon to Plain Container running on Fedora Server bare-metal.