[Detector Support]: Coral M.2 B+M Key via ESXi

[sanderlv]

Describe the problem you are having

I have installed a Coral M.2 Accelerator B+M key in my NUC12 which runs ESXi.

Successfully passed through the device via ESXi to my Ubuntu VM.

But I have no luck (yet) in activating it for Frigate.

Please help out.

Version

0.12.0-da3e197

Frigate config file

birdseye:
  enabled: False
#  width: 1280
#  height: 720
#  # 1 is the highest quality, and 31 is the lowest. Lower quality feeds utilize less CPU resources.
#  quality: 8
#  # Optional: Mode of the view. Available options are: objects, motion, and continuous
#  #   objects - cameras are included if they have had a tracked object within the last 30 seconds
#  #   motion - cameras are included if motion was detected in the last 30 seconds
#  #   continuous - all cameras are included always
#  mode: objects

database:
  path: /db/frigate.db

#logger:
#  default: info
#  logs:
#    frigate.record: debug

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p

record:
  enabled: true
  retain:
    # Optional: Number of days to retain recordings regardless of events (default: shown below)
    # NOTE: This should be set to 0 and retention should be defined in events section below
    #       if you only want to retain recordings of events.
    days: 0
    # Optional: Mode for retention. Available options are: all, motion, and active_objects
    #   all - save all recording segments regardless of activity
    #   motion - save all recordings segments with any detected motion
    #   active_objects - save all recording segments with active/moving objects
    # NOTE: this mode only applies when the days setting above is greater than 0
    mode: all
  # Optional: Event recording settings
  events:
    retain:
      default: 14

rtmp:
  enabled: False

# MQTT Config ##########################################################################################################
mqtt:
  host: 192.168.1.142
  # Optional: port (default: shown below)
  port: 1883
  # Optional: topic prefix (default: shown below)
  # WARNING: must be unique if you are running multiple instances
  topic_prefix: frigate-2
  # Optional: client id (default: shown below)
  # WARNING: must be unique if you are running multiple instances
  client_id: frigate-2
  # Optional: user
  user: mqtt
  # Optional: password
  # NOTE: Environment variables that begin with 'FRIGATE_' may be referenced in {}.
  #       eg. password: '{FRIGATE_MQTT_PASSWORD}'
  password: mqtt
  # Optional: interval in seconds for publishing stats (default: shown below)
  stats_interval: 60

# Cameras Config ######################################################################################################
cameras:
# FRONT Camera ########################################################################################################
  voordeur:
    ffmpeg:
     inputs:
        - path: rtsp://admin:pass@192.168.3.20:554/Streaming/Channels/101/
          roles:
            - detect
            - record
    detect:
      enabled: True
      max_disappeared: 2
      width: 2560
      height: 1440
      fps: 6
    snapshots:
      enabled: true
      crop: false
      bounding_box: True
    objects:
      track:
        - person
        - dog
        - cat
    motion:
      mask:
        - '1090,81,1216,0,1382,34,1532,99,1518,259,1426,363,1123,247'
        - '1766,498,1753,669,1778,722,1829,773,1863,825,1906,884,1924,955,1953,1065,2237,1169,2255,1128,2295,1046,2338,946,2378,841,2410,743,2215,647,2190,582,2138,512,2010,402,1859,351'

detectors:
  coral:
    type: edgetpu
    device: ""
    #device: usb
    #model:
     # path: "/custom_model.tflite"

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "128mb" # update for your cameras based on calculation above
    devices:
      #- /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /dev/bus/usb:/dev/bus/usb
      - /etc/localtime:/etc/localtime:ro
      - /frigate/config/config.yml:/config/config.yml
      - /frigate/media:/media/frigate
      - /frigate/db:/db
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "password"

Relevant log output

2023-04-30 18:42:47.114954831  [INFO] Starting Frigate...
2023-04-30 18:42:47.709649341  [2023-04-30 18:42:47] frigate.app                    INFO    : Starting Frigate (0.12.0-da3e197)
2023-04-30 18:42:47.726215058  [2023-04-30 18:42:47] peewee_migrate                 INFO    : Starting migrations
2023-04-30 18:42:47.729330264  [2023-04-30 18:42:47] peewee_migrate                 INFO    : There is nothing to migrate
2023-04-30 18:42:47.737097285  [2023-04-30 18:42:47] frigate.app                    INFO    : Output process started: 217
2023-04-30 18:42:47.742928960  [2023-04-30 18:42:47] frigate.app                    INFO    : Camera processor started for voordeur: 221
2023-04-30 18:42:47.744742566  [2023-04-30 18:42:47] frigate.app                    INFO    : Capture process started for voordeur: 222
2023-04-30 18:42:47.751527544  [2023-04-30 18:42:47] detector.coral                 INFO    : Starting detection process: 216
2023-04-30 18:42:47.751531417  [2023-04-30 18:42:47] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as 
2023-04-30 18:42:47.751674491  [2023-04-30 18:42:47] frigate.detectors.plugins.edgetpu_tfl ERROR   : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.
2023-04-30 18:42:47.751750986  Process detector:coral:
2023-04-30 18:42:47.755081166  Traceback (most recent call last):
2023-04-30 18:42:47.755083896    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 160, in load_delegate
2023-04-30 18:42:47.755084797      delegate = Delegate(library, options)
2023-04-30 18:42:47.755085692    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 119, in __init__
2023-04-30 18:42:47.755089319      raise ValueError(capture.message)
2023-04-30 18:42:47.755096291  ValueError
2023-04-30 18:42:47.755097048  
2023-04-30 18:42:47.755098040  During handling of the above exception, another exception occurred:
2023-04-30 18:42:47.755099111  
2023-04-30 18:42:47.755099858  Traceback (most recent call last):
2023-04-30 18:42:47.755100729    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-04-30 18:42:47.755101474      self.run()
2023-04-30 18:42:47.755102282    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-04-30 18:42:47.755111601      self._target(*self._args, **self._kwargs)
2023-04-30 18:42:47.755112472    File "/opt/frigate/frigate/object_detection.py", line 98, in run_detector
2023-04-30 18:42:47.755113703      object_detector = LocalObjectDetector(detector_config=detector_config)
2023-04-30 18:42:47.755114435    File "/opt/frigate/frigate/object_detection.py", line 52, in __init__
2023-04-30 18:42:47.755119054      self.detect_api = create_detector(detector_config)
2023-04-30 18:42:47.755120093    File "/opt/frigate/frigate/detectors/__init__.py", line 24, in create_detector
2023-04-30 18:42:47.755121103      return api(detector_config)
2023-04-30 18:42:47.755122071    File "/opt/frigate/frigate/detectors/plugins/edgetpu_tfl.py", line 37, in __init__
2023-04-30 18:42:47.755123056      edge_tpu_delegate = load_delegate("libedgetpu.so.1.0", device_config)
2023-04-30 18:42:47.755129079    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 162, in load_delegate
2023-04-30 18:42:47.755130076      raise ValueError('Failed to load delegate from {}\n{}'.format(
2023-04-30 18:42:47.755131073  ValueError: Failed to load delegate from libedgetpu.so.1.0
2023-04-30 18:42:47.755132020  
2023-04-30 18:42:50.344231478  [2023-04-30 18:42:50] frigate.util                   ERROR   : Unable to poll intel GPU stats: No device filter specified and no discrete/integrated i915 devices found
2023-04-30 18:42:50.344234337  
2023-04-30 18:42:51.517257131  [2023-04-30 18:42:51] frigate.video                  ERROR   : voordeur: Unable to read frames from ffmpeg process.
2023-04-30 18:42:51.517430772  [2023-04-30 18:42:51] frigate.video                  ERROR   : voordeur: ffmpeg process is not running. exiting capture thread...

Operating system

Other Linux

Install method

Docker Compose

Coral version

M.2

Any other information that may be helpful

Succesfully seen in ESXi and passed though:

[NickM-27]

You're not passing the device in via docker so that's likely the first problem.

You also haven't defined the device correctly in the Frigate config so that's another problem.

Please read the docs https://docs.frigate.video/configuration/detectors#edge-tpu-detector

[sanderlv]

Can you please help out with the correct passing for docker?

And for frigate config?

I have tried multiple settings but to no avail.

I have read the docs :-)

[NickM-27]

You need to pass /dev/apex_0 in the docker config and you need the device type in frigate config to be pci

[sanderlv]

Got now:

compose:

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "128mb" # update for your cameras based on calculation above
    devices:
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
      - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /frigate/config/config.yml:/config/config.yml
      - /frigate/media:/media/frigate
      - /frigate/db:/db
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "password"

in frigate:

detectors:
  coral_pci:
    type: edgetpu
    device: pci

or this:

detectors:
  coral:
    type: edgetpu
    device: ""

No luck:

frigate  | 2023-04-30 19:04:04.774284742  [2023-04-30 19:04:04] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as
frigate  | 2023-04-30 19:04:04.775248216  [2023-04-30 19:04:04] frigate.detectors.plugins.edgetpu_tfl ERROR   : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.
frigate  | 2023-04-30 19:04:04.775501741  Process detector:coral:

[NickM-27]

Get a shell in the frigate container and run ls /dev/

[sanderlv]

root@102a8322e87d:/opt/frigate# ls /dev/
agpgart        cpu              fb0      hwrng         loop2  mapper  nvram  random  sda3      stdin   tty11  tty17  tty22  tty28  tty33  tty39  tty44  tty5   tty55  tty60  tty9    ttyS13  ttyS19  ttyS24  ttyS3   ttyS7      uinput   vcs3   vcsa2  vcsu1  vfio         vsock
autofs         cpu_dma_latency  fd       input         loop3  mcelog  port   rfkill  sg0       stdout  tty12  tty18  tty23  tty29  tty34  tty4   tty45  tty50  tty56  tty61  ttyS0   ttyS14  ttyS2   ttyS25  ttyS30  ttyS8      urandom  vcs4   vcsa3  vcsu2  vga_arbiter  zero
bsg            cuse             full     kmsg          loop4  mem     ppp    rtc0    shm       tty     tty13  tty19  tty24  tty3   tty35  tty40  tty46  tty51  tty57  tty62  ttyS1   ttyS15  ttyS20  ttyS26  ttyS31  ttyS9      userio   vcs5   vcsa4  vcsu3  vhci         zfs
btrfs-control  dm-0             fuse     loop-control  loop5  mqueue  psaux  sda     snapshot  tty0    tty14  tty2   tty25  tty30  tty36  tty41  tty47  tty52  tty58  tty63  ttyS10  ttyS16  ttyS21  ttyS27  ttyS4   ttyprintk  vcs      vcs6   vcsa5  vcsu4  vhost-net
bus            dri              hidraw0  loop0         loop6  net     ptmx   sda1    snd       tty1    tty15  tty20  tty26  tty31  tty37  tty42  tty48  tty53  tty59  tty7   ttyS11  ttyS17  ttyS22  ttyS28  ttyS5   udmabuf    vcs1     vcsa   vcsa6  vcsu5  vhost-vsock
core           ecryptfs         hpet     loop1         loop7  null    pts    sda2    stderr    tty10   tty16  tty21  tty27  tty32  tty38  tty43  tty49  tty54  tty6   tty8   ttyS12  ttyS18  ttyS23  ttyS29  ttyS6   uhid       vcs2     vcsa1  vcsu   vcsu6  vmci
root@102a8322e87d:/opt/frigate# 

[NickM-27]

So the coral isn't being passed in, means the coral device isn't being passed from the host OS to whatever is running docker.

[sanderlv]

It is an Ubuntu VM:

ubuntu@frigate-02:~$ lspci
00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 01)
00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 01)
00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 08)
00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
00:07.7 System peripheral: VMware Virtual Machine Communication Interface (rev 10)
00:0f.0 VGA compatible controller: VMware SVGA II Adapter
02:00.0 Serial Attached SCSI controller: VMware PVSCSI SCSI Controller (rev 02)
02:01.0 USB controller: VMware USB3 xHCI 1.0 Controller
02:02.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
02:03.0 System peripheral: Global Unichip Corp. Coral Edge TPU
02:04.0 SATA controller: VMware SATA AHCI controller
ubuntu@frigate-02:~$

with just normal docker?

[NickM-27]

The VM may need the coral driver installed as well

[sanderlv]

Installed following: https://coral.ai/docs/m2/get-started/#2a-on-linux

but still no luck... any other tip :-)?

[sanderlv]

this is strange and above my knowledge level:

ubuntu@frigate-02:/frigate/compose$ lspci -nn | grep 089a
02:03.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a]
ubuntu@frigate-02:/frigate/compose$ ls /dev/apex_0
ls: cannot access '/dev/apex_0': No such file or directory
ubuntu@frigate-02:/frigate/compose$

Same problem here:
google-coral/edgetpu#407

I had to "enable" secure boot in the VM, could not pass.... had to enter a password (did it) and then on boot on the console I Selected the first option (boot))...

How do I disable secure boot on a linux (VM)
EDIT: I disables it in ESXi, now it did not ask any on startup and started right up.

But still:

ubuntu@frigate-02:~$ ls /dev/apex_0
ls: cannot access '/dev/apex_0': No such file or directory
ubuntu@frigate-02:~$

[eogmau]

Anything other than usb coral won't work in esxi as you need to install the coral drivers on esxi which do not exist. Proxmox allows to install drivers as it's Debian based.

[sanderlv]

I do not have a USB coral, it is an M.2 Coral. the device ID is correct.

USB Coral is another problem.

What I am trying to say is that only usb version of coral works with esxi. M.2 coral requires a driver to be I stalled on host esxi and that driver doesn't exist.

Ah sorry, misread. I believe the USB does not work at all in ESXi... I have it running in Proxmox yes, but I need it to run in ESXi as well...

[eogmau]

What I am trying to say is that only usb version of coral works with esxi. M.2 coral requires a driver to be installed on host esxi and that driver doesn't exist.

[sanderlv]

What I am trying to say is that only usb version of coral works with esxi. M.2 coral requires a driver to be installed on host esxi and that driver doesn't exist.

Why should/couldn't/doesn't it work in ESXi? All outputs in the VM look fine up until the

ls: cannot access '/dev/apex_0': No such file or directory

[NickM-27]

Do you see /dev/apex_0 on the host? Have you installed the driver on the host? if not then it won't work, and I believe what @eogmau is saying is correct

[eogmau]

M.2 coral it's only possible with Proxmox from my knowledge because there is no coral PCI driver compiled for esxi, the driver exist for Debian based distros.

[eogmau]

What I am trying to say is that only usb version of coral works with esxi. M.2 coral requires a driver to be installed on host esxi and that driver doesn't exist.

Why should/couldn't/doesn't it work in ESXi? All outputs in the VM look fine up until the

ls: cannot access '/dev/apex_0': No such file or directory

You will need a driver installed for m.2 coral on esxi. The driver doesn't exists for esxi.

[sanderlv]

The host is an ESXi 8.x server. I do not dare to touch or install a driver in there...

No I do not see it...

[root@localhost:~] ls /dev/apex_0
ls: /dev/apex_0: No such file or directory
[root@localhost:~]

But... as it is only "passed through" why wouldn't it work in an Ubuntu VM?

[NickM-27]

The host is an ESXi 8.x server. I do not dare to touch or install a driver in there...

No I do not see it...

[root@localhost:~] ls /dev/apex_0
ls: /dev/apex_0: No such file or directory
[root@localhost:~]

But... as it is only "passed through" why wouldn't it work in an Ubuntu VM?

because the driver has to exist at every layer, since your host OS is the layer that actually integrates with the hardware itself

[eogmau]

You don't see it because exsi cannot talk to it as it requires a driver to be installed on esxi.
No one up to date was able to use m.2 coral with esxi.

[sanderlv]

That's (another) pity...

Is "the driver" preinstalled on proxmox? I never did that there... (or is it not needed for usb?).

[NickM-27]

The USB coral has an internal driver that loads when it is run.

The PCI driver is not preinstalled in proxmox but it works becuse proxmox is debian based.

[fliespl]

@sanderlv slighthly out of topic. Did you use it on m.2. 40x22 port inside nuc12? I am planning to buy one as well, but would like to confirm first it works on this slot first.

[sanderlv]

@sanderlv slighthly out of topic. Did you use it on m.2. 40x22 port inside nuc12? I am planning to buy one as well, but would like to confirm first it works on this slot first.

It technically works. You will however need a circular saw (don't ask :-) )

EDIT: See how desperately I want this to work in VMware ESXi (whaaha)

[fliespl]

@sanderlv thanks much! You should've posted the other one as well as a "warning" (just assuming based on your answer something went really wrong first time ;) )

[sanderlv]

What other one? Nothing went wrong... only thing going wrong is VMware ESXi (8.x) is not wanting to work with Coral. Nor USB, nor M.2...

[fliespl]

@sanderlv Ah, got it now :) I thought you mentioned "circular saw" need, cause you broke the first one by using other tools! :D

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.