Is it a good idea to run Frigate under TrueNAS

[DougLorenz]

I have been trying to get Frigate to run stable on TrueNAS for a couple of weeks now, and have been having numerous problems. I am migrating over from running Blue Iris on a virtual Windows machine, and initially had great results with Frigate, but have been running into difficulties.

The system should be powerful enough. It's running on a Ryzen 5600G with 32G of memory and TrueNAS Scale. A number of other apps are running on the system, but nothing demanding. I currently have 8 IP cameras attached. I have a PCIe card with a dual TPU Coral installed, but I've never been able to access it. I also have a USB Coral attached, and I have gotten it to work, but it tends to crash unrecoverably within 24 hours. Running with CPU detectors the system seems to function, but I've also gotten some annoying crashes.

Is there anything inherently wrong with running Frigate as an app under TrueNAS scale? Aside from the fact that there seems to be no way for me to access my PCIe Coral device(s) from inside a TrueNAS app, is there something that is causing unrecoverable ffmpeg crashes?

Or is there a better way to install Frigate under TrueNAS? I'm currently using the TrueCharts app, but there are other catalogs out there...

[NickM-27]

Unfortunately there are some reports that TrueNAS scale makes things difficult

a user details that they were able to get it to work https://www.reddit.com/r/frigate_nvr/comments/11yaykx/using_coral_m2_version_with_truenas_scale/

[DougLorenz]

Yeah, I've read a couple cases where people have forced TrueNAS to load PCIe drivers. One down side to that approach is the drivers get trashed whenever the system updates itself.

But the system is also crashing from regular ffmpeg errors. In this case the system had been running for about four hours and glitched a few times on a couple of different cameras. It ultimately recovered, but I've seen a few times over the last week where eventually the whole app would crash after a series of these ffmpeg errors.

2023-09-11 22:12:13.458128707  [INFO] Starting Frigate...
2023-09-11 22:12:14.241829001  [2023-09-11 15:12:14] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-09-11 22:12:14.284681296  [2023-09-11 15:12:14] frigate.app                    INFO    : Creating directory: /tmp/cache
2023-09-11 22:12:14.285822152  [2023-09-11 15:12:14] peewee_migrate                 INFO    : Starting migrations
2023-09-11 22:12:14.288399215  [2023-09-11 15:12:14] peewee_migrate                 INFO    : There is nothing to migrate
2023-09-11 22:12:14.296401073  [2023-09-11 15:12:14] detector.cpu1                  INFO    : Starting detection process: 579
2023-09-11 22:12:14.297698390  [2023-09-11 15:12:14] frigate.detectors              WARNING : CPU detectors are not recommended and should only be used for testing or for trial purposes.
2023-09-11 22:12:14.298120642  [2023-09-11 15:12:14] detector.cpu2                  INFO    : Starting detection process: 581
2023-09-11 22:12:14.298757505  [2023-09-11 15:12:14] frigate.app                    INFO    : Output process started: 583
2023-09-11 22:12:14.299454049  [2023-09-11 15:12:14] frigate.detectors              WARNING : CPU detectors are not recommended and should only be used for testing or for trial purposes.
2023-09-11 22:12:14.301620260  [2023-09-11 15:12:14] frigate.app                    INFO    : Camera processor started for garage: 586
2023-09-11 22:12:14.303449549  [2023-09-11 15:12:14] frigate.app                    INFO    : Camera processor started for front_porch: 588
2023-09-11 22:12:14.305565470  [2023-09-11 15:12:14] frigate.app                    INFO    : Camera processor started for east_side_yard: 590
2023-09-11 22:12:14.307867741  [2023-09-11 15:12:14] frigate.app                    INFO    : Camera processor started for driveway: 591
2023-09-11 22:12:14.310369584  [2023-09-11 15:12:14] frigate.app                    INFO    : Camera processor started for front_yard_ptz: 593
2023-09-11 22:12:14.312865667  [2023-09-11 15:12:14] frigate.app                    INFO    : Camera processor started for front_gate_doorbell: 594
2023-09-11 22:12:14.315784361  [2023-09-11 15:12:14] frigate.app                    INFO    : Camera processor started for front_doorbell: 596
2023-09-11 22:12:14.318910387  [2023-09-11 15:12:14] frigate.app                    INFO    : Camera processor started for den: 598
2023-09-11 22:12:14.321193309  [2023-09-11 15:12:14] frigate.app                    INFO    : Capture process started for garage: 599
2023-09-11 22:12:14.324569484  [2023-09-11 15:12:14] frigate.app                    INFO    : Capture process started for front_porch: 603
2023-09-11 22:12:14.327145117  [2023-09-11 15:12:14] frigate.app                    INFO    : Capture process started for east_side_yard: 608
2023-09-11 22:12:14.330784245  [2023-09-11 15:12:14] frigate.app                    INFO    : Capture process started for driveway: 616
2023-09-11 22:12:14.335138207  [2023-09-11 15:12:14] frigate.app                    INFO    : Capture process started for front_yard_ptz: 623
2023-09-11 22:12:14.338321963  [2023-09-11 15:12:14] frigate.app                    INFO    : Capture process started for front_gate_doorbell: 631
2023-09-11 22:12:14.343501480  [2023-09-11 15:12:14] frigate.app                    INFO    : Capture process started for front_doorbell: 647
2023-09-11 22:12:14.346990107  [2023-09-11 15:12:14] frigate.app                    INFO    : Capture process started for den: 654
2023-09-12 02:31:45.235158600  [2023-09-11 19:31:45] watchdog.garage                INFO    : No frames received from garage in 20 seconds. Exiting ffmpeg...
2023-09-12 02:31:45.235234971  [2023-09-11 19:31:45] watchdog.garage                INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-12 02:31:45.311008664  [2023-09-11 19:31:45] frigate.video                  ERROR   : garage: Unable to read frames from ffmpeg process.
2023-09-12 02:31:45.311158735  [2023-09-11 19:31:45] frigate.video                  ERROR   : garage: ffmpeg process is not running. exiting capture thread...
2023-09-12 02:31:45.339822875  [2023-09-11 19:31:45] watchdog.front_doorbell        INFO    : No frames received from front_doorbell in 20 seconds. Exiting ffmpeg...
2023-09-12 02:31:45.339888515  [2023-09-11 19:31:45] watchdog.front_doorbell        INFO    : Waiting for ffmpeg to exit gracefully...
2023-09-12 02:31:45.373096474  [2023-09-11 19:31:45] frigate.video                  ERROR   : front_doorbell: Unable to read frames from ffmpeg process.
2023-09-12 02:31:45.373215335  [2023-09-11 19:31:45] frigate.video                  ERROR   : front_doorbell: ffmpeg process is not running. exiting capture thread...
2023-09-12 02:31:55.312915485  [2023-09-11 19:31:55] watchdog.garage                ERROR   : Ffmpeg process crashed unexpectedly for garage.
2023-09-12 02:31:55.312971465  [2023-09-11 19:31:55] watchdog.garage                ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-12 02:31:55.313029045  [2023-09-11 19:31:55] ffmpeg.garage.detect           ERROR   : [rtsp @ 0x562cde6a8c40] RTP: PT=60: bad cseq e5f2 expected=05b3
2023-09-12 02:31:55.313082066  [2023-09-11 19:31:55] ffmpeg.garage.detect           ERROR   : [rtsp @ 0x562cde6a8c40] RTP: PT=61: bad cseq e5f2 expected=da10
2023-09-12 02:31:55.378852629  [2023-09-11 19:31:55] watchdog.front_doorbell        ERROR   : Ffmpeg process crashed unexpectedly for front_doorbell.
2023-09-12 02:31:55.378966740  [2023-09-11 19:31:55] watchdog.front_doorbell        ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2023-09-12 02:31:55.379040330  [2023-09-11 19:31:55] ffmpeg.front_doorbell.detect   ERROR   : [rtsp @ 0x55a46fb6cc40] RTP: PT=60: bad cseq 0e86 expected=3f3f
2023-09-12 02:31:55.379090381  [2023-09-11 19:31:55] ffmpeg.front_doorbell.detect   ERROR   : [rtsp @ 0x55a46fb6cc40] RTP: PT=61: bad cseq 0e85 expected=da04
2023-09-12 02:32:45.495011476  [2023-09-11 19:32:45] watchdog.east_side_yard        ERROR   : No new recording segments were created for east_side_yard in the last 120s. restarting the ffmpeg record process...
2023-09-12 02:32:45.495076147  [2023-09-11 19:32:45] watchdog.east_side_yard        INFO    : Terminating the existing ffmpeg process...
2023-09-12 02:32:45.495135037  [2023-09-11 19:32:45] watchdog.east_side_yard        INFO    : Waiting for ffmpeg to exit gracefully...

I haven't yet heard any reports of people getting this to run properly on TrueNAS as an application, even though TrueCharts is listing their app in the 'stable' catalog.

[NickM-27]

There are a number of users that have been running truenas without these types of issues, some created issues when 0.12 came out saying that things were working well except an issue popped up due to the weird way that truenas complains that frigate has Nvidia env variables set when no nvidia GPU is installed

[DougLorenz]

Ok, I made the mods to the underlying TrueNAS, and I now have three TPUs that appear to work. (The PCI dual TPU unit appears to be a good deal faster than the USB!) I'm guessing that's more than enough muscle for eight cameras...

Startup looks good:

2023-09-12 06:22:01.527577002  [INFO] Starting Frigate...
2023-09-12 06:22:45.102144931  [2023-09-11 23:22:45] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-09-12 06:22:45.177591577  [2023-09-11 23:22:45] frigate.app                    INFO    : Creating directory: /tmp/cache
2023-09-12 06:22:45.178797101  [2023-09-11 23:22:45] peewee_migrate                 INFO    : Starting migrations
2023-09-12 06:22:45.442459791  [2023-09-11 23:22:45] peewee_migrate                 INFO    : There is nothing to migrate
2023-09-12 06:22:45.490243252  [2023-09-11 23:22:45] detector.coral_usb             INFO    : Starting detection process: 588
2023-09-12 06:22:45.491633827  [2023-09-11 23:22:45] detector.coral_pci1            INFO    : Starting detection process: 590
2023-09-12 06:22:45.494162095  [2023-09-11 23:22:45] detector.coral_pci2            INFO    : Starting detection process: 592
2023-09-12 06:22:45.583598674  [2023-09-11 23:22:45] frigate.app                    INFO    : Output process started: 594
2023-09-12 06:22:45.583667925  [2023-09-11 23:22:45] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci:1
2023-09-12 06:22:46.181309265  [2023-09-11 23:22:45] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci:0
2023-09-12 06:22:48.979547556  [2023-09-11 23:22:45] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-09-12 06:22:48.982082975  [2023-09-11 23:22:45] frigate.app                    INFO    : Camera processor started for garage: 597
2023-09-12 06:22:48.982178486  [2023-09-11 23:22:48] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-09-12 06:22:48.982244327  [2023-09-11 23:22:45] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-09-12 06:22:48.982298598  [2023-09-11 23:22:45] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2023-09-12 06:22:48.982359668  [2023-09-11 23:22:45] frigate.app                    INFO    : Camera processor started for front_porch: 599
2023-09-12 06:22:48.982402449  [2023-09-11 23:22:45] frigate.app                    INFO    : Camera processor started for east_side_yard: 600
2023-09-12 06:22:48.982449919  [2023-09-11 23:22:45] frigate.app                    INFO    : Camera processor started for driveway: 601
2023-09-12 06:22:48.982501690  [2023-09-11 23:22:45] frigate.app                    INFO    : Camera processor started for front_yard_ptz: 602
2023-09-12 06:22:48.982550831  [2023-09-11 23:22:45] frigate.app                    INFO    : Camera processor started for front_gate_doorbell: 603
2023-09-12 06:22:48.982602801  [2023-09-11 23:22:45] frigate.app                    INFO    : Camera processor started for front_doorbell: 604
2023-09-12 06:22:48.982649162  [2023-09-11 23:22:45] frigate.app                    INFO    : Camera processor started for den: 605
2023-09-12 06:22:48.982699692  [2023-09-11 23:22:45] frigate.app                    INFO    : Capture process started for garage: 606
2023-09-12 06:22:48.982744593  [2023-09-11 23:22:45] frigate.app                    INFO    : Capture process started for front_porch: 609
2023-09-12 06:22:48.982794643  [2023-09-11 23:22:45] frigate.app                    INFO    : Capture process started for east_side_yard: 613
2023-09-12 06:22:48.982854914  [2023-09-11 23:22:45] frigate.app                    INFO    : Capture process started for driveway: 617
2023-09-12 06:22:48.982905665  [2023-09-11 23:22:45] frigate.app                    INFO    : Capture process started for front_yard_ptz: 621
2023-09-12 06:22:48.982952095  [2023-09-11 23:22:45] frigate.app                    INFO    : Capture process started for front_gate_doorbell: 625
2023-09-12 06:22:48.983005106  [2023-09-11 23:22:45] frigate.app                    INFO    : Capture process started for front_doorbell: 629
2023-09-12 06:22:48.983054876  [2023-09-11 23:22:45] frigate.app                    INFO    : Capture process started for den: 633

Let's see how long it lasts before everything goes to hell...

[keionbis]

Ive been fighting with this all day and havent found a way to actually pass my pci coral TPU into frigate. Did you have to do anything to actually pass through the PCI device?

[DougLorenz]

I've disconnected the USB Coral, and have been running with just the PCIe dual TPU Coral for the past couple of days, and it seems to be stable.

I am getting some occasional ffmpeg crashes from one camera or another every few hours, but it doesn't last more than a minute or so and the ffmpeg process appears to recover. These could be just glitches from the cameras which cause ffmpeg to fail. Is this common for other people?

The big point about my TrueNAS installation is the understanding that it is going to break. Eventually TrueNAS will send out an update that erases the apex drivers and brings down my frigate installation until I manually go in and reinstall them. This is not a long-term solution.

[NickM-27]

These could be just glitches from the cameras which cause ffmpeg to fail. Is this common for other people?

depends on the type of camera. It is pretty common for reolink and wifi cameras in general, but should not be happening with recommended cameras (dahua, hikvisikon, amcrest)

[DougLorenz]

Doesn't seem to be limited to wifi and all but one of my cameras are Amcrest.

Two days isn't enough time for reliable data. It could be freaking sunspots for all I know... The system is working though, and that's a good start.

[T3chArmy]

Posted this in the other thread, my Frigate is running as a Docker within TrueNAS Scale 23.10-beta1.

Frigate version: 0.13.0-9185753
Installation type : Docker
Base OS: TrueNAS 23.10-Beta1
Hardware: Intel i7-8700 w/ USB Coral
Model: default edgetpu_model.tflite 320x320
Cameras: 2x 2592x1944 (20 FPS, 5 FPS Detect) cameras, and 1x 1920x1080 camera all using RTSP through Go2RTC
Labels: person, car, dog, cat
Inference speed: 8ms

[T3chArmy]

(Sorry running Scale 23.10, UI is slightly updated, but all the same options are there in Scale 22.12.3)

[DougLorenz]

I'm running Scale 23.10 as well. Bluefin is more trouble than it's worth, seems less buggy to run the beta Cobia...

Definitely for comparison purposes I'll use 0.12 for my first Docker test. Gotta try to compare apples to apples as best as I can.

[DougLorenz]

Here's something to consider as I am currently learning definitions... TrueNAS Scale 23.10 Cobia is not running Docker, it is running containerd. Does that change anything?

[T3chArmy]

@DougLorenz containerd is "Docker". containerd is the actual runtime, where as Docker as whole is the management tools for containerd.

[DougLorenz]

Thanks. I'm making it more complicated for myself than it needs to be.