[Detector Support]: Frigate tensorrt image truenas scale. Not generating trt models. #10372

dinezttv · 2024-03-11T08:42:29Z

dinezttv
Mar 11, 2024

Describe the problem you are having

Trying to setup frigate on truenas scale with the tensorrt image.
Tensorrt image provided by truechart.
Only thing i need is the trt files to be generated and downloaded but the container gets stuck and wont deploy.

Version

0.13.2

Frigate config file

mqtt:
  enabled: false
detectors:
  tensorrt:
    type: tensorrt
    device: 0 #This is the default, select the first GPU

model:
  path: /config/model_cache/tensorrt/yolov7-320.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 320
  height: 320
cameras:
  Cam_Front:
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://admin:@10.10.10.20:554/h264Preview_01_sub
          roles:
            - detect
        - path: rtsp://admin:@10.10.10.20:554/h265Preview_01_main
          roles:
            - record
    rtmp:
      enabled: False
    detect:
      width: 640
      height: 480
      fps: 5
    motion:
      mask:
        - 0,609,432,333,782,147,1031,0,329,0,174,0,115,0,0,0,0,27,0,86,0,210
    objects:
      track:
        - person
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: True
      retain:
        default: 2
  Cam_Back_Right:
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://admin:@10.10.10.21:554/h264Preview_01_sub
          roles:
            - detect
        - path: rtsp://admin:@10.10.10.21:554/h265Preview_01_main
          roles:
            - record
    rtmp:
      enabled: False
    detect:
      width: 640
      height: 480
      fps: 5
    motion:
      mask:
        - 1591,0,1920,0,1920,118
    objects:
      track:
        - person
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: True
      retain:
        default: 2
  Cam_Back_Left:
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://admin:.@10.10.10.22:554/h264Preview_01_sub
          roles:
            - detect
        - path: rtsp://admin:.@10.10.10.22:554/h265Preview_01_main
          roles:
            - record
    rtmp:
      enabled: False
    detect:
      width: 640
      height: 480
      fps: 5
    motion:
      mask:
        - 1591,0,1920,0,1920,118
    objects:
      track:
        - person
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: True
      retain:
        default: 2
  Cam_Back_Left:
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp://admin:.@10.10.10.22:554/h264Preview_01_sub
          roles:
            - detect
        - path: rtsp://admin:.@10.10.10.22:554/h265Preview_01_main
          roles:
            - record
    rtmp:
      enabled: False
    detect:
      width: 640
      height: 480
      fps: 5
    motion:
      mask:
        - 487,0,0,0,0,620,158,271,364,265
    objects:
      track:
        - person
    snapshots:
      enabled: True
      timestamp: True
      bounding_box: True
      retain:
        default: 2
database:
  path: /config/frigate.db
record:
  enabled: True
  retain:
    days: 7
    mode: motion
  events:
    retain:
      default: 14
      mode: active_object

docker-compose file or Docker CLI command

N/A

Relevant log output

2024-03-11T10:14:28.909931078+02:00 s6-rc: info: service s6rc-fdholder: starting
2024-03-11T10:14:28.909988318+02:00 s6-rc: info: service s6rc-oneshot-runner: starting
2024-03-11T10:14:28.911335911+02:00 s6-rc: info: service s6rc-oneshot-runner successfully started
2024-03-11T10:14:28.911449091+02:00 s6-rc: info: service fix-attrs: starting
2024-03-11T10:14:28.912329482+02:00 s6-rc: info: service s6rc-fdholder successfully started
2024-03-11T10:14:28.914440276+02:00 s6-rc: info: service fix-attrs successfully started
2024-03-11T10:14:28.914495216+02:00 s6-rc: info: service legacy-cont-init: starting
2024-03-11T10:14:28.917075411+02:00 s6-rc: info: service legacy-cont-init successfully started
2024-03-11T10:14:28.917139531+02:00 s6-rc: info: service trt-model-prepare: starting
2024-03-11T10:14:28.917201951+02:00 s6-rc: info: service log-prepare: starting
2024-03-11T10:14:28.924796313+02:00 s6-rc: info: service log-prepare successfully started
2024-03-11T10:14:28.924798173+02:00 Generating the following TRT Models: yolov7-320
2024-03-11T10:14:28.924837353+02:00 Downloading yolo weights
2024-03-11T10:14:28.924861523+02:00 s6-rc: info: service nginx-log: starting
2024-03-11T10:14:28.924952693+02:00 s6-rc: info: service go2rtc-log: starting
2024-03-11T10:14:28.925045343+02:00 s6-rc: info: service frigate-log: starting
2024-03-11T10:14:28.925682044+02:00 s6-rc: info: service nginx-log successfully started
2024-03-11T10:14:28.925728995+02:00 s6-rc: info: service go2rtc-log successfully started
2024-03-11T10:14:28.925794615+02:00 s6-rc: info: service go2rtc: starting
2024-03-11T10:14:28.925823185+02:00 s6-rc: info: service frigate-log successfully started
2024-03-11T10:14:28.926468156+02:00 s6-rc: info: service go2rtc successfully started
2024-03-11T10:14:28.926524656+02:00 s6-rc: info: service go2rtc-healthcheck: starting
2024-03-11T10:14:28.927353347+02:00 s6-rc: info: service go2rtc-healthcheck successfully started
2024-03-11T10:14:28.948178022+02:00 2024-03-11 08:14:28.948046121  [INFO] Removing stale config from last run...
2024-03-11T10:14:28.948701462+02:00 2024-03-11 08:14:28.948682862  [INFO] Preparing new go2rtc config...
2024-03-11T10:14:29.236120328+02:00 2024-03-11 08:14:29.236022838  [INFO] Starting go2rtc...
2024-03-11T10:14:29.301808867+02:00 2024-03-11 08:14:29.301711946  10:14:29.301 INF go2rtc version 1.8.4 linux/amd64
2024-03-11T10:14:29.302150297+02:00 2024-03-11 08:14:29.302105197  10:14:29.302 INF [api] listen addr=:1984
2024-03-11T10:14:29.302195767+02:00 2024-03-11 08:14:29.302154337  10:14:29.302 INF [rtsp] listen addr=:8554
2024-03-11T10:14:29.302336707+02:00 2024-03-11 08:14:29.302317417  10:14:29.302 INF [webrtc] listen addr=:8555
2024-03-11T10:14:34.120435970+02:00 
2024-03-11T10:14:34.120854560+02:00 Creating yolov7-320.cfg and yolov7-320.weights
2024-03-11T10:14:34.124393107+02:00 
2024-03-11T10:14:34.124406607+02:00 Done.
2024-03-11T10:14:38.930107229+02:00 2024-03-11 08:14:38.930015109  [INFO] Starting go2rtc healthcheck service...
2024-03-11T10:14:41.231109406+02:00 
2024-03-11T10:14:41.231199126+02:00 Generating yolov7-320.trt. This may take a few minutes.
2024-03-11T10:14:41.231204966+02:00 
2024-03-11T10:14:44.268480362+02:00 /usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py:147: DeprecationWarning: Use network created with NetworkDefinitionCreationFlag::EXPLICIT_BATCH flag instead.
2024-03-11T10:14:44.268564832+02:00   builder.max_batch_size = MAX_BATCH_SIZE
2024-03-11T10:14:44.268896983+02:00 /usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py:149: DeprecationWarning: Use set_memory_pool_limit instead.
2024-03-11T10:14:44.268942173+02:00   config.max_workspace_size = 1 << 30
2024-03-11T10:14:44.269076563+02:00 /usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py:172: DeprecationWarning: Use build_serialized_network instead.
2024-03-11T10:14:44.269100833+02:00   engine = builder.build_engine(network, config)

Operating system

Other Linux

Install method

Docker Compose

Coral version

Other

Any other information that may be helpful

OS Version:TrueNAS-SCALE-23.10.2
Cpu: AMD 3950x
Ram: 32Gig
SSD: NVME

NickM-27 · 2024-03-11T13:37:34Z

NickM-27
Mar 11, 2024
Collaborator Sponsor

What tells you that it is stuck and not just taking a while?

1 reply

dinezttv Mar 11, 2024
Author

i guess the Kubernetes events. Startup probe failed: dial tcp 10.10.10.1:5000: connect: connection refused
and second i have it up on deploying for over 3 hours..

prjcarr · 2024-03-12T03:52:01Z

prjcarr
Mar 12, 2024

I have having this exact same issue. Not sure if its a frigate issue or a truecharts issue. I can run the standard frigate image fine using the same truecharts app, and it passes through my gpu without issue, but when choosing the tensorrt image I get:

Startup probe failed: dial tcp 172.16.1.33:5000: connect: connection refused

Looking at the pod logs I see:

2024-03-12 14:45:10.769021+11:002024-03-12 03:45:10.768851354  [INFO] Starting go2rtc healthcheck service...
2024-03-12 14:45:15.415066+11:002024-03-12T14:45:15.415066555+11:00
2024-03-12 14:45:15.415944+11:00Creating yolov7-320.cfg and yolov7-320.weights
2024-03-12 14:45:15.420299+11:002024-03-12T14:45:15.420299167+11:00
2024-03-12 14:45:15.420328+11:00Done.
2024-03-12 14:45:18.082050+11:002024-03-12T14:45:18.082050850+11:00
2024-03-12 14:45:18.082085+11:00Generating yolov7-320.trt. This may take a few minutes.
2024-03-12 14:45:18.082088+11:002024-03-12T14:45:18.082088715+11:00
2024-03-12 14:45:19.413711+11:00/usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py:147: DeprecationWarning: Use network created with NetworkDefinitionCreationFlag::EXPLICIT_BATCH flag instead.
2024-03-12 14:45:19.413748+11:00builder.max_batch_size = MAX_BATCH_SIZE
2024-03-12 14:45:19.413750+11:00/usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py:149: DeprecationWarning: Use set_memory_pool_limit instead.
2024-03-12 14:45:19.413752+11:00config.max_workspace_size = 1 << 30
2024-03-12 14:45:19.413754+11:00/usr/local/src/tensorrt_demos/yolo/onnx_to_tensorrt.py:172: DeprecationWarning: Use build_serialized_network instead.
2024-03-12 14:45:19.413758+11:00engine = builder.build_engine(network, config)

Also of note might be, when I run

sudo k3s kubectl get pods --all-namespaces | grep ix-frigate
NAMESPACE        NAME                                           READY   STATUS                     RESTARTS        AGE
ix-frigate       frigate-798554bc75-d6kkw                       0/1     Running                    14 (3m1s ago)   89m

Perhaps the pod is being restarted too quickly when the probe fails?

1 reply

dinezttv Mar 12, 2024
Author

Somehow @StevenMcElligott got it to work.
Check the truechart discord under scale apps search the thread with name "Frigate Config Reolink 510A Nvidia 1650 Super Encoding/Decoding"
Meanwhile we have the same issues.. And i still don't how to fix it.

RichE85 · 2024-04-02T13:39:04Z

RichE85
Apr 2, 2024

Hey I am having the same issue. For some reason when generating yolo models the python process keeps restarting after getting to about 760MiB of data and keeps looping like this for hours(not sure if this is expected behavior). I was able to generate the tiny models without this issue (yolov7-tiny-288 and
yolov7-tiny-416) but not the full yolov7-640 or 320. Do we just need to let it run longer? 12 hours+? I'm running this with Quadro RTX 4000 btw.

2 replies

dinezttv Apr 2, 2024
Author

Solution by @prjcarr
Launch the truecharts custom app - use repository ghcr.io/blakeblackshear/frigate and tag stable-tensorrt
Disable all probes in the custom app
Setup the service port with what ever you want, its only temporary
Just use a host path (set up a temp folder) for the config and mount as /config, do the same for /media/frigate
For security context I had to runAsUser 0 and runAsGroup 0
I also set privileged mode in advanced settings. Not sure if this makes a difference
Add your GPU
Start the pod and wait for the trt file to be built
Shell into the frigate pod and copy your trt from your temp host path into the PVC config\model_cache\tensorrt\8.5.3 directory
Start the frigate pod
Delete the custom app pod

RichE85 Apr 2, 2024

Thanks! Worked immediately!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Detector Support]: Frigate tensorrt image truenas scale. Not generating trt models. #10372

{{title}}

Replies: 3 comments 4 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[Detector Support]: Frigate tensorrt image truenas scale. Not generating trt models. #10372

dinezttv Mar 11, 2024

Describe the problem you are having

Version

Frigate config file

docker-compose file or Docker CLI command

Relevant log output

Operating system

Install method

Coral version

Any other information that may be helpful

Replies: 3 comments · 4 replies

NickM-27 Mar 11, 2024 Collaborator Sponsor

dinezttv Mar 11, 2024 Author

prjcarr Mar 12, 2024

dinezttv Mar 12, 2024 Author

RichE85 Apr 2, 2024

dinezttv Apr 2, 2024 Author

RichE85 Apr 2, 2024

dinezttv
Mar 11, 2024

Replies: 3 comments 4 replies

NickM-27
Mar 11, 2024
Collaborator Sponsor

dinezttv Mar 11, 2024
Author

prjcarr
Mar 12, 2024

dinezttv Mar 12, 2024
Author

RichE85
Apr 2, 2024

dinezttv Apr 2, 2024
Author