Upgrade TensorRT to 8.5.3

[NateMeyer]

This pull request updates the TensorRT libraries to 8.5.3 in order to support the latest GPUs from Nvidia. Should address Issue #6666

This update also is able to do the model conversion from within the frigate image, which allows us to integrate it into the startup scripts.

Tasks:

• Update Libraries
• Update scripts to convert models locally
• Setup scripts to convert models automatically
• Update documentation
• Testing

[netlify]

✅ Deploy Preview for frigate-docs canceled.

Name	Link
🔨 Latest commit	3b21f86
🔍 Latest deploy log	https://app.netlify.com/sites/frigate-docs/deploys/64a6425907f0ef00097e39cf

[NateMeyer]

I have a test image pushed if anyone wants to help verify this works on new (4000 series) and old (900 series) GPUs. This change also streamlines the model generation process, and no pre-conversion steps are necessary, just add the model you want to use as an environment variable in your docker-compose.yml.

Using image ghcr.io/natemeyer/frigate:trt-8.5-7761a1a-tensorrt, add the following to your docker-compose file:

services:
  frigate:
    image: ghcr.io/natemeyer/frigate:trt-8.5-7761a1a-tensorrt
    environment:
      - USE_FP16=false
      - YOLO_MODELS=yolov7-tiny-416

The trt-models folder no longer needs to be mapped as a volume into the container. The model in the YOLO_MODELS variable will be downloaded, converted, and saved into the path /config/model_cache/tensorrt when the container starts up.

You will also need to add this new path into the config.yml to point to the converted model.

model:
  path: /config/model_cache/tensorrt/yolov7-tiny-416.trt

Note: I had issues using the latest driver on my 1080ti, the system would crash when running object detection. I rolled back to the 531.68 driver and it is working fine. Not sure if this issue extends to any other cards or setups, or if I'm the lucky one.

[doctorttt]

Hi Nate. I followed your instructions above and ghcr.io/natemeyer/frigate:trt-8.5-7761a1a-tensorrt works on my 4060ti with yolov7-tiny-416. The whole installation process was smooth and flawless; also no errors in log. Please see the screenshots pasted here for your reference. By the way, are the other yolo7 models available for testing: yolov7x-320, yolov7x-640, yolov7-640, yolov7-320. Also, are yolo8 models available for testing? Thanks

detectors:
  tensorrt:
    type: tensorrt
    device: 0 

model:
  path: /config/model_cache/tensorrt/yolov7-tiny-416.trt 
  input_tensor: nchw
  input_pixel_format: rgb
  width: 416
  height: 416

[NateMeyer]

Yes, all the same models should be available as before. Just set them in the YOLO_MODELS variable and point to the model in the config file. You can use a comma-separated list to convert multiple models just like the previous process. The yolo example repo from yeahme49 didn't include any yolov8 models to my knowledge.

[doctorttt]

Yes, all the same models should be available as before. Just set them in the YOLO_MODELS variable and point to the model in the config file. You can use a comma-separated list to convert multiple models just like the previous process. The yolo example repo from yeahme49 didn't include any yolov8 models to my knowledge.

I tried the other yolo7 models yolov7x-320, yolov7x-640, yolov7-640, yolov7-320 but couldn't get them to work. I'm stuck with unable to start service trt-model-prepare error message. Please advise. Thanks Nate

[NateMeyer]

@doctorttt Thanks for the heads up. I think the conversion script was too verbose, so I redirected the stdout to /dev/null. Give it a try with the latest update. Pushing a new image, ghcr.io/natemeyer/frigate:trt-8.5-3b21f86-tensorrt. If you see this issue again, I'll need more of the log to see what it's complaining about.

[doctorttt]

ghcr.io/natemeyer/frigate:trt-8.5-3b21f86-tensorrt

@NateMeyer ghcr.io/natemeyer/frigate:trt-8.5-3b21f86-tensorrt works! I was able to get yolov7x-320, yolov7x-640, yolov7-640, yolov7-320 to work on my 4060ti. The conversion for each went smoothly and no complaints in frigate.

[NateMeyer]

@doctorttt One last note, you should be able to set the USE_FP16 flag to true. I believe that is primarily for older generation cards that don't have FP16 support like Pascal and Maxwell. If you change that and want to regenerate the models, just clear out that model_cache folder.

[doctorttt]

@doctorttt One last note, you should be able to set the USE_FP16 flag to true. I believe that is primarily for older generation cards that don't have FP16 support like Pascal and Maxwell. If you change that and want to regenerate the models, just clear out that model_cache folder.

@NateMeyer Thanks - I flipped USE_FP16 to true and regenerated all yolo7 models - no issue. :)

[trixor]

Can confirm this works on a NVIDIA GeForce RTX 2060. One thing that isn't clear, if I want to provide a custom or fine tuned YOLO model, would I be able to provide it somehow through the external env as PT or ONNX. and get Frigate on startup to do the conversion or does this need to happen manually outside of the container and do we need to store the model and map it like this without an env variable like this:

model:
  path: /config/model_cache/tensorrt/yolov7-tiny-custom-generated.trt

Thanks again for the hard work!

[madsciencetist]

@trixor you'll need to do the conversion yourself and supply the .trt file. However, it would be straightforward to upgrade docker/support/tensorrt_detector/rootfs/etc/s6-overlay/s6-rc.d/trt-model-prepare/run to be able to ingest a user-provided YOLO model.

[trixor]

@madsciencetist thanks! Great, that's how I understood it after reading the code but wanted to verify. I'll see if I can get that to work and submit a PR if able.