Skip to content
This repository has been archived by the owner on Jul 10, 2023. It is now read-only.

Pipeline failure in some multi-GPU systems #98

Closed
whbruce opened this issue Dec 7, 2021 · 1 comment
Closed

Pipeline failure in some multi-GPU systems #98

whbruce opened this issue Dec 7, 2021 · 1 comment
Labels
bug Something isn't working

Comments

@whbruce
Copy link

whbruce commented Dec 7, 2021

Pipelines can sometimes fail in multi GPU systems due to video decode incompatibility. Pipeline will return an error when starting and service log will contain an entry like this

{"levelname": "INFO", "asctime": "2021-11-30 22:48:40,620", "message": "Creating Instance of Pipeline object_detection/person_vehicle_bike", "module": "pipeline_manager"}
{"levelname": "ERROR", "asctime": "2021-11-30 22:48:47,369", "message": "Error on Pipeline 1: gst-stream-error-quark: No valid frames decoded before end of stream (7): ../gst-libs/gst/video/gstvideodecoder.c(1246): gst_video_decoder_sink_event_default (): /GstPipeline:pipeline11/GstURIDecodeBin:source/GstDecodeBin:decodebin2/GstVaapiDecodeBin:vaapidecodebin2/GstVaapiDecode:vaapidecode2:\nno valid frames found", "module": "gstreamer_pipeline"}

Workaround is to disable GPU decode by clearing environment variable LIBVA_DRIVER_NAME
e.g. if using run script to start service:

./run.sh -v /tmp:/tmp -e LIBVA_DRIVER_NAME=
@whbruce whbruce added the bug Something isn't working label Dec 7, 2021
@whbruce whbruce changed the title Pipeline failure in multi-GPU systems Pipeline failure in some multi-GPU systems Dec 7, 2021
@whbruce
Copy link
Author

whbruce commented Oct 24, 2022

@whbruce whbruce closed this as completed Oct 24, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant