2024.0.2
Intel® Deep Learning Streamer Pipeline Framework Release 2024.0.2
Intel® Deep Learning Streamer (Intel® DL Streamer) Pipeline Framework is a streaming media analytics framework, based on GStreamer* multimedia framework, for creating complex media analytics pipelines. It ensures pipeline interoperability and provides optimized media, and inference operations using Intel® Distribution of OpenVINO™ Toolkit Inference Engine backend, across Intel® architecture, CPU, discreate GPU, integrated GPU and NPU.
This release includes Intel® DL Streamer Pipeline Framework elements to enable video and audio analytics capabilities, (e.g., object detection, classification, audio event detection), and other elements to build end-to-end optimized pipeline in GStreamer* framework.
The complete solution leverages:
- Open source GStreamer* framework for pipeline management
- GStreamer* plugins for input and output such as media files and real-time streaming from camera or network
- Video decode and encode plugins, either CPU optimized plugins or GPU-accelerated plugins based on VAAPI
- Deep Learning models converted from training frameworks TensorFlow*, Caffe* etc. from Open Model Zoo (OMZ)
- The following elements in the Pipeline Framework repository:
| Element | Description |
|---|---|
| gvadetect | Performs object detection on a full-frame or region of interest (ROI) using object detection models such as YOLOv4, MobileNet SSD, Faster-RCNN etc. Outputs the ROI for detected objects. |
| gvaclassify | Performs object classification. Accepts the ROI as an input and outputs classification results with the ROI metadata. |
| gvainference | Runs deep learning inference on a full-frame or ROI using any model with an RGB or BGR input. |
| gvaaudiodetect | Performs audio event detection using AclNet model. |
| gvatrack | Performs object tracking using zero-term, or imageless tracking algorithms. Assigns unique object IDs to the tracked objects. |
| gvametaaggregate | Aggregates inference results from multiple pipeline branches |
| gvametaconvert | Converts the metadata structure to the JSON format. |
| gvametapublish | Publishes the JSON metadata to MQTT or Kafka message brokers or files. |
| gvapython | Provides a callback to execute user-defined Python functions on every frame. Can be used for metadata conversion, inference post-processing, and other tasks. |
| gvawatermark | Overlays the metadata on the video frame to visualize the inference results. |
| gvafpscounter | Measures frames per second across multiple streams in a single process |
For the details of supported platforms, please refer to System Requirements section.
For installing Pipeline Framework with the prebuilt binaries or Docker* or to build the binaries from the open source, please refer to Intel® DL Streamer Pipeline Framework installation guide
New in this Release
| Title | High-level description |
|---|---|
| Support for ‘gst-va’ in addition to ‘gst-vaapi’ | Support for ‘gst-va’ in addition to ‘gst-vaapi’ |
| Add support for EfficentNetv2 (classification), MaskRCNN (instance segmentation) and Yolo8-OBB (oriented bounding box) | New classification model supported EfficentNetv2, new instance segmentation model supported MaskRCNN and oriented bounding box model as well added Yolo8-OBB |
| Support additional GETI models: segmentation, obb | GETI public models support added |
| Generalized method to deploy new models without need for model-proc file | Support model information embedded into AI model descriptors according to OpenVINO Model API |
| Release docker images on DockerHUB: runtime | Added docker images on DockerHUB: runtime |
| Added support for OpenVINO 2024.1.0 | Added support for OpenVINO 2024.1.0 |
Acknowledgements
Thanks for contributions from the DL Streamer developer community:
@aminatef
@russkel
Fixed issues
| Issue # | Issue Description | Fix | Affected platforms |
|---|---|---|---|
| 407 | EfficientNet-B1 support | We do not plan to support older DL Streamer releases with API1.0, I highly recommend to switch to newer version compatible with latest OpenVINO | All |
| 410 | cant run againts my camera feed | Config error, user opened a new issue for tracking the yolo issue and was able to see cameras now | All |
| 412 | with Docker cmd, cant create and dowload models inside docker | Config error. Without $ sign, when assigning a value (using $ to retrieve the value of a variable, e.g. to print the value): `$ export MODELS_PATH=/home/dlstreamer/temp/models1 | All |
| 413 | ffmpeg_openvino build failed, LibAV | Possible config error, missing libraries but no feedback given for 2 weeks so the issue was closed | All |
| 415 | cant run against Efficientnet-b0 due to model exceeds allowable size of 10MB | Resolved, user was able to get it running with last suggestion to use implemented RealSense specific gstreamer plugins, like •https://github.com/WKDSMRT/realsense-gstreamer => realsensesrc •https://gitlab.com/aivero/legacy/public/gstreamer/gst-realsense => realsensesrc A couple years old... |
All |
| 416 | detection with yolo not available on latest | Continue with "merged" command-line, using videobox and or videomixer (or many different other ways from the internet). You might need to start again... and checking the setup on your HOST. I'm using Ubuntu 22.04LTS. Created a non-root-user. Adding the user to video and render groups. Installed docker and configured to use Docker as non-root (without using "sudo" when using "docker run"). Before starting the container, I just call xhost +. Passing the render-group-id to "docker run" (in my case --group-add=110) docker run -it --net=host --device=/dev/dri --device=/dev/video0 --device=/dev/video1 --group-add=110 -v ~/.Xauthority:/home/dlstreamer/.Xauthority -v /tmp/.X11-unix -e DISPLAY=$DISPLAY -v /dev/bus/usb dlstreamer /bin/bash (not using -u 0 --privileged) |
All |
Known Issues
| Issue | Issue Description |
|---|---|
VAAPI memory with decodebin |
If you are using decodebin in conjunction with vaapi-surface-sharing preprocessing backend you should set caps filter using "video/x-raw(memory:VASurface)" after decodebin to avoid issues with pipeline initialization |
Artifacts on sycl_meta_overlay |
Running inference results visualization on GPU via sycl_meta_overlay may produce some partially drawn bounding boxes and labels |
| Preview Architecture 2.0 Samples | Preview Arch 2.0 samples have known issues with inference results |
Memory grow with meta_overlay |
Some combinations of meta_overlay and encoders can lead to memory grow |
System Requirements
Please refer to Intel® DL Streamer documentation.
Installation Notes
There are several installation options for Pipeline Framework:
- Install Pipeline Framework from pre-built Debian packages
- Build Docker image from docker file and run Docker image
- Build Pipeline Framework from source code
For more detailed instructions please refer to Intel® DL Streamer Pipeline Framework installation guide.
Samples
The samples folder in Intel® DL Streamer Pipeline Framework repository contains command line, C++ and Python examples.
Legal Information
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software, or service activation. Learn more at intel.com, or from the OEM or retailer.
No computer system can be absolutely secure. Intel does not assume any liability for lost or stolen data or systems or any damages resulting from such losses.
You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.
No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.
This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.
The products and services described may contain defects or errors which may cause deviations from published specifications. Current characterized errata are available on request.
Intel, the Intel logo, and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries.
FFmpeg is an open source project licensed under LGPL and GPL. See https://www.ffmpeg.org/legal.html. You are solely responsible for determining if your use of FFmpeg requires any additional licenses. Intel is not responsible for obtaining any such licenses, nor liable for any licensing fees due, in connection with your use of FFmpeg.
GStreamer is an open source framework licensed under LGPL. See https://gstreamer.freedesktop.org/documentation/frequently-asked-questions/licensing.html. You are solely responsible for determining if your use of GStreamer requires any additional licenses. Intel is not responsible for obtaining any such licenses, nor liable for any licensing fees due, in connection with your use of GStreamer.
*Other names and brands may be claimed as the property of others.
© 2024 Intel Corporation.
