-
Notifications
You must be signed in to change notification settings - Fork 0
Low latency image processing over TCP IP data stream
| Item | Value |
|---|---|
| Guideline Number | 30 |
| Guideline Responsible (Name, Affiliation) | Alvin Sashala Naik, Thales |
| Guideline Reviewer (Name, Affiliation) | Paul Rodriguez, HIPPEROS |
| Guideline Audience (Category) | System architect |
| Guideline Expertise (Category) | Application developers, System architect |
| Guideline Keywords (Category) | TCP/IP, Gstreamer, OpenCV, FFMPEG |
Use low-latency IP stream for sensor input and image processing rather than too generic libraries.
Integration constraints from Thales Electron Devices (TED) to provide short (<70ms) and guaranteed latency between the image capture and its display.
Recommended implementation method of the guideline along with a solid motivation for the recommendation
In order to transfer a video stream over Ethernet, several protocols are available but some of them are particularly well designed to offer the low-latency required by the Tulipp use cases.
On top of the protocol, the library that processes the images and connect to the communication medium also has an impact on the latency and must be correctly chosen and tuned.
Even though largely used in the image processing domains, OpenCV is not a good fit for embedded targets as its memory footprint is very large and is implemented for general purpose processors and GPUs.
One approach that provides low-latency is running the GStreamer library pipeline. The processing can then be done through OpenVX with zero-copy between the processing functions. The output can then again be done using GStreamer.
More importantly, one can use circular buffers to overcome the drawback of using UDP as a transport protocol for the video. When the application can afford to drop frames, the circular buffers can be flushed whenever needed.
RTSP is a standard used to transport control over video streams on the internet, e.g. it offers a way to start/stop or pause a video stream. This standard comes with the RTP protocol, the standard that transport the video it-self. They have been developed to transport real-time video streams and is widely used for serving videos on Internet, e.g. TV over Internet.
GigE Vision is a protocol developed for industrial applications to transmit high-speed video over Ethernet, relying on UDP.
All these protocols, even if they are meant to be real-time, are relying on the Internet Protocol Stack and the Ethernet drivers.
Today, many companies are using standard NICs to transfer image data from the camera into the host memory or directly to a display controller with low latency and jitter. However, when image processing is added, sharing system resources such as the buses, CPU, and memory, the overall jitter for every process in the system is significantly increased, which has a detrimental effect on maximum guaranteed response time and/or determinism.
Therefore if dedicated hardware that can off-load the IP protocols are available on the system, they must be used. And if not available, then it shall be considered to include them in the final hardware design. A great care must also be taken in the choice of the IP stack which performance must be measured prior to add any other protocol on top of it.
Compression mechanisms like MJPEG, often used to reduce the amount of data to transfer, must be avoided as they introduce an unknown latency. For some applications like the medical ones, it is even forbidden to compress images as it might introduce artifacts or remove details from the original picture.
Implement different pipelines using Gstreamer or any low-latency protocols (e.g. one for the display and another from the sensor). To keep benefit from a low-latency streaming protocol, it will be necessary to perform image processing using zero-copy libraries such as OpenVX.
This guideline is not actually used in the medical use case of the project while it is using the SD-card as the input and the HDMI as the output, but it is used for the final product receiving images from express PCI and sending the processed images over GigE Vision.
none
TULIPP Guideline Wiki