The RedEye project was born from the need to stream video from a potentially moving camera to feed an High Def display for humans and pehraps a low def stream suitable to be consumed by Computer Vision algorithms.
Depending on the specific application, in many cases outputs from the Computer Vision algorithms will provide feedback into the control system of other peripherials like motors, valves and the like.
RedEye software has been designed keeping with a similar pipeline metaphor as the GSTreamer library that our video ingestion and delivery pipelines are built from.
- Capture device: captures images or video from cameras, a filesystem or a URL.
- Injestion Sink: for bringing in image and video of given formats.
- Filtering: converting images between formats, compression, encryption.
- CV Pipeline: accept images into the process pipeline
- Apply pipeline of image processing and CV tasks
- Video Storage: provides local and cloud based storage of long term archives and short term caching.
Note the dependency on Store the storage manager.
Software and Frameworks
OpenCV will provide Computer Vision functionality, GStreamer will provide us with video streaming delivery and ingestion functions. The specific applications will dictate what algorithms are used during the CV processing pipeline.
First we'd like to accept streams directly from the Camera CSI interfaces, pipe them into the Display and CV pipelines.
- File images: png, jpg, ...
- File video: mp4, mov, h264, ...
- HTTP images: save as file images
We are going to need to accept a realtime High Definition video stream that will be displayed for consumption by humans.
A video stream will be piped into Computer Vision Pipeline, which in turn may produce output video and/or images for local and remote storage.
Camera --+ +--> HD Display (annotated) | | File --+-> Incoming --> CV --> Motion --+ | | HTTP --+ +--> Store
Raspberry Pi Camera
NVidia Jetson Nano w/ RPi Camera
NVidia Jetson Nano also with the RPi camera.
USB Video Camera
RedEye also supports Linux USB video cameras, simply due to the fact that USB video support already comes to GSTreamer and OpenCV, hence we will for the most part take advantage of these, and likewise, take advantage of video sources supplied by the host OS and associated drivers, for example the video4linux package. Likewise MacOS USB and builtin cameras are also supported, I do not know if or what additional effort is required for Windows.
Videos can also be sourced from files including: mp4, mov, h264 and other formats. They can also be consumed from a URL for example: an HTTP/HTM5 Video or RTSP realtime stream.
The video from these cameras can be divided into multiple streams with different formats. For example, video from a car camera might be converted into a high res stream and and a low res stream. These two streams might also use different compression and encoding algorithms.
The High Def stream above could be consumed by HD Video Display, while the second low res stream will be consumed by the OpenCV library.
Cloud Storage and Support
One or more of the afore mentioned streams may need to be archived for future replay. In this case the Store module can be used to stash video-clips and images on one of the major Cloud provider storage.
Getting quality high speed video in realtime turned out to be much more challenging and complex than I initially expected. If you are interested in any aspect of real time streaming video, especially as it applies to Computer Vision algorithms, drop me a line, I would love to chat and swap ideas!