Enhancement: Real-time Video Stabilization in vidgear

[abhiTronix]

Real-time Video Stabilization in vidgear

Introduction:

Video stabilization refers to a family of methods used to reduce the blurring & distortion associated with the motion of the camera. In other words, it compensates for any angular movement, equivalent to yaw, pitch, roll, and x and y translations of the camera. A related problem common in videos shot from mobile phones. The camera sensors in these phones contain what is known as an electronic rolling shutter. When taking a picture with a rolling shutter camera, the image is not captured instantaneously. Instead, the camera captures the image one row of pixels at a time, with a small delay when going from one row to the next. Consequently, if the camera moves during capture, it will cause image distortions ranging from shear in the case of low-frequency motions (for instance an image captured from a drone) to wobbly distortions in the case of high-frequency perturbations (think of a person walking while recording video). These distortions are especially noticeable in videos where the camera shake is independent across frames. The ability to locate, identify, track and stabilize objects at different poses and backgrounds is important in many real-time video applications. Object detection, tracking, alignment, and stabilization have been a research area of great interest in computer vision and pattern recognition due to the challenging nature of some slightly different objects such as faces, where algorithms should be precise enough to identify, track and focus one individual from the rest.

Real-Time Video Stabilization:

A few months back, while researching on my humanoid, I experienced significant jitteriness at the output due to motion in the cameras/Servos/platform that was causing tracked features to get lost on the way and thus resulting in false-positive movement of humanoid eyes. So, In order to eliminate this problem, I decided to implement a real-time video stabilizer. Therefore I studied & experimented various methods published in various research papers and online resources and finally came to the conclusion that some state-of-the-art video stabilization methods can achieve a quite good visual effect, but they always cost a lot of time. On the other hand, other real-time video stabilization methods cannot generate satisfactory results.

Goal:

Our goal is to implement real-time video stabilization for vidgear which can provide a good balance between stabilization and latency at expense of little to no computational power requirement thereby ideal for the raspberry pi too. Secondly, It must be implemented using OpenCV Computer Vision library for open-source considerations.

Resources:

• Going through various methods published in various research papers and online resources, I think the Simple video stabilization using OpenCV by nghiaho12 works the best on my Raspberry Pi. It is less computationally expensive and there is a C++ implementation available for getting things started with.

TODO

• Implement a Real-Time Video Stabilizer from scratch in python
• Not extra dependency must be used except the existing ones
• Must provide good stabilization and low latency with no extra resources
• Merge stabilizer with VideoGear Class
• Must be compatible with any video stream and able to perform at High FPS.

[abhiTronix]

Purposed Fix

PR #24 introduces Powerful Real-time Video Stabilizer for vidgear from scratch based on findings of by nghiaho12 sample code. This stabilizer can work both standalone and with VidGear Multi-Threaded Classes and thereby is solely integrated into vidgear's VideoGear Class. This feature for any given video stream can be activated with stabilize boolean flag during initialization in VideoGear Class.

Features :

• Real-time stabilization and low latency with no extra resources.
• Works exceptionally well with low frequency jitter.
• Compatible with any vidgear's video streams including live network streams.
• Works both standalone and with VidGear Multi-Threaded Classes.
• Directly integrated into vidgear's VideoGear Class.

[abhiTronix]

Successfully resolved and merged in bdc02a2