Super Resolution Video - Video Upscale - 2x SRGAN

Summary: Slides

Upscale video from low resolution to high resolution to reduce the effect of low bandwidth internet on video chat (meeting) or to reduce streaming video size without losing video quality.

Problem to Solve:

Eliminate pixelated videos in streaming, video call/chat/conference due to poor Internet.

Solution:

Requirements:

Speed: Real time video rendering. Target: 0.04 second/frame ~ 25 FPS (frame per second) video.

Image Quality: Acceptable.

Result:

Image Quality:

Comparison is between:

• Low resolution input.
• Sony Vegas Pro 17.0 (a software for video editing, similar to Adobe A.I.).
• OpenCV (a python library for processing images).
• My model.
• The real high resolution image.

Image quality is better than Sony Vegas Pro and OpenCV, but still far away from real high resolution.

Image quality of my model is acceptable.

Speed:

My model rendering time has not yet reached real time rendering speed.

Project Details:

Datasets:

1. REDS_VTSR:

REalistic and Dynamic Scenes dataset for Video Temporal Super-Resolution (frame interpolation) (REDS_VTSR) dataset includes ~70 GB data:

• 13,500 frames.
• Shaking camera movements.
• Outside environment and acivities.

-> Suitable for live streaming videos.

REDS_VTSR sample video

2. Pexels:

• 5,332 frames
• Steady camera movements.
• In-door environment and acivities.
• Capture a lot of human faces.

-> Suitable for video call.

Pexels sample video

Training:

GAN model:

Model reference: Fast-SRGAN

Input: 160x90p videos.

Output: 320x180p videos.

2x upscale.

Production - Demo:

Pipeline:

1. Use OpenCV to turn input video frames into images.
2. Input those images to Generator Model to generate high resolution images.
3. Then, use OpenCV to turn high resolution images to a video for display.

Flask app:

• Upscale a whole video via upload.
• Upscale real time video recording from webcam.