This repository provides a solution for automatically detecting and extracting panning shots in video files. Panning shots are a common technique in animation where the camera or background moves horizontally, creating an illusion of motion and depth. This technique not only adds visual interest but also serves as a cost-saving measure by minimizing the need to animate individual elements separately.
However a lot of high quality shots are left out for use in datasets as a result of this animation technique. This repos seeks to recover those shots by analyzing a video identifying the parts of the video that are likely panning shots and stitching together in an attempt to recover the original full picutre used for it.
This is achiveced through a combination of Machine learning/computer vision techniques. The approach is to use a ML optical flow model (Raft) to identify the likely parts of the video that
are likely to be a panninshot. Then double check it by taking the crop of the intersection between the two frames and checking if its the same picture using an
image hashing algorithm. the rationale is that if those two frames are part of a panning shot than the intersectio that optical flow model says they both should share should be the
same picture. This is done for the frames of the video and the frames that are likely panning shots are put into groups and a stitch is attempted using opencvs stitching class.
A visual representation of the concept is shown in the animation below
To use this script you first need to download the optical flow models from PINTO model zoo and place it in the models folder. The script supports running onnx files or tensortRT engines if you convert those files.
Explantion of script parameters
--videodir The filepath of the video file to analyze
--outdir The directory to save the image stitches and log files
--starting_timestamp The time to start analysing the video
--ending_timestamp The time to stop analysing the video
--ratio The ratio of the output .5 resizes the image to half their dimensions before stitching for example
--min_range the minumum range of detected optical movement in order to attempt stitching
--use_trt Flag to use tensorRT
--frame_skip how many frames to skip before running optical flow inference
--model_path the path of the optical flow model
a gui was also added for ease of use
git clone https://github.com/bdiaz29/Moving-Shot-Extraction-
pip install -r requirements.txt
The Stitching process isn't always perfect and so you should always do a manual inspection of the stiches before including them in a dataset. I'm still trying to figure out the best approach to dealing with frames that are panning where there is character movement.