Shot Threading and Scene Detection
This is an implementation of some basic video processing tools. Currently, it assumes that shot detection has been performed. It also assumes that you have a suitable means of "reading" through video in Matlab, probably best with a frame burst.
The shot threading is further used for scene detection, as proposed in:
StoryGraphs: Visualizing Character Interactions as a Timeline
Makarand Tapaswi, Martin Bäuml, and Rainer Stiefelhagen
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014
Project page | Paper download | StoryGraphs code
Ubuntu 12.04, Ubuntu 14.04, with Matlab versions: R2013a onwards.
first_init.m script will be called on running
startup.m the first time. This will ask you to download sample shot frames for one video of The Big Bang Theory. The images have been downscaled by 2. Please follow the other instructions.
Initialize the video data.
VideoStruct = BBT(1, 1); initParams;
Compute similarity between every shot and N subsequent shots
ssim = shot_similarity(VideoStruct, params);
Convert similarity to threads (by finding maximal cliques)
[Threads, shot_assigned] = similarity_to_threads(ssim);
Visualize threading using python template engine
visualize_threads_via_htmlrender(VideoStruct, Threads, shot_assigned);
Compute scene change locations
scene_breaks = dp_scenes(VideoStruct, params);
Visualize scene detection using python template engine
visualize_scenes_via_htmlrender(VideoStruct, scene_breaks, params);
Create support for a TV episode by providing the correct interface to VideoStruct (see initializers), and a way to load frames of the video. As an example, the repository includes a few frames of shots of one episode Big Bang Theory.
- VLFeat: Vision Libraries in C++ with a nice interface in Matlab
- MEX/C++ n-d Image Histogram: Thanks to Boris Schauerte, BSD 2-Clause license
- are_images_similar.m Check SIFT-based similarity between 2 images.
- shot_similarity.m Compute shot similarity on the whole video (can use parfor).
- dp_scenes.m Scene detection using dynamic programming.
Python Jinja can be used to automatically generate HTML pages to visualize and/or debug both the shot threading and scene detection results. See this blog post for a sneak peak
- 05 Mar 2015: v1.0: Complete working implementation shot threading + scene detection
- 06 Feb 2015: v0.1: A complete working implementation of shot threading