ShowSegmentation

GSoC 2019 project with Red Hen Lab. The goal is to create an algorithm that can automatically find boundaries between TV shows in unannotated recordings and also find the names of the shows identified. The final method uses face recognition to find all the faces present in the video and clusters them to group the faces by persons. We then separate anchors/hosts of shows from those persons that are not anchors/hosts using a few rules. All anchors found using this method are named using our custom classifier built on the MS-Celeb dataset. More information about the project statement can be found here.

Mentors: Anna Bonazzi, Sumit Vohra

Blog detailing the research and working can be found at eonr.github.io

You can skip the first 4 weeks if you only want to read about the final product.

Usage

1. Clone the repo to your machine

git clone https://www.github.com/eonr/ShowSegmentation

2. Install the required python packages using either of these commands

pip install numpy pandas matplotlib opencv-python scikit-learn face_recognition

pip install -r requirements.txt

3. Download the anchors-encodings pickle and place it in this location.

ShowSegmentation/final_usable_code/final_celeb_detection/final_pickles/anchors-with-TV-encodings.pickle

4. Navigate to ShowSegmentation/final_usable_code/

cd ShowSegmentation/final_usable_code

5. segment_video.py takes 3 inputs, the path to the input video, path to the output location and a flag --verbose.

python3 segment_video.py path/to/input/video.mp4 path/to/store/output --verbose

6. Make sure that the input video's name follows RedHenLab's Rosenthal dataset's format. Here's an example

1980-06-03_0000_US_00020088_V0_U2_M9_EG1_DB.mp4

Singularity Usage

Setup

**This is for those using the singularity image (segmentation_production.simg) on the CWRU HPC Cluster.

1. Connect to the CWR VPN.
2. Login to the cluster using your CWR ID and your credentials. Example:

ssh abc12@rider.case.edu

3. Navigate to the project's location on the cluster.

cd /mnt/rds/redhen/gallina/Singularity/Show_Segmentation/final_usable_code

4. Request a computing node using

srun --mem=16gb --pty /bin/bash

5. Load singularity 2.5.1 to your environment using

module load singularity/2.5.1

6. I have made segment_video.py for testing and segment_Rosenthal.py for final production. After setup, read the Testing section or the Production section according to the requirement.

Testing

1. segment_video.py is made to work on a single video file. It takes 3 inputs (in this order)

• path/to/input/video.mp4
• path/to/output/directory (where the output will be stored)
• --verbose (an optional flag which will make the program print progress statements like 'done extracting faces', 'done clustering faces' etc.)

2. The main command is of the form

singularity exec -B /mnt ../segmentation_production.simg python3 segment_video.py {INPUT_VIDEO_PATH} {OUTPUT_PATH} {--verbose}

3. The Rosenthal dataset is present at /mnt/rds/redhen/gallina/Rosenthal, we can take some video file from this as our input.
4. An example command for the file 1998-01/1998-01-01/1998-01-01_0000_US_00019495_V3_VHS50_MB20_H17_WR.mp4 is

singularity exec -B /mnt ../segmentation_production.simg python3 segment_video.py /mnt/rds/redhen/gallina/Rosenthal/1998/1998-01/1998-01-01/1998-01-01_0000_US_00019495_V3_VHS50_MB20_H17_WR.mp4 mnt/path/to/output/directory --verbose

Production

1. segment_Rosenthal.py is made to work recursively on all the video files present in /mnt/rds/redhen/gallina/Rosenthal/ and store the outputs in /mnt/rds/redhen/gallina/RosenthalSplit/
2. --verbose flag mentioned earlier is set to False by default for production.
3. Run the script using

singularity exec -B /mnt ../segmentation_production.simg python3 segment_Rosenthal.py

Please raise an issue if you run into any errors.

Future work

• Explore subtitle analysis to identify anchor/show names.
• Using IMDb dataset to identify show names from anchors’ names (identified using MSCeleb).
• If possible, replace the current celeb detection method with Azure’s Computer Vision service.
• Currently the most time consuming process in the program is that of going frame by frame and extracting faces. This can be sped up using multi-threading or any other means possible.