Object-Based-Mask-Generation

Object based mask generator is used to generate mask from a video from dataset of Office activity recognition. Mainly the task of this is to differentiate the video from irrelevant background object so that network can concentrate into important objects to classify better. It was based on Inception Resnet V2 feature extractor which performs best among all of the existed benchmarked model on COCO dataset.

This was developed for an alternative stream of optical flow. Our proposed network "SegCodeNet" is published here. A preprint for this is in the link.

Pipeline of the work is

This is implemented on Python 3 and TensorFlow. Here 13 objects from COCO dataset has been included for segmentation as these objects are important in office activity dataset. It's based on Instance segmentation and then assign specific color to same type of objects which is almost similar for detecting same type of class.

Some example of our work

Coloring scheme for segmentation masks obtained from different class-relevant objects are :

Relevant Objects	Color	Color Code (R,G,B)
Person	White	(255,255,255)
TV, laptop, monitor	Red	(255,0,0)
Bottle, cup, wine glass	Blue	(0,0,255)
Cell phone	Cyan	(0,255,255)
Microwave, oven	Yellow	(255,255,0)
Sink & Light	Blue	(100,150,200)
Paper, book	Magenta	(255,0,255)
Keyboard	Green	(0,255,0)
Background/irrelevant objects	Black	(0,0,0)

The repository includes:

• Source code of Mask R-CNN built on Inception Resnet V2 backbone to generate masked frame/video.

• Source code of Privacy Protection. Main goal of this task to provide privacy to privacy sensitive objects in a video. Some example of privacy protection is given in this repo.

Getting Started

1. Clone this repository

2. Install dependencies

   conda install -c anaconda tensorflow-gpu
   conda install -c anaconda pillow
   conda install -c anaconda opencv
   conda install -c anaconda matplotlib

3. Download pre-trained weights,classes names and related file from the Tensorflow model zoo. Download 'mask_rcnn_inception_resnet_v2_atrous_coco' from 'COCO-trained models' table. Put the .rar file in 'object_detection' folder. You can also download other models which gives mask as output. We select inception_resnet_V2 as it gives better result though computationally expensive.

4. Create two folders in 'object_detection' folder 'test_directory_folder', 'protected_directory_folder' respectively for test videos and protected videos.

5. Run the 'masking.py' in command window from 'object_detection' folder to generate masked video/frame.

  python masking.py

6. (Extra) Run the 'protected.py' in command window from 'object_detection' folder generate privacy protected video/frame.

  python protected.py

You can follow Mask_RCNN_Dependencies Installation to install every dependency required for this project.

Requirements

Python 3.4, TensorFlow 1.3,anaconda,opencv and other common packages. (If having trouble with TF 2.0 then you can install older version by: pip install tensorflow-gpu==1.14)

Dataset

• This was FPV-O office dataset given in an open competition name Video and Image Processing Cup 2019
• Details about dataset can be found from this paper

Related Works

• You can see related works and projects on which we worked from VIP Cup 2019 where we placed 2nd Runners Up. Our team 'BUET Synapticans', we are undergrad students of Bangladesh University of Engineering and University.

• Privacy protection using different framework.Privacy_Protection_MASKRCNN_Resnet101, Privacy_Protection_MaskRCNN_InceptionV2

Acknowledgement

Tons of code was downloaded from theirs repo
https://github.com/tensorflow/models