Skip to content

Latest commit

 

History

History
182 lines (150 loc) · 8.33 KB

README.md

File metadata and controls

182 lines (150 loc) · 8.33 KB

erAIser - Remove an object in video using AI

file video

첫슬라이드

Contents

  1. erAIser
  2. Example
  3. Demo screenshot
  4. Usage
    • Environment setup
    • Run demo
  5. References
  6. Contributors

erAIser


‘erAIser’ is a service that provides a specific object erased video by using video object segmentation and video inpainting methods.

Most of video inpainting model need segmentation mask of objects. But it is hard to get in normal way. For your convenience, we used a deep learning model that allows users to easily obtain segmentation masks. We combined this video object segmentation model with the video inpainting model to increase usability. Additionally, we use AANet to change object to other object in object erased video.

Our team consists of nine members of ‘Tobigs’ who are interested in computer vision task.

All you have to do is draw the object bounding box that you want to erase in the first frame. Then 'erAIser' will make a video of the object being erased. Let’s make your own video of a specific object being erased with ‘erAIser’!

Example



Demo screenshot


front


You can look around our web at the following link. For your information, the model is currently not working on the web due to the GPU environment.

web link.

Usage

Environment Setup

This code was tested in the following environments

  • Ubuntu 18.04.5
  • Python 3.7
  • Pytorch 1.8.1
  • CUDA 9.0
  • GCC 5.5 (gnu c++14 compiler)

If you don't use gnu c++14 compiler, then you will encounter CUDA build error

  1. Clone the repository & Setup
git clone https://github.com/shkim960520/tobigs-image-conference.git
cd tobigs-image-conference
conda create -n erAIser python=3.7 -y
conda activate erAIser
conda install cudatoolkit=9.0 -c pytorch -y
pip install -r requirements.txt
bash install.sh
  1. Setup python path
export PYTHONPATH=$PWD:$PYTHONPATH
cd vos/
export PYTHONPATH=$PWD:$PYTHONPATH
cd ../vi/
export PYTHONPATH=$PWD:$PYTHONPATH
cd ../web/
export PYTHONPATH=$PWD:$PYTHONPATH
cd ../AANet/
export PYTHONPATH=$PWD:$PYTHONPATH
cd ../

Demo

  1. Setup your environment
  2. Download the Deep Video Inpainting model
cd vi/results/vinet_agg_rec

file_id="1_v0MBUjWFLD28oMfsG6YwG_UFKmgZ8_7"
file_name="save_agg_rec_512.pth"
curl -sc /tmp/cookie "https://drive.google.com/uc?export=download&id=${file_id}" > /dev/null
code="$(awk '/_warning_/ {print $NF}' /tmp/cookie)"
curl -Lb /tmp/cookie "https://drive.google.com/uc?export=download&confirm=${code}&id=${file_id}" -o ${file_name}

file_id="12TRCSwixAo9AqyZ0zXufNKhxuHJWG-RV"
file_name="save_agg_rec.pth"
curl -sc /tmp/cookie "https://drive.google.com/uc?export=download&id=${file_id}" > /dev/null
code="$(awk '/_warning_/ {print $NF}' /tmp/cookie)"
curl -Lb /tmp/cookie "https://drive.google.com/uc?export=download&confirm=${code}&id=${file_id}" -o ${file_name}

cd ../../../
  1. Download the Siammask model
wget http://www.robots.ox.ac.uk/~qwang/SiamMask_DAVIS.pth

file_id="1IKZWpMeWXq-9osBqG7e7bTABumIZ32gB"
file_name="checkpoint_e19.pth"
curl -sc /tmp/cookie "https://drive.google.com/uc?export=download&id=${file_id}" > /dev/null
code="$(awk '/_warning_/ {print $NF}' /tmp/cookie)"
curl -Lb /tmp/cookie "https://drive.google.com/uc?export=download&confirm=${code}&id=${file_id}" -o ${file_name}
  1. Download the AANet model
cd AANet/
mkdir checkpoints
cd checkpoints/

file_id="1DT6_SZHTkmuEWvCfs07F2mGLSkgxYplo"
file_name="4dataset384.pth"
curl -sc /tmp/cookie "https://drive.google.com/uc?export=download&id=${file_id}" > /dev/null
code="$(awk '/_warning_/ {print $NF}' /tmp/cookie)"
curl -Lb /tmp/cookie "https://drive.google.com/uc?export=download&confirm=${code}&id=${file_id}" -o ${file_name}

cd ../../
  1. Make results directory for saving result video
mkdir results

results is defualt setting. You can change this.

6-1. Run inference.py for erasing

python3 inference.py --resume checkpoint_e19.pth --config config_inference.json

6-2. Run inference.py for change object to other (ex. person, animation character)

python3 inference.py --resume SiamMask_DAVIS.pth --config config_inference.json --using_aanet True

The result video will be saved in results.

References

  • Wang, Qiang, et al. "Fast online object tracking and segmentation: A unifying approach." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
  • Wang, Tianyu, et al. "Instance shadow detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
  • Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Deep video inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5792–5801, 2019.
  • Siarohin, Aliaksandr and Woodford, et al. "Motion Representations for Articulated Animation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

Contributors


김태한
Inpainting

이유진
Inpainting, Web

김상현
Video Object Segmentation, Web

김민경
Video Object Segmentation

서아라
Inpainting

장혜림
Video Object Segmentation

권오현
Video Object Segmentation, Web

박진수
Inpainting, AAnet

오주영
Inpainting, Web