unconference-2019-pix2pix

ITP 2019 Unconference Pix2Pix Workshop

Pix2Pix

Data Preparation

Training Pix2Pix

Use the Model

Pix2Pix

What is it?

Pix2Pix, or Image-to-Image Translation, can learn how to convert images of one type into another type of new image. Pix2Pix uses a kind of generative algorithm called Conditional Adversarial Network(cGAN) where the generation process is “conditioned” on the input image. It learns a mapping from an input image to an output image.

Pros

• no assumptions
• learns the objective during training
• requires small number of samples

Cons

• overfitting to the training samples; repetitive || patchy

What it does?

Interactive Demo - Christopher Hesse
ML4A - Gene Kogan

What we will build?

Create own dataset

It should consist of:

• 512x256 size (pairs of 256x256 images)
• hundreds of images (in same format)
• identical direction (A to B || B to A)

Tools

instagram-scraper
body-pix / person segmentation

Training

Prerequisite

downgrade python3

https://chriscummins.cc/2018/brew-python-downgrade/

Install Python3

# check python3 version
python3 --version

# install xcode
xcode-select --install

# install homebrew
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

# check homebrew
brew doctor

# install python3 with homebrew
brew install python3

# check python3 version
python3 --version

Install pip

python --version
sudo easy_install pip

Install Tensorflow

pip install tensorflow

# make sure tensorflow installed
python -c "import tensorflow; print(tensorflow.__version__)"

Install Spell CLI

pip install spell

spell login
spell whoami

Clone the repo

git clone https://github.com/affinelayer/pix2pix-tensorflow.git
cd pix2pix-tensorflow
open .

Download dataset

download(24MB, 277 images)

# move downloaded folder to pix2pix-tensorflow folder

# Let spell know the newly downloaded dataset
git add .
git commit -m "added input image data"

Train the model

# train the model

# If you add `--ngf 32 --ndf 32` when training the model, the output model will be smaller, and it will take less time to train.
# The command reduces the number of filters on generator and discriminator from 64(default) to 32.

spell run --machine-type V100 \
          --framework tensorflow \
          "python pix2pix.py \
          --mode train \
          --input_dir input \
          --output_dir output \
          --max_epochs 50 \
          --which_direction BtoA \
          --ngf 32 --ndf 32"

# After the run is finished (~5min)
# check out the result 
spell ls runs/YOUR_RUN_NUMBER
spell ls runs/YOUR_RUN_NUMBER/output

Test the model

download test_images, locate into pix2pix-tensorflow root folder
download test images

# in pix2pix-tensorflow root folder
mkdir output
cd output

# Copy the result folder, takes ~5min
spell cp runs/YOUR_RUN_NUMBER/output

# test the model
cd ..
python pix2pix.py \
     --mode test \
     --output_dir model_test \
     --input_dir test_images \
     --checkpoint output

# After testing, you should be able to see output images in the model_test folder by opening the `index.html` file in the browser

Wanna fairly decent result? 🕒💸 is the key.

Export the model

# export the model
python pix2pix.py --mode export --output_dir export/ --checkpoint output/ --which_direction BtoA

# It will create a new export folder

# Port the model to tensorflow.js (python 2 doesn’t have tempfile, so use python3 instead)
cd server
cd static
mkdir model
cd ..
python3 tools/export-checkpoint.py --checkpoint ../export --output_file static/model/YOUR_MODEL_NAME_BtoA.pict

# You should be able to get a file named YOUR_MODEL_NAME_BtoA.pict in /server/static/model/ folder.

Use the model with ml5

Download ported model (2000epochs)

model download

Clone repo & run the app

git clone https://github.com/dongphilyoo/unconference-2019-pix2pix.git
cd pix2pix-ml5-demo
mkdir model
cd model

# copy the model file you got to model folder
cp /PATH/TO/SOURCE/FILE .

# open app.js in editor, put the model file path into the line:
# pix2pix = ml5.pix2pix('model/model_BtoA_2000.pict', modelLoaded);

# run server
python -m SimpleHTTPServer

# go to http://localhost:8000

Acknowledgement

Yining Shi's training process tutorial
https://github.com/yining1023/pix2pix_spell

Resources

Spell Documentation
pix2pix original paper
NVIDIA pix2pixHD (Pytorch)
NVIDIA vid2vid (Pytorch)
pix2pix tensorflow implementation
ml5.js pix2pix
Medium Post
humansofnowhere