README.adoc

Animation Stylization Collection

Table of Contents

• Environment
• Algorithm
  • AnimeGAN: A Novel Lightweight GAN for Photo Animation
  • Learning to Cartoonize Using White-box Cartoon Representations
  • Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation(minivision)
• Repository structure
• Participation and Contribution

Environment

conda create -n torch python=3.8
conda activate torch
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1 -c pytorch
pip install pytorch-lightning==1.0.2 opencv-python matplotlib joblib scikit-image torchsummary webdataset albumentations more_itertools

Algorithm

AnimeGAN: A Novel Lightweight GAN for Photo Animation

Setup	If you just want to use it, just skip the following step download dataset from here and unzip download pretrain VGG19 from here and unzip, then put it to models/vgg19.npy
Train	change configs/animegan_pretrain.yaml dataset › root to your path change configs/animeganv2.yaml dataset › root to your path pre-training generator make train CODE=scripts/animegan_pretrain.py CFG=configs/animegan_pretrain.yaml training generator (use Ctrl+c can stop) make train CODE=scripts/animeganv2.py CFG=configs/animeganv2.yaml check progress make tensorboard LOGDIR=logs/animeganv2/
test	you can download my pretrained model from here. run test command make infer CODE=scripts/animeganv2.py \ CKPT=logs/animeganv2/version_0/checkpoints/epoch=17.ckpt \ EXTRA=image_path:asset/animegan_test2.jpg # (1) make infer CODE=scripts/animeganv2.py \ CKPT=logs/animeganv2/version_0/checkpoints/epoch=17.ckpt \ EXTRA=help # (2) The EXTRA paramter can pass image path or image directory path Read help.
Result

📝	If you want to change style, you have run scripts/animegan_datamean.py get dataset mean difference, then change configs/animeganv2.yaml dataset › data_mean.

Learning to Cartoonize Using White-box Cartoon Representations

Setup	If you just want to use it, just skip the following step download dataset from here and unzip download pretrain VGG19 from here and unzip, then put it to models/vgg19.npy
Train	change configs/whitebox_pretrain.yaml dataset › root to your path change configs/whitebox.yaml dataset › root to your path pre-training generator make train CODE=scripts/whiteboxgan_pretrain.py CFG=configs/whitebox_pretrain.yaml training generator (use Ctrl+c can stop) make train CODE=scripts/whiteboxgan.py CFG=configs/whitebox.yaml check progress make tensorboard LOGDIR=logs/whitebox
test	you can download my pretrained model from here. you can use choice whitebox-v2.zip or whitebox.zip run test command make infer CODE=scripts/whiteboxgan.py \ CKPT=logs/whitebox/version_0/checkpoints/epoch=4.ckpt \ EXTRA=image_path:asset/whitebox_test.jpg # (1) make infer CODE=scripts/whiteboxgan.py \ CKPT=logs/whitebox/version_0/checkpoints/epoch=4.ckpt \ EXTRA=image_path:tests/test.flv,device:cuda,batch_size:4 # (2) # ffmpeg -i xx.mp4 -vcodec libx265 -crf 28 xxx.mp4 make infer CODE=scripts/whiteboxgan.py \ CKPT=logs/whitebox/version_0/checkpoints/epoch=4.ckpt \ EXTRA=help # (3) The EXTRA paramter can pass image path or image directory path Using GPU convert video, if GPU ran out of memory, pleas use cpu Read help.
Result

📝	The model › superpixel_fn has a great influence on the style. The slic,sscolor refer from offical code. defualt use sscolor. Pytorch version and official version compare results in here

Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation(minivision)

Setup	If you just want to use it, just skip the following step download dataset from here and unzip download model_mobilefacenet.pth from here and put into ./models
Train	change configs/uagtit.yaml dataset › root to your path training generator (use Ctrl+c can stop) make train CODE=scripts/uagtit.py CFG=configs/uagtit.yaml check progress make tensorboard LOGDIR=logs/uagtit
test	you can download my pretrained model from here. (I have not enough GPU and time, so this model effect not so good) this model requires an input image that only contains the human head. Since I don’t have time to migrate the previous tools to PyTorch, need to rely on the previous library to process images.(you need clone this repo) python tools/face_crop_and_mask.py \ --data_path test/model_image \ --save_path test/model_image_faces \ --use_face_crop True \ --use_face_algin False \ --face_crop_ratio 1.3 run test command make infer CODE=scripts/uagtit.py \ CKPT=logs/uagtit/version_13/checkpoints/epoch=15.ckpt \ EXTRA=image_path:asset/uagtit_test.png
Result

📝	The model › light control the model version, the light version requires GPU memory > 8G,non-light version requires GPU memory > 22G. Maybe you need training more epoch to get better results.

Repository structure

Path	Description
AnimeStylized	Repository root folder
├  asset	Folder containing readme image assets
├  configs	Folder containing configs defining model/data paramters, training hyperparamters.
├  datamodules	Folder with various dataset objects and transfroms.
├  losses	Folder containing various loss functions for training, Only very general used loss functions are added here.
├ models	Folder containing all the models and training objects
├  optimizers	Folder with common used optimizers
├  scripts	Folder with running scripts for training and inference
├  utils	Folder with various utility functions

📝

configs Each algorithm has a corresponding config file. Config file uses the YAML format datamodules The dataset.py,dsfunction.py,dstransform.py contains common data module object’s basic component Basically, each algorithm has a corresponding xxxds.py models Basically, each algorithm has a corresponding xxxnet.py Now, only have the gan architecture model, in future maybe add more. scripts Each algorithm has a corresponding xxx.py, implement the main training step and inference here Each algorithm must add run_common(xxxModule, xxxDataModule) in main function, then you can use general trainer to training or testing

Participation and Contribution

1. Add custom LightningDataModule object as xxxds.py in datamodules dir.

2. Add custom Module object model architecture as xxxnet.py in networks dir.

3. Add custom LightningDataModule training script as xxx.py in scripts dir

4. Add config file in configs dir, the paramters follow your custom LightningModule and LightningDataModule

5. trianing your algorithm