Guided Image-to-Image Translation with Bi-Directional Feature Transformation

[Project | Paper]

Official Pytorch implementation for Guided Image-to-Image Translation with Bi-Directional Feature Transformation. Please contact Badour AlBahar (badour@vt.edu) if you have any questions.

Prerequisites

This codebase was developed and tested with:

• Python2.7
• Pytorch 0.4.1.post2
• CUDA 8.0

Datasets

• Pose transfer:
  We use DeepFashion dataset. We follow the train/test splits provided by Pose guided person image generation. We provide the data in pickle format here.

• Texture transfer: We use the dataset provided by textureGAN. [shoes dataset, handbags dataset, clothes dataset].

• Depth Upsampling:
  We use the NYU v2 dataset. We provide the mat files here.

Train

1. Pose transfer:

python train.py --dataroot /root/DeepFashion/ --name exp_name --netG bFT_resnet --dataset_mode pose --input_nc 3 --guide_nc 18 --output_nc 3 --lr 0.0002 --niter 100 --niter_decay 0 --batch_size 8 --use_GAN --netD basic --beta1 0.9 --checkpoints_dir ./pose_checkpoints

2. Texture transfer:

python train.py --dataroot /root/training_handbags_pretrain/ --name exp_name --netG bFT_unet --dataset_mode texture --input_nc 1 --guide_nc 4 --output_nc 3 --niter 100 --niter_decay 0 --batch_size 256 --lr 0.0002 --use_GAN --netD basic --n_layers 7 --beta1 .9 --checkpoints_dir ./texture_checkpoints

3. Depth Upsampling:

python train.py --dataroot /root/NYU_RGBD_matfiles/ --name exp_name --netG bFT_resnet --dataset_mode depth --input_nc 1     --guide_nc 3 --output_nc 1 --lr 0.0002 --niter 500 --niter_decay 0 --batch_size 2 --checkpoints_dir ./depth_checkpoints --depthTask_scale [4, 8, or 16]

Test

You can specify which epoch to test by specifying --epoch or use the default which is the latest epoch. Results will be saved in --results_dir.

1. Pose transfer:

python test.py --dataroot /root/DeepFashion/ --name exp_name --netG bFT_resnet --dataset_mode pose --input_nc 3 --guide_nc 18 --output_nc 3 --checkpoints_dir ./pose_checkpoints --task pose --results_dir ./pose_results

2. Texture transfer:

python test.py --dataroot /root/training_handbags_pretrain/ --name exp_name --netG bFT_unet --n_layers 7 --dataset_mode texture --input_nc 1 --guide_nc 4 --output_nc 3 --checkpoints_dir ./texture_checkpoints --task texture --results_dir ./texture_results

3. Depth Upsampling:

python test.py --dataroot /root/NYU_RGBD_matfiles/ --name exp_name --netG bFT_resnet --dataset_mode depth --input_nc 1 --guide_nc 3 --output_nc 1 --checkpoints_dir ./depth_checkpoints --task depth  --depthTask_scale [4, 8, or 16] --results_dir ./depth_results

Pretrained checkpoints

• Download the pretrained checkpoints here.

• Test: For example, to test the depth upsampling task with scale 16:

python test.py --dataroot /root/NYU_RGBD_matfiles/ --name depth_16 --netG bFT_resnet --dataset_mode depth --input_nc 1 --guide_nc 3 --output_nc 1 --checkpoints_dir ./checkpoints/pretrained/ --task depth  --depthTask_scale 16 --results_dir ./depth_results

Evaluate

You can specify which epoch to evaluate by specifying --epoch or use the default which is the latest epoch. Results will be saved in --results_dir.

1. Pose transfer:
Please note that the inception score evaluation requires tensorflow. We evaluate with tensorflow 1.4.0.

python evaluate.py --dataroot /root/DeepFashion/ --name pose --netG bFT_resnet --dataset_mode pose --input_nc 3 --guide_nc 18 --output_nc 3 --checkpoints_dir ./checkpoints/pretrained/ --task pose --results_dir ./pose_results

This will save the results in --results_dir and compute both SSIM and IS metrics.

2. Texture transfer: Please download the pretrained model of textureGAN in ./resources from bags, shoes, and clothes. For example, to test the pretrained texture transfer model for the bags dataset:

python evaluate.py --dataroot /root/training_handbags_pretrain/ --name texture_bags --netG bFT_unet --n_layers 7 --dataset_mode texture --input_nc 1 --guide_nc 4 --output_nc 3 --checkpoints_dir ./checkpoints/pretrained/ --task texture --results_dir ./texture_results

This will save the output of bFT and textureGAN in --results_dir for 10 random input texture patches per test image. The results can then be used to compute FID and LPIPS.

3. Depth Upsampling:

python evaluate.py --dataroot /root/NYU_RGBD_matfiles/ --name depth_16 --netG bFT_resnet --dataset_mode depth --input_nc 1 --guide_nc 3 --output_nc 1 --checkpoints_dir ./checkpoints/pretrained/ --task depth  --depthTask_scale 16 --results_dir ./depth_results

This will save the results in --results_dir and compute their RMSE metric.

Acknowledgments

This code is heavily borrowed from CycleGAN and pix2pix in PyTorch. We thank Shih-Yang Su for the code review.