deep-photo-styletransfer-tf

This is a Windows fork of the Linux based pure Tensorflow implementation of Deep Photo Styletransfer, the torch implementation could be found here. This adds specific instructions for Windows users and header files to allow usage of original Linux specific code.

This implementation support L-BFGS-B (which is what the original authors used) and Adam in case the ScipyOptimizerInterface incompatible when Tensorflow upgrades to higher version.

This implementation may seem to be a little bit simpler thanks to Tensorflow's automatic differentiation

Additionally, there is no dependency on MATLAB thanks to another repository computing Matting Laplacian Sparse Matrix. Below is example of transferring the photo style to another photograph.

Disclaimer

This software is published for academic and non-commercial use only.

Setup

Dependencies

• CUDA 8.0GA2
• cuDNN 6.0 for CUDA 8.0
• Visual Studio 15/15 Community
• Anaconda Python
• Tensorflow
• Numpy
• Pillow
• Scipy
• PyCUDA (used in smooth local affine, tested on CUDA 8.0)

Download the VGG-19 model weights

The VGG-19 model of tensorflow is adopted from VGG Tensorflow with few modifications on the class interface. The VGG-19 model weights is stored as .npy file and could be download here. After downloading, copy the weight file to the ./project/vgg19 directory

Setting up instructions

• Install CUDA 8.0
• Copy folders from cuDNN 6.0 to CUDA 8.0 path
• Launch anaconda prompt
• Install tensorflow. GPU version recommended, follow instructions for install via anaconda.
• Activate the environment - activate env-name
• Install PyCUDA - pip install ./path/to/pycuda.whl
• Install Pillow - conda install pillow
• Install SciPy - conda install scipy
• Install Visual Studio with C++ libraries
• Add Visual Studio to system path
• Add getopt.h and unistd.h to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include
• Try the example below

Usage

Basic Usage

You need to specify the path of content image, style image, content image segmentation, style image segmentation and then run the command

python deep_photostyle.py --content_image_path <path_to_content_image> --style_image_path <path_to_style_image> --content_seg_path <path_to_content_segmentation> --style_seg_path <path_to_style_segmentation> --style_option 2

Example:

python deep_photostyle.py --content_image_path ./examples/input/in11.png --style_image_path ./examples/style/tar11.png --content_seg_path ./examples/segmentation/in11.png --style_seg_path ./examples/segmentation/tar11.png --style_option 2

Other Options

--style_option specifies three different ways of style transferring. --style_option 0 is to generate segmented intermediate result like torch file neuralstyle_seg.lua in torch. --style_option 1 uses this intermediate result to generate final result like torch file deepmatting_seg.lua. --style_option 2 combines these two steps as a one line command to generate the final result directly.

--content_weight specifies the weight of the content loss (default=5), --style_weight specifies the weight of the style loss (default=100), --tv_weight specifies the weight of variational loss (default=1e-3) and --affine_weight specifies the weight of affine loss (default=1e4). You can change the values of these weight and play with them to create different photos.

--serial specifies the folder that you want to store the temporary result out_iter_XXX.png. The default value of it is ./. You can simply mkdir result and set --serial ./result to store them. Again, the temporary results are simply clipping the image into [0, 255] without smoothing. Since for now, the smoothing operations need pycuda and pycuda will have conflict with tensorflow when using single GPU

Run python deep_photostyle.py --help to see a list of all options

Image Segmentation

This repository doesn't offer image segmentation script and simply use the segmentation image from the torch version. The mask colors used are also the same as them. You could specify your own segmentation model and mask color to customize your own style transfer.

Examples

Here are more results from tensorflow algorithm (from left to right are input, style, torch results and tensorflow results)

Acknowledgement

• This Windows modification was done when Andrew Ginns was a masters student at the University of Bristol

• Tensorflow code was done when Yang Liu was a research intern at Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies, under the supervision of Prof. Mingli Song and Yongcheng Jing.

• The tensorflow implementation basically follows the torch code.

• Uses martinbenson's python code to compute Matting Laplacian.

Citation

If you find this code useful for your research, please cite:

@misc{YangPhotoStyle2017,
  author = {Yang Liu},
  title = {deep-photo-style-transfer-tf},
  publisher = {GitHub},
  organization={Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies},
  year = {2017},
  howpublished = {\url{https://github.com/LouieYang/deep-photo-styletransfer-tf}}
}

If you find this modification useful, please cite:

@misc{YangPhotoStyle2017-AndrewWindowsPort,
  author = {Andrew Ginns},
  title = {deep-photo-style-transfer-tf},
  publisher = {GitHub},
  organization={University of Bristol},
  year = {2017},
  howpublished = {\url{https://github.com/andrewginns/deep-photo-styletransfer-tf}}
}

Original Code Contact

Feel free to contact me if there is any question (Yang Liu lyng_95@zju.edu.cn).

Windows modification Contact

Please contact me for issues pertaining to running on Windows (Andrew Ginns ag17634@my.bristol.ac.uk)