Implementation of a neural network that can transfer the style of an arbitrary image to another photo.
Much of the code (e.g the layers) is implemented in my neural network toolbox. The training procedure can be found here. This repository contains only the testing code. To replicate my work, please also clone the experimental branch of my nntoolbox repository.
I train the thing for 850 iterations, using COCO dataset (resize to 256 for each size), and the train_9 subset of the wikiart dataset. For each dataset, I split 80% of them as training data and use the rest for evaluating. I train the network for a total of 850 iterations (1 "epoch"). Some look pretty good:
Other less so:
It seems to work pretty well on resized COCO data (even on untrained/unseen photos), but does not generalize that well too random photos. I suspect the problems lie in resolution discrepancy. Or maybe I just haven't trained for long enough (although in my experiment the quality of the images seem to degrade as I trained more and more).
I decided to download the entire wikiart dataset, and use the same preprocessing for both image in the pair (i.e resize to 512 for the smaller side then random crop a square patch of size 256). I also increased the style weight to 10.0, reduced the learning rate to 1e-4 and used a multiplicative learning rate decay. Here are some results after training for 15598 iterations:
It seems like the stylistic elements are more distinct now. The general stroke style, texture, and color scheme are transferred to the source image, without destroying its content structure.
Code for reproducing my bigger experiment can be found in src/test. Note that this requires the experimental branch of my nn-toolbox repo (which will be released soon on PyPI).
Pretrained model (of bigger experiment) can be downloaded here.
-
Xun Huang, Serge Belongie. Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization: https://arxiv.org/abs/1703.06868
-
Art images are from WikiArt.org. All images are protected by copyright and utilize the images only for the purposes of data mining, which constitutes a form of fair use. Data can be download from: https://www.kaggle.com/c/painter-by-numbers/data
-
COCO dataset: http://cocodataset.org/#download Licensed under a Creative Commons Attribution 4.0 License.
-
All test images are either from the COCO dataset, except for the cat photos, which are from:
https://www.pexels.com/photo/cat-whiskers-kitty-tabby-20787
https://pxhere.com/en/photo/997773
- Other AdaIN implementations that are very helpful for my own attempt:
https://github.com/naoto0804/pytorch-AdaIN
https://github.com/xunhuang1995/AdaIN-style
- The fastai library and lessons have been really useful (I adapted some of the elements of fastai library in my nn-toolbox repository and here).