This is a Jupyter notebook project that implements Neural Style Transfer using Swift for TensorFlow.
Attribution from left to right:
- King of Hearts / Wikimedia Commons / CC-BY-SA-3.0 [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)]
- A Starry Night by Vincent van Gogh, c. June 1889
- Image generated by employing the Neural Style Transfer algorithm detailed by Gatys et al.
Simply put, Neural Style Transfer [Gatys et al.] is a process by which we take the style of one image, the content of another image and generate a new image that exhibits the same stylistic features as the style image while preserving the high-level structure of the content image.
The neural part, as you might expect, is where the real magic happens. If you've read about or taken a course on Convolutional Neural Networks, you've probably encountered the wonderful paper Visualizing and Understanding Convolutional Networks by Zeiler and Fergus. This paper keenly illustrates the fact that each kernel in a conv net acts as a sort of feature extractor. The features start out primitive and local but as the net goes deeper, the features become more abstract and global. Neural Style Transfer exploits this fact, extracting style information from the activations of various layers. We also extract content information from the higher layers of the network. We use this information to compute what is referred to in the literature as the "Perceptual Loss".
Given an input image, our goal is to minimize Perceptual Loss with respect to that image. In order to compute the Perceptual Loss, we need the following:
- A style image
- A content image
- A styled image. This is the output. We can initialize it a number of ways (which we'll explore below).
- A way of extracting style information from layer activations.
- A pretrained conv net (we'll use VGG-19 [Karen Simonyan, Andrew Zisserman]).
We'll use the Adam optimizer [Diederik P. Kingma, Jimmy Ba] to optimize our "styled" image.