Generating-Faces-with-a-GAN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generating Faces With A Deep Convolutional Generative Adversarial Network\n",
    "\n",
    "Recently I found a great repo posted by Erik Lindernoren [https://github.com/eriklindernoren/Keras-GAN/blob/master/dcgan/dcgan.py] providing wire frame code for a whole bunch of different GANs. I've wanted to give GANs a try for a while so, I grabbed the DCGAN code from this repo, a celebrity faces data set posted by Mirantha Jayathilaka at [https://www.floydhub.com/mirantha/datasets/celeba]. Following methods and using code from the DCGAN and gan_rgb folders in Erik Lindernoren's repo, I set up the the DCGAN code to train on RGB images from the Celeba data set. \n",
    "\n",
    "On a hunch, I wanted to modify the generator network to accept random noise in the same size and shape as the image it is generating, rather than a random vector of length 100. My thought was, why 100? I couldn't not think of any theoretical reason to keep the length of the random vector to 100, or any guidence on the size. Without reseraching, I decided to make the random input the same size and dimentions as the output that is to be produced. Following Erik's code, I used his crop and resize pre-processing to generate and discriminate with 28x28 pixel images cropped close to the faces. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports \n",
    "# Code from [https://github.com/eriklindernoren/Keras-GAN/blob/master/dcgan/dcgan.py]\n",
    "\n",
    "from __future__ import print_function, division\n",
    "\n",
    "from keras.datasets import mnist\n",
    "from keras.datasets import cifar10\n",
    "from keras.layers import Input, Dense, Reshape, Flatten, Dropout\n",
    "from keras.layers import BatchNormalization, Activation, ZeroPadding2D\n",
    "from keras.layers.advanced_activations import LeakyReLU\n",
    "from keras.layers.convolutional import UpSampling2D, Conv2D\n",
    "from keras.models import Sequential, Model\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "import tensorflow as tf\n",
    "from scipy.misc import imread, imsave\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import sys\n",
    "import os\n",
    "from PIL import Image\n",
    "from glob import glob\n",
    "\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New Noise object\n",
    "Here, I was double checking the new noise object formation. I wanted to use a noise object the same size and dimentions as the prefered output, rather than the random 1 d vector with 100 numbers. It was just the first thing I thought of to try different. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Testing the new noise object for generator input\n",
    "img_rows = 28 \n",
    "img_cols = 28\n",
    "channels = 3\n",
    "img_shape = (img_rows, img_cols, channels)\n",
    "\n",
    "z = Input(shape = (img_rows, img_cols, channels))\n",
    "noise = np.random.normal(0, 1, (16, img_rows, img_cols, channels))\n",
    "np.shape(noise)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New Models for Generator and Discriminator\n",
    "Additionally, I tried new models for the generator and discriminator. For the generator, I experimented with a non-fully convolutional layer, but a model that looks somewhat simmilar to an image classifier, but ends with a reshpaing of a single vector into a 28x28 RGB image matrix. It contained about 26 million parameters. For the discriminator, I tried something similar, building a model with early convolutional layers, and latter fully connected layers. The discriminator model had about 56 million parameters.\n",
    "\n",
    "The idea here was to make a smart and stable enough generator to compete with a stable discriminator that has the potential to be more intelligent. I thought that this would create a difficult set up for the generator, but not so difficutl that there would not be enough error signal from the discriminator to train on. Essentially, training a forger really hard to beat a very smart discriminator. \n",
    "\n",
    "This may seem a bit weird, but I was working with the assumption that really small generator loss is not nessesarity the goal, but getting good looking pictures fast is. The values of the generator loss will not consistantly relate to a particular image qualtiy, as the meaning of the generator loss changes as the dicriminator learns. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Code from [https://github.com/eriklindernoren/Keras-GAN/blob/master/dcgan/dcgan.py]\n",
    "# define the GAN class which contains methods for building, training, and saving outputs and training information \n",
    "#   for both the generator and discriminator\n",
    "class GAN():\n",
    "    def __init__(self):\n",
    "        self.img_rows = 28 \n",
    "        self.img_cols = 28\n",
    "        self.channels = 3\n",
    "        self.img_shape = (self.img_rows, self.img_cols, self.channels)\n",
    "\n",
    "        optimizer = Adam(0.0002, 0.5)\n",
    "\n",
    "        # Build and compile the discriminator\n",
    "        self.discriminator = self.build_discriminator()\n",
    "        self.discriminator.compile(loss='binary_crossentropy', \n",
    "            optimizer=optimizer,\n",
    "            metrics=['accuracy'])\n",
    "\n",
    "        # Build and compile the generator\n",
    "        self.generator = self.build_generator()\n",
    "        self.generator.compile(loss='binary_crossentropy', optimizer=optimizer)\n",
    "\n",
    "        # The generator takes noise as input and generated imgs\n",
    "        z = Input(shape = (self.img_rows, self.img_cols, self.channels,))\n",
    "        img = self.generator(z)\n",
    "\n",
    "        # For the combined model we will only train the generator\n",
    "        self.discriminator.trainable = False\n",
    "\n",
    "        # The valid takes generated images as input and determines validity\n",
    "        valid = self.discriminator(img)\n",
    "\n",
    "        # The combined model  (stacked generator and discriminator) takes\n",
    "        # noise as input => generates images => determines validity \n",
    "        self.combined = Model(z, valid)\n",
    "        self.comself.discriminator.trainablebined.compile(loss='binary_crossentropy', optimizer=optimizer)\n",
    "\n",
    "    def build_generator(self):\n",
    "\n",
    "        noise_shape = (self.img_rows, self.img_cols, self.channels,)\n",
    "        \n",
    "        model = Sequential()\n",
    "        model.add(Conv2D(128, kernel_size=3, input_shape = noise_shape, padding=\"same\"))\n",
    "        model.add(Activation(\"relu\"))\n",
    "        model.add(BatchNormalization(momentum=0.8))\n",
    "        model.add(UpSampling2D())\n",
    "        model.add(Conv2D(64, kernel_size=3, padding=\"same\"))\n",
    "        model.add(Activation(\"relu\"))\n",
    "        model.add(BatchNormalization(momentum=0.8))\n",
    "        model.add(Flatten())\n",
    "        model.add(Dense(256))\n",
    "        model.add(LeakyReLU(alpha=0.2))\n",
    "        model.add(BatchNormalization(momentum=0.8))\n",
    "        model.add(Dense(512))\n",
    "        model.add(LeakyReLU(alpha=0.2))\n",
    "        model.add(BatchNormalization(momentum=0.8))\n",
    "        model.add(Dense(1024))\n",
    "        model.add(LeakyReLU(alpha=0.2))\n",
    "        model.add(BatchNormalization(momentum=0.8))\n",
    "        model.add(Dense(np.prod(self.img_shape), activation='tanh'))\n",
    "        model.add(Reshape(self.img_shape))\n",
    "\n",
    "        model.summary()\n",
    "\n",
    "        noise = Input(shape=noise_shape)\n",
    "        img = model(noise)\n",
    "\n",
    "        return Model(noise, img)\n",
    "\n",
    "    def build_discriminator(self):\n",
    "\n",
    "        img_shape = (self.img_rows, self.img_cols, self.channels)\n",
    "        \n",
    "        model = Sequential()\n",
    "        model.add(Conv2D(128, kernel_size=3, input_shape = img_shape, padding=\"same\"))\n",
    "        model.add(Activation(\"relu\"))\n",
    "        model.add(Conv2D(64, kernel_size=3, padding=\"same\"))\n",
    "        model.add(Activation(\"relu\"))\n",
    "        model.add(Flatten(input_shape=img_shape))\n",
    "        model.add(Dense(512))\n",
    "        model.add(LeakyReLU(alpha=0.2))\n",
    "        model.add(Dense(256))\n",
    "        model.add(LeakyReLU(alpha=0.2))\n",
    "        model.add(Dense(1, activation='sigmoid'))\n",
    "        model.summary()\n",
    "\n",
    "        img = Input(shape=img_shape)\n",
    "        validity = model(img)\n",
    "\n",
    "        return Model(img, validity)\n",
    "\n",
    "    def get_image(self, image_path, width, height, mode):\n",
    "    \n",
    "        image = Image.open(image_path)\n",
    "        # image = image.resize([width, height], Image.BILINEAR)\n",
    "        if image.size != (width, height):  \n",
    "        # Remove most pixels that aren't part of a face\n",
    "            face_width = face_height = 108\n",
    "            j = (image.size[0] - face_width) // 2\n",
    "            i = (image.size[1] - face_height) // 2\n",
    "            image = image.crop([j, i, j + face_width, i + face_height])\n",
    "            image = image.resize([width, height])\n",
    "    \n",
    "        return np.array(image.convert(mode))\n",
    "\n",
    "    def get_batch(self, image_files, width, height, mode):\n",
    "        data_batch = np.array(\n",
    "            [self.get_image(sample_file, width, height, mode) for sample_file in image_files])\n",
    "\n",
    "        return data_batch    \n",
    "\n",
    "    def train(self, epochs, batch_size=128, save_interval=50):\n",
    "        \n",
    "        data_dir = '../data'\n",
    "        X_train = self.get_batch(glob(os.path.join(data_dir, '*.jpg'))[:5000], 28, 28, 'RGB')\n",
    "\n",
    "\n",
    "        #Rescale -1 to 1 !!!!! so should my random generations\n",
    "        X_train = (X_train.astype(np.float32) - 127.5) / 127.5\n",
    "\n",
    "\n",
    "        half_batch = int(batch_size / 2)\n",
    "\n",
    "        #Create lists for logging the losses\n",
    "        d_loss_logs_r = []\n",
    "        d_loss_logs_f = []\n",
    "        g_loss_logs = []\n",
    "\n",
    "        for epoch in range(epochs):\n",
    "\n",
    "            # ---------------------\n",
    "            #  Train Discriminator\n",
    "            # ---------------------\n",
    "\n",
    "            # Select a random half batch of images\n",
    "            idx = np.random.randint(0, X_train.shape[0], half_batch)\n",
    "            imgs = X_train[idx]\n",
    "\n",
    "            noise = np.random.normal(0, 1, (half_batch, self.img_rows, self.img_cols, self.channels))\n",
    "\n",
    "            # Generate a half batch of new images\n",
    "            gen_imgs = self.generator.predict(noise)\n",
    "\n",
    "            # Train the discriminator\n",
    "            d_loss_real = self.discriminator.train_on_batch(imgs, np.ones((half_batch, 1)))\n",
    "            d_loss_fake = self.discriminator.train_on_batch(gen_imgs, np.zeros((half_batch, 1)))\n",
    "            d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\n",
    "\n",
    "\n",
    "            # ---------------------\n",
    "            #  Train Generator\n",
    "            # ---------------------\n",
    "\n",
    "            noise = np.random.normal(0, 1, (batch_size, self.img_rows, self.img_cols, self.channels))\n",
    "\n",
    "            # The generator wants the discriminator to label the generated samples\n",
    "            # as valid (ones)\n",
    "            valid_y = np.array([1] * batch_size)\n",
    "\n",
    "            # Train the generator\n",
    "            g_loss = self.combined.train_on_batch(noise, valid_y)\n",
    "\n",
    "            # Plot the progress\n",
    "            print (\"%d [D loss: %f, acc.: %.2f%%] [G loss: %f]\" % (epoch, d_loss[0], 100*d_loss[1], g_loss))\n",
    "\n",
    "            #Append the logs with the loss values in each training step\n",
    "            d_loss_logs_r.append([epoch, d_loss[0]])\n",
    "            d_loss_logs_f.append([epoch, d_loss[1]])\n",
    "            g_loss_logs.append([epoch, g_loss])\n",
    "\n",
    "            # If at save interval => save generated image samples\n",
    "            if epoch % save_interval == 0:\n",
    "                self.save_imgs(epoch)\n",
    "\n",
    "\n",
    "            #Convert the log lists to numpy arrays\n",
    "            d_loss_logs_r_a = np.array(d_loss_logs_r)\n",
    "            d_loss_logs_f_a = np.array(d_loss_logs_f)\n",
    "            g_loss_logs_a = np.array(g_loss_logs)\n",
    "\n",
    "            #Generate the plot at the end of training\n",
    "            plt.plot(d_loss_logs_r_a[:,0], d_loss_logs_r_a[:,1], label=\"Discriminator Loss - Real\")\n",
    "            plt.plot(d_loss_logs_f_a[:,0], d_loss_logs_f_a[:,1], label=\"Discriminator Loss - Fake\")\n",
    "            plt.plot(g_loss_logs_a[:,0], g_loss_logs_a[:,1], label=\"Generator Loss\")\n",
    "            plt.xlabel('Epochs')\n",
    "            plt.ylabel('Loss')\n",
    "            plt.legend()\n",
    "            plt.title('Variation of losses over epochs')\n",
    "            plt.grid(True)\n",
    "            plt.show()     \n",
    "\n",
    "    def save_imgs(self, epoch):\n",
    "        r, c = 3, 3\n",
    "        noise = np.random.normal(0, 1, (r * c, self.img_rows, self.img_cols, self.channels))\n",
    "        gen_imgs = self.generator.predict(noise)\n",
    "\n",
    "        # Rescale images 0 - 1\n",
    "        gen_imgs = (1/2.5) * gen_imgs + 0.5\n",
    "\n",
    "        fig, axs = plt.subplots(r, c)\n",
    "        cnt = 0\n",
    "        for i in range(r):\n",
    "            for j in range(c):\n",
    "                axs[i,j].imshow(gen_imgs[cnt, :,:,:])\n",
    "                axs[i,j].axis('off')\n",
    "                cnt += 1\n",
    "        fig.savefig(\"output/%d.png\" % epoch)\n",
    "        plt.close()\n",
    "\n",
    "\n",
    "#if __name__ == '__main__':\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run it\n",
    "Here I used code from Erik's repo, I just pulled it out of .py file context, and put it here to run in the notebook\n",
    "\n",
    "I ran for a total of 2,900 times on a laptop with 16 GB ram and a quad-core i5. This is a pretty humble effort, and although the resulting images weren't great, I was surprised with the resutls given the hardware I used and the small amount of time. You can definately learn alot about gans without logging into AWS and spending some cash on a big cluster, but for great high-resolution results, I think something like that might be nesseary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out was removed, it contains may picutres and made the notebook too large\n",
    "gan = GAN()\n",
    "gan.train(epochs=30000, batch_size=32, save_interval=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resulting Images\n",
    "The images weren't too bad, but still lacked some realism and consistancy by the time I stopped trianing. The convolutional / fully connected generator produced a particular style of image, kind of cubist. This is likely from the size of the filters and the fact that the image is only 28x28, so the general idea shows some promise for scaling up to higher res. Also, there wasn't much variation in the produced images. Each training iteration, they where different, but within a single batch there was a lot of similarity. \n",
    "\n",
    "The training process was hard to get a hold of, as generator loss was pretty variable and not very simple to relate to image qualtiy. In general, the generator loss increased after an initial decrease, but ocilated up and down as well. One pattern I noticed was ocilations in the generator loss, and the better images seemed to be coming out somewhere on the \"down swing\" of the generator loss, but not always on the minimum. Based on my experiece training this model, I am pretty sure results could be drastically different with changes to model training and balance. I think there is a lot to gained in how good and how fast the model will learn to generate images with better training procudures.\n",
    "\n",
    "\n",
    "**Training Results**\n",
    "<img src=\"last_fit_out.png\" width=\"500\" />\n",
    "\n",
    "\n",
    "**Generator (26 MM parameters) and Discriminator (56 MM parametes) with multiple convolutional layers at 2900 Epochs. Generating from 28x28x3 niose array, same size and shape as output**\n",
    "<img src=\"output/2900.png\" width=\"500\" />\n",
    "\n",
    "**Generator (26 MM parameters) and Discriminator (56 MM parametes) with multiple convolutional layers at 2750 Epochs. Generating from 28x28x3 niose array, same size and shape as output**\n",
    "<img src=\"output/2750.png\" width=\"500\" />\n",
    "\n",
    "For comparison, Erik's repo contained a GAN desiged to produce these 28x28 celeb images from a non-convoutional network. That model starts with a vector of 100 random number and uses a generator model with about 1 million trainable prameters, and a discriminator with about 3 million parameters. Although the model is much simpiler, it still captures a lot of the face structure. However, there is clear noise still present in the images, which was long gone from the convolutional network outputs by this epoch. Here are the non-convoutional results from epoch 3000.\n",
    "\n",
    "**Generator (1 MM parameters) and Discriminator (3 MM parameters) with without convolutional layers at 2900 Epochs. Generating from 1D noise vector length 100**\n",
    "<img src=\"output/non_conv.png\" width=\"500\" />\n",
    "\n",
    "The convolutional model made better looking pictures at 3000 epochs, but I suspect it is overfitted and is getting close to spitting back just copies of images. It still has a long way to go to realistic and needs more consistant, and more diverse results. Better training, model balance, and more computing power could lead to a method capable of generating more realistic and higher reslotuion images. Learning more about the input noise, training procudures, and balancing the generator and discrimiator model could greatily improve the quality, consistancy, and shorten the training time for GANs. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}