t81_558_class_07_3_style_gan.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 7: Generative Adversarial Networks**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 7 Material\n",
    "\n",
    "* Part 7.1: Introduction to GANS for Image and Data Generation [[Video]](https://www.youtube.com/watch?v=0QnCH6tlZgc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_07_1_gan_intro.ipynb)\n",
    "* Part 7.2: Implementing a GAN in Keras [[Video]](https://www.youtube.com/watch?v=T-MCludVNn4&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_07_2_Keras_gan.ipynb)\n",
    "* **Part 7.3: Face Generation with StyleGAN and Python** [[Video]](https://www.youtube.com/watch?v=Wwwyr7cOBlU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_07_3_style_gan.ipynb)\n",
    "* Part 7.4: GANS for Semi-Supervised Learning in Keras [[Video]](https://www.youtube.com/watch?v=ZPewmEu7644&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_07_4_gan_semi_supervised.ipynb)\n",
    "* Part 7.5: An Overview of GAN Research [[Video]](https://www.youtube.com/watch?v=cvCvZKvlvq4&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_07_5_gan_research.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 7.3: Face Generation with StyleGAN and Python\n",
    "\n",
    "GANs have appeared frequently in the media, showcasing their ability to generate extremely photorealistic faces.  One significant step forward for realistic face generation was nVidia StyleGAN, which was introduced in the following paper.\n",
    "\n",
    "* Karras, T., Laine, S., & Aila, T. (2018). [A style-based generator architecture for generative adversarial networks](https://arxiv.org/abs/1812.04948). *arXiv preprint arXiv:1812.04948*.\n",
    "\n",
    "In this part we will make use of StyleGAN.  We will also preload weights that nVidia trained on.  This will allow us to generate high resolution photorealistic looking faces, such as this one.\n",
    "\n",
    "![GAN](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/gan-example246.png \"GAN\")\n",
    "\n",
    "The above image was generated with StyleGAN, using Google CoLab.  Following the instructions in this section, you will be able to create faces like this of your own.  \n",
    "\n",
    "While the above image looks much more realistic than the previous set of images, it is not perfect.  There are usually a number of tell-tail signs that you are looking at a computer generated image.  One of the most obvious is usually the surreal, dream-like backgrounds.  The background does not look obviously fake, at first glance; however, upon closer inspection you usually can't quite discern exactly what a GAN generated background actually is.  Also look at the image character's left eye.  It is slightly unrealistic looking, especially near the eyelashes.\n",
    "\n",
    "Look at the following GAN face.  Can you spot any imperfections?\n",
    "\n",
    "![GAN](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/gan-example221.png \"GAN\")\n",
    "\n",
    "Notice the earrings?  GANs sometimes have problems with symmetry, particularly earrings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keras Sequence vs Functional Model API\n",
    "\n",
    "Most of the neural networks create in this course have made use of the Keras sequence object.  You might have noticed that we briefly made use of another type of neural network object for the ResNet, the Model.  These are the [two major means](https://keras.io/getting-started/functional-api-guide/) of constructing a neural network in Keras:\n",
    "\n",
    "* [Sequential](https://keras.io/getting-started/sequential-model-guide/) - Simplified interface to Keras that supports most models where the flow of information is a simple sequence from input to output. \n",
    "* [Keras Functional API](https://keras.io/getting-started/functional-api-guide/) - More complex interface that allows neural networks to be constructed of reused layers, multiple input layers, and supports building your own recurrent connections.\n",
    "\n",
    "It is important to point out that these are not two specific types of neural network.  Rather, they are two means of constructing neural networks in Keras.  Some types of neural network can be implemented in either, such as dense feedforward neural networks (like we used for the Iris and MPG datasets).  However, other types of neural network, like ResNet and GANs can only be used in the Functional Model API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating High Rez GAN Faces with Google CoLab\n",
    "\n",
    "This notebook demonstrates how to run [NVidia StyleGAN](https://github.com/NVlabs/stylegan) inside of a Google CoLab notebook.  I suggest you use this to generate GAN faces from a pretrained model.  If you try to train your own, you will run into compute limitations of Google CoLab.\n",
    "\n",
    "Make sure to run this code on a GPU instance.  GPU is assumed.\n",
    "\n",
    "First, map your G-Drive, this is where your GANs will be written to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this for Google CoLab\n",
    "from google.colab import drive\n",
    "drive.mount('/content/drive', force_remount=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, clone StyleGAN from GitHub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/NVlabs/stylegan.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify that StyleGAN has been cloned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls /content/stylegan/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add the StyleGAN folder to Python so that you can import it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, \"/content/stylegan\")\n",
    "\n",
    "import dnnlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below is based on code from NVidia. This actually generates your images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.\n",
    "#\n",
    "# This work is licensed under the Creative Commons Attribution-NonCommercial\n",
    "# 4.0 International License. To view a copy of this license, visit\n",
    "# http://creativecommons.org/licenses/by-nc/4.0/ or send a letter to\n",
    "# Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.\n",
    "\n",
    "\"\"\"Minimal script for generating an image using pre-trained StyleGAN generator.\"\"\"\n",
    "\n",
    "import os\n",
    "import pickle\n",
    "import numpy as np\n",
    "import PIL.Image\n",
    "import dnnlib\n",
    "import dnnlib.tflib as tflib\n",
    "import config\n",
    "\n",
    "def main():\n",
    "    # Initialize TensorFlow.\n",
    "    tflib.init_tf()\n",
    "\n",
    "    # Load pre-trained network.\n",
    "    url = 'https://drive.google.com/uc?id=1MEGjdvVpUsu1jB4zrXZN7Y4kBBOzizDQ' # karras2019stylegan-ffhq-1024x1024.pkl\n",
    "    with dnnlib.util.open_url(url, cache_dir=config.cache_dir) as f:\n",
    "        _G, _D, Gs = pickle.load(f)\n",
    "        # _G = Instantaneous snapshot of the generator. Mainly useful for resuming a previous training run.\n",
    "        # _D = Instantaneous snapshot of the discriminator. Mainly useful for resuming a previous training run.\n",
    "        # Gs = Long-term average of the generator. Yields higher-quality results than the instantaneous snapshot.\n",
    "\n",
    "    # Print network details.\n",
    "    Gs.print_layers()\n",
    "\n",
    "    # Pick latent vector.\n",
    "    rnd = np.random.RandomState()\n",
    "    \n",
    "\n",
    "    latents = rnd.randn(1, Gs.input_shape[1])\n",
    "\n",
    "    # Generate image.\n",
    "    fmt = dict(func=tflib.convert_images_to_uint8, nchw_to_nhwc=True)\n",
    "    images = Gs.run(latents, None, truncation_psi=0.7, randomize_noise=True, output_transform=fmt)\n",
    "\n",
    "    # Save image.\n",
    "    os.makedirs(config.result_dir, exist_ok=True)\n",
    "    png_filename = os.path.join(config.result_dir, f'/content/drive/My Drive/images/example1.png')\n",
    "    PIL.Image.fromarray(images[0], 'RGB').save(png_filename)\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}