1. Load and Visualize Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Facial Keypoint Detection\n",
    "  \n",
    "This project will be all about defining and training a convolutional neural network to perform facial keypoint detection, and using computer vision techniques to transform images of faces.  The first step in any challenge like this will be to load and visualize the data you'll be working with. \n",
    "\n",
    "Let's take a look at some examples of images and corresponding facial keypoints.\n",
    "\n",
    "<img src='images/key_pts_example.png' width=50% height=50%/>\n",
    "\n",
    "Facial keypoints (also called facial landmarks) are the small magenta dots shown on each of the faces in the image above. In each training and test image, there is a single face and **68 keypoints, with coordinates (x, y), for that face**.  These keypoints mark important areas of the face: the eyes, corners of the mouth, the nose, etc. These keypoints are relevant for a variety of tasks, such as face filters, emotion recognition, pose recognition, and so on. Here they are, numbered, and you can see that specific ranges of points match different portions of the face.\n",
    "\n",
    "<img src='images/landmarks_numbered.jpg' width=30% height=30%/>\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and Visualize Data\n",
    "\n",
    "The first step in working with any dataset is to become familiar with your data; you'll need to load in the images of faces and their keypoints and visualize them! This set of image data has been extracted from the [YouTube Faces Dataset](https://www.cs.tau.ac.il/~wolf/ytfaces/), which includes videos of people in YouTube videos. These videos have been fed through some processing steps and turned into sets of image frames containing one face and the associated keypoints.\n",
    "\n",
    "#### Training and Testing Data\n",
    "\n",
    "This facial keypoints dataset consists of 5770 color images. All of these images are separated into either a training or a test set of data.\n",
    "\n",
    "* 3462 of these images are training images, for you to use as you create a model to predict keypoints.\n",
    "* 2308 are test images, which will be used to test the accuracy of your model.\n",
    "\n",
    "The information about the images and keypoints in this dataset are summarized in CSV files, which we can read in using `pandas`. Let's read the training CSV and get the annotations in an (N, 2) array where N is the number of keypoints and 2 is the dimension of the keypoint coordinates (x, y).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import the required libraries\n",
    "import glob\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "\n",
    "import cv2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "key_pts_frame = pd.read_csv('data/training_frames_keypoints.csv')\n",
    "\n",
    "n = 0\n",
    "image_name = key_pts_frame.iloc[n, 0]\n",
    "key_pts = key_pts_frame.iloc[n, 1:].as_matrix()\n",
    "key_pts = key_pts.astype('float').reshape(-1, 2)\n",
    "\n",
    "print('Image name: ', image_name)\n",
    "print('Landmarks shape: ', key_pts.shape)\n",
    "print('First 4 key pts: {}'.format(key_pts[:4]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# print out some stats about the data\n",
    "print('Number of images: ', key_pts_frame.shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Look at some images\n",
    "\n",
    "Below, is a function `show_keypoints` that takes in an image and keypoints and displays them.  As you look at this data, **note that these images are not all of the same size**, and neither are the faces! To eventually train a neural network on these images, we'll need to standardize their shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def show_keypoints(image, key_pts):\n",
    "    \"\"\"Show image with keypoints\"\"\"\n",
    "    plt.imshow(image)\n",
    "    plt.scatter(key_pts[:, 0], key_pts[:, 1], s=20, marker='.', c='m')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Display a few different types of images by changing the index n\n",
    "\n",
    "# select an image by index in our data frame\n",
    "n = 0\n",
    "image_name = key_pts_frame.iloc[n, 0]\n",
    "key_pts = key_pts_frame.iloc[n, 1:].as_matrix()\n",
    "key_pts = key_pts.astype('float').reshape(-1, 2)\n",
    "\n",
    "plt.figure(figsize=(5, 5))\n",
    "show_keypoints(mpimg.imread(os.path.join('data/training/', image_name)), key_pts)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset class and Transformations\n",
    "\n",
    "To prepare our data for training, we'll be using PyTorch's Dataset class. Much of this this code is a modified version of what can be found in the [PyTorch data loading tutorial](http://pytorch.org/tutorials/beginner/data_loading_tutorial.html).\n",
    "\n",
    "#### Dataset class\n",
    "\n",
    "``torch.utils.data.Dataset`` is an abstract class representing a\n",
    "dataset. This class will allow us to load batches of image/keypoint data, and uniformly apply transformations to our data, such as rescaling and normalizing images for training a neural network.\n",
    "\n",
    "\n",
    "Your custom dataset should inherit ``Dataset`` and override the following\n",
    "methods:\n",
    "\n",
    "-  ``__len__`` so that ``len(dataset)`` returns the size of the dataset.\n",
    "-  ``__getitem__`` to support the indexing such that ``dataset[i]`` can\n",
    "   be used to get the i-th sample of image/keypoint data.\n",
    "\n",
    "Let's create a dataset class for our face keypoints dataset. We will\n",
    "read the CSV file in ``__init__`` but leave the reading of images to\n",
    "``__getitem__``. This is memory efficient because all the images are not\n",
    "stored in the memory at once but read as required.\n",
    "\n",
    "A sample of our dataset will be a dictionary\n",
    "``{'image': image, 'keypoints': key_pts}``. Our dataset will take an\n",
    "optional argument ``transform`` so that any required processing can be\n",
    "applied on the sample. We will see the usefulness of ``transform`` in the\n",
    "next section.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset, DataLoader\n",
    "\n",
    "class FacialKeypointsDataset(Dataset):\n",
    "    \"\"\"Face Landmarks dataset.\"\"\"\n",
    "\n",
    "    def __init__(self, csv_file, root_dir, transform=None):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "            csv_file (string): Path to the csv file with annotations.\n",
    "            root_dir (string): Directory with all the images.\n",
    "            transform (callable, optional): Optional transform to be applied\n",
    "                on a sample.\n",
    "        \"\"\"\n",
    "        self.key_pts_frame = pd.read_csv(csv_file)\n",
    "        self.root_dir = root_dir\n",
    "        self.transform = transform\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.key_pts_frame)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        image_name = os.path.join(self.root_dir,\n",
    "                                self.key_pts_frame.iloc[idx, 0])\n",
    "        \n",
    "        image = mpimg.imread(image_name)\n",
    "        \n",
    "        # if image has an alpha color channel, get rid of it\n",
    "        if(image.shape[2] == 4):\n",
    "            image = image[:,:,0:3]\n",
    "        \n",
    "        key_pts = self.key_pts_frame.iloc[idx, 1:].as_matrix()\n",
    "        key_pts = key_pts.astype('float').reshape(-1, 2)\n",
    "        sample = {'image': image, 'keypoints': key_pts}\n",
    "\n",
    "        if self.transform:\n",
    "            sample = self.transform(sample)\n",
    "\n",
    "        return sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've defined this class, let's instantiate the dataset and display some images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Construct the dataset\n",
    "face_dataset = FacialKeypointsDataset(csv_file='data/training_frames_keypoints.csv',\n",
    "                                      root_dir='data/training/')\n",
    "\n",
    "# print some stats about the dataset\n",
    "print('Length of dataset: ', len(face_dataset))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Display a few of the images from the dataset\n",
    "num_to_display = 3\n",
    "\n",
    "for i in range(num_to_display):\n",
    "    \n",
    "    # define the size of images\n",
    "    fig = plt.figure(figsize=(20,10))\n",
    "    \n",
    "    # randomly select a sample\n",
    "    rand_i = np.random.randint(0, len(face_dataset))\n",
    "    sample = face_dataset[rand_i]\n",
    "\n",
    "    # print the shape of the image and keypoints\n",
    "    print(i, sample['image'].shape, sample['keypoints'].shape)\n",
    "\n",
    "    ax = plt.subplot(1, num_to_display, i + 1)\n",
    "    ax.set_title('Sample #{}'.format(i))\n",
    "    \n",
    "    # Using the same display function, defined earlier\n",
    "    show_keypoints(sample['image'], sample['keypoints'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transforms\n",
    "\n",
    "Now, the images above are not of the same size, and neural networks often expect images that are standardized; a fixed size, with a normalized range for color ranges and coordinates, and (for PyTorch) converted from numpy lists and arrays to Tensors.\n",
    "\n",
    "Therefore, we will need to write some pre-processing code.\n",
    "Let's create four transforms:\n",
    "\n",
    "-  ``Normalize``: to convert a color image to grayscale values with a range of [0,1] and normalize the keypoints to be in a range of about [-1, 1]\n",
    "-  ``Rescale``: to rescale an image to a desired size.\n",
    "-  ``RandomCrop``: to crop an image randomly.\n",
    "-  ``ToTensor``: to convert numpy images to torch images.\n",
    "\n",
    "\n",
    "We will write them as callable classes instead of simple functions so\n",
    "that parameters of the transform need not be passed everytime it's\n",
    "called. For this, we just need to implement ``__call__`` method and \n",
    "(if we require parameters to be passed in), the ``__init__`` method. \n",
    "We can then use a transform like this:\n",
    "\n",
    "    tx = Transform(params)\n",
    "    transformed_sample = tx(sample)\n",
    "\n",
    "Observe below how these transforms are generally applied to both the image and its keypoints.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torchvision import transforms, utils\n",
    "# tranforms\n",
    "\n",
    "class Normalize(object):\n",
    "    \"\"\"Convert a color image to grayscale and normalize the color range to [0,1].\"\"\"        \n",
    "\n",
    "    def __call__(self, sample):\n",
    "        image, key_pts = sample['image'], sample['keypoints']\n",
    "        \n",
    "        image_copy = np.copy(image)\n",
    "        key_pts_copy = np.copy(key_pts)\n",
    "\n",
    "        # convert image to grayscale\n",
    "        image_copy = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)\n",
    "        \n",
    "        # scale color range from [0, 255] to [0, 1]\n",
    "        image_copy=  image_copy/255.0\n",
    "        \n",
    "        # scale keypoints to be centered around 0 with a range of [-1, 1]\n",
    "        # mean = 100, sqrt = 50, so, pts should be (pts - 100)/50\n",
    "        key_pts_copy = (key_pts_copy - 100)/50.0\n",
    "\n",
    "\n",
    "        return {'image': image_copy, 'keypoints': key_pts_copy}\n",
    "\n",
    "\n",
    "class Rescale(object):\n",
    "    \"\"\"Rescale the image in a sample to a given size.\n",
    "\n",
    "    Args:\n",
    "        output_size (tuple or int): Desired output size. If tuple, output is\n",
    "            matched to output_size. If int, smaller of image edges is matched\n",
    "            to output_size keeping aspect ratio the same.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, output_size):\n",
    "        assert isinstance(output_size, (int, tuple))\n",
    "        self.output_size = output_size\n",
    "\n",
    "    def __call__(self, sample):\n",
    "        image, key_pts = sample['image'], sample['keypoints']\n",
    "\n",
    "        h, w = image.shape[:2]\n",
    "        if isinstance(self.output_size, int):\n",
    "            if h > w:\n",
    "                new_h, new_w = self.output_size * h / w, self.output_size\n",
    "            else:\n",
    "                new_h, new_w = self.output_size, self.output_size * w / h\n",
    "        else:\n",
    "            new_h, new_w = self.output_size\n",
    "\n",
    "        new_h, new_w = int(new_h), int(new_w)\n",
    "\n",
    "        img = cv2.resize(image, (new_w, new_h))\n",
    "        \n",
    "        # scale the pts, too\n",
    "        key_pts = key_pts * [new_w / w, new_h / h]\n",
    "\n",
    "        return {'image': img, 'keypoints': key_pts}\n",
    "\n",
    "\n",
    "class RandomCrop(object):\n",
    "    \"\"\"Crop randomly the image in a sample.\n",
    "\n",
    "    Args:\n",
    "        output_size (tuple or int): Desired output size. If int, square crop\n",
    "            is made.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, output_size):\n",
    "        assert isinstance(output_size, (int, tuple))\n",
    "        if isinstance(output_size, int):\n",
    "            self.output_size = (output_size, output_size)\n",
    "        else:\n",
    "            assert len(output_size) == 2\n",
    "            self.output_size = output_size\n",
    "\n",
    "    def __call__(self, sample):\n",
    "        image, key_pts = sample['image'], sample['keypoints']\n",
    "\n",
    "        h, w = image.shape[:2]\n",
    "        new_h, new_w = self.output_size\n",
    "\n",
    "        top = np.random.randint(0, h - new_h)\n",
    "        left = np.random.randint(0, w - new_w)\n",
    "\n",
    "        image = image[top: top + new_h,\n",
    "                      left: left + new_w]\n",
    "\n",
    "        key_pts = key_pts - [left, top]\n",
    "\n",
    "        return {'image': image, 'keypoints': key_pts}\n",
    "\n",
    "\n",
    "class ToTensor(object):\n",
    "    \"\"\"Convert ndarrays in sample to Tensors.\"\"\"\n",
    "\n",
    "    def __call__(self, sample):\n",
    "        image, key_pts = sample['image'], sample['keypoints']\n",
    "         \n",
    "        # if image has no grayscale color channel, add one\n",
    "        if(len(image.shape) == 2):\n",
    "            # add that third color dim\n",
    "            image = image.reshape(image.shape[0], image.shape[1], 1)\n",
    "            \n",
    "        # swap color axis because\n",
    "        # numpy image: H x W x C\n",
    "        # torch image: C X H X W\n",
    "        image = image.transpose((2, 0, 1))\n",
    "        \n",
    "        return {'image': torch.from_numpy(image),\n",
    "                'keypoints': torch.from_numpy(key_pts)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test out the transforms\n",
    "\n",
    "Let's test these transforms out to make sure they behave as expected. As you look at each transform, note that, in this case, **order does matter**. For example, you cannot crop a image using a value smaller than the original image (and the orginal images vary in size!), but, if you first rescale the original image, you can then crop it to any size smaller than the rescaled size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# test out some of these transforms\n",
    "rescale = Rescale(100)\n",
    "crop = RandomCrop(50)\n",
    "composed = transforms.Compose([Rescale(250),\n",
    "                               RandomCrop(224)])\n",
    "\n",
    "# apply the transforms to a sample image\n",
    "test_num = 500\n",
    "sample = face_dataset[test_num]\n",
    "\n",
    "fig = plt.figure()\n",
    "for i, tx in enumerate([rescale, crop, composed]):\n",
    "    transformed_sample = tx(sample)\n",
    "\n",
    "    ax = plt.subplot(1, 3, i + 1)\n",
    "    plt.tight_layout()\n",
    "    ax.set_title(type(tx).__name__)\n",
    "    show_keypoints(transformed_sample['image'], transformed_sample['keypoints'])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the transformed dataset\n",
    "\n",
    "Apply the transforms in order to get grayscale images of the same shape. Verify that your transform works by printing out the shape of the resulting data (printing out a few examples should show you a consistent tensor size)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# define the data tranform\n",
    "# order matters! i.e. rescaling should come before a smaller crop\n",
    "data_transform = transforms.Compose([Rescale(250),\n",
    "                                     RandomCrop(224),\n",
    "                                     Normalize(),\n",
    "                                     ToTensor()])\n",
    "\n",
    "# create the transformed dataset\n",
    "transformed_dataset = FacialKeypointsDataset(csv_file='data/training_frames_keypoints.csv',\n",
    "                                             root_dir='data/training/',\n",
    "                                             transform=data_transform)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# print some stats about the transformed data\n",
    "print('Number of images: ', len(transformed_dataset))\n",
    "\n",
    "# make sure the sample tensors are the expected size\n",
    "for i in range(5):\n",
    "    sample = transformed_dataset[i]\n",
    "    print(i, sample['image'].size(), sample['keypoints'].size())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Iteration and Batching\n",
    "\n",
    "Right now, we are iterating over this data using a ``for`` loop, but we are missing out on a lot of PyTorch's dataset capabilities, specifically the abilities to:\n",
    "\n",
    "-  Batch the data\n",
    "-  Shuffle the data\n",
    "-  Load the data in parallel using ``multiprocessing`` workers.\n",
    "\n",
    "``torch.utils.data.DataLoader`` is an iterator which provides all these\n",
    "features, and we'll see this in use in the *next* notebook, Notebook 2, when we load data in batches to train a neural network!\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ready to Train!\n",
    "\n",
    "Now that you've seen how to load and transform our data, you're ready to build a neural network to train on this data.\n",
    "\n",
    "In the next notebook, you'll be tasked with creating a CNN for facial keypoint detection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:ai]",
   "language": "python",
   "name": "conda-env-ai-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}