2 - Neural Networks Hello World.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hello World in Deep Learning\n",
    "\n",
    "### Deep learning is simply an artificial neural network with multiple hidden layers. It is the hidden layers that make them deep. A general neural network may only have a few hidden layers (1 or 2). Deep neural networks also differs from general neural networks in the types of layers that are used as we will see.\n",
    "\n",
    "<BR>\n",
    "<BR>\n",
    "<img src=\"./images/nn.png\" width=\"600px\"> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The MNIST Dataset\n",
    "\n",
    "### MNIST is a dataset of digital images representing handwritten digits. It also includes labels for the images. For example, the labels for the images below would be 5, 0, 4, and 1.\n",
    "<BR>\n",
    "<img src=\"./images/MNIST.png\" width=\"300px\"> \n",
    "<BR>\n",
    "    \n",
    "### The MNIST data is split into three parts: \n",
    "\n",
    "* 55,000 data points of training data (mnist.train)\n",
    "* 10,000 points of test data (mnist.test)\n",
    "* 5,000 points of validation data (mnist.validation). \n",
    "\n",
    "### This split is very important!  We want to learn from data that is different from what we test on so we are sure we have learned a model that generalizes well.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Representation\n",
    "\n",
    "### Each image is 28 pixels by 28 pixels representing 784 features.\n",
    "<BR>\n",
    "<img src=\"./images/MNIST-Matrix.png\" width=\"700px\"> \n",
    "<BR> \n",
    "    \n",
    "### We will flatten the data and throw away the 2D structure of the image. We will talk about taking advantage of structure later . . .\n",
    "<BR>\n",
    "<img src=\"./images/mnist-train-xs.png\" width=\"400px\"> \n",
    "<BR>\n",
    "\n",
    "# Converting labels to \"one-hot vectors\"\n",
    "\n",
    "### A one-hot vector is a vector with 0 everywhere except having a 1 in the dimensional position representing our label. In our case the label representing our digit. \n",
    "\n",
    "#### For example a label of 3 = [0,0,0,1,0,0,0,0,0,0].\n",
    "<BR>\n",
    "<img src=\"./images/mnist-train-ys.png\" width=\"400px\"> \n",
    "<BR>    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Softmax Regression\n",
    "\n",
    "### There are 10 possibilities that a given image can be.\n",
    "\n",
    "### Softmax provides a distributin over possible outcomes. That is to say, for a given image that is actually an image of a nine our model may determine that there is a 80% chance of it being a 9, a 10% chance of an 8, and some small probability of it being one of the pother possibilities because models are not perfect!\n",
    "\n",
    "<BR>\n",
    "<img src=\"./images/softmax-weights.png\" width=\"400px\"> \n",
    "    \n",
    "#### Red = Negative weight, Blue = Positive weight\n",
    "\n",
    "<center>$evidence_i = \\displaystyle \\sum_{j=0} p(\\space W_{i,j} \\space x_j \\space + \\space b_i) $</center>\n",
    "\n",
    "<center>$y = softmax(evidence)$</center>\n",
    "\n",
    "<BR>\n",
    "<img src=\"./images/softmax-regression-scalargraph.png\" width=\"400px\"> \n",
    "<BR>\n",
    "<img src=\"./images/softmax-regression-vectorequation.png\" width=\"400px\">     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's define our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A placeholder is not a value! We just need to define how we will hold onto our training cases. \n",
    "# \"None\" means that this dimension can be any length.\n",
    "x = tf.placeholder(tf.float32, [None, 784])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need to define some variables for our weights and biases. A variable is a modifiable tensor element.\n",
    "# We will want to initialize our variables to zero.\n",
    "W = tf.Variable(tf.zeros([784, 10]))\n",
    "b = tf.Variable(tf.zeros([10]))\n",
    "y_hat = tf.matmul(x, W) + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now we need to train our model\n",
    "\n",
    "#### First we have to understand what it means  for a model to be good! IN machine learning we actually define what it means for a model to be bad. We call this our cost of loss function. Our goal is to minimize our loss.\n",
    "\n",
    "#### A commonly used loss function is \"cross-entropy\". Cross-entropy loss increases as the predicted probability diverges from the actual label. It looks something like this . . .\n",
    "\n",
    "<BR>\n",
    "<img src=\"./images/cross_entropy.png\" width=\"400px\"> \n",
    "<BR>\n",
    "\n",
    "<center>$H_{y'}(y) = - \\displaystyle \\sum_{i} y'_i \\space log(y_i) $</center>\n",
    "\n",
    "#### Where  <i>y</i> is our predicted probability distribution, <i>y'</i> is our true distribution (the one-hot vector with the digit labels)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### When we calcualte our evidence we end up with unnormalized log probabilities such as below . . .\n",
    "\n",
    "$evidence_i = \\displaystyle \\sum_{j=0} p(\\space W_{i,j} \\space x_j \\space + \\space b_i) $\n",
    "\n",
    "#### Let's assume a 3 class problem . . . \n",
    "* Training Case 1 Prediction: [ 0.5,  1.5,  0.1]\n",
    "* Training Case 2 Prediction: [ 2.2,  1.3,  1.7]\n",
    "\n",
    "#### These outputs do not sum to one, that is they are unromalized probabilities\n",
    "\n",
    "#### Now Softmax is going to normalize these into linear probabilites\n",
    "\n",
    "$y_{hat} = softmax(evidence)$\n",
    "\n",
    "* Training Case 1 Softmax: [0.227863, 0.61939586, 0.15274114]\n",
    "* Training Case 2 Softmax: [0.49674623,0.20196195,0.30129182]\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need a new placeholder for our actual label\n",
    "y = tf.placeholder(tf.float32, [None, 10])\n",
    "# Now we can define our loss function. Cross entropy with logits and we average across our batch.\n",
    "# We generate unnormalized log probabilities (aka logits) and we want the outputs normalized linear probabilities\n",
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=y, logits=y_hat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gradient Descent\n",
    "<BR>\n",
    "<img align=\"left\" style=\"float: l;\" src=\"./images/gd.png\" width=\"400px\">\n",
    "\n",
    "<img style=\"float: l;\" src=\"./images/gd-learning.png\" width=\"350px\">\n",
    "<BR>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define our gradient descent optimizer\n",
    "# We want to minimize our loss function, that is cross entropy. We will set a learning rate of 0.5\n",
    "train_step = tf.train.GradientDescentOptimizer(0.2).minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's create a Tensorflow session\n",
    "sess = tf.InteractiveSession()\n",
    "# We need to initialize the variables that we creted.\n",
    "tf.global_variables_initializer().run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting data\\train-images-idx3-ubyte.gz\n",
      "Extracting data\\train-labels-idx1-ubyte.gz\n",
      "Extracting data\\t10k-images-idx3-ubyte.gz\n",
      "Extracting data\\t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "# Let's train our model\n",
    "mnist = input_data.read_data_sets(\"data\", one_hot=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's train our model with 1000 batches of 100\n",
    "for _ in range(10000):\n",
    "    batch_xs, batch_ys = mnist.train.next_batch(100)\n",
    "    sess.run(train_step, feed_dict={x: batch_xs, y: batch_ys})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9219\n"
     ]
    }
   ],
   "source": [
    "# Test trained model\n",
    "correct_prediction = tf.equal(tf.argmax(y_hat, 1), tf.argmax(y, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "print(sess.run(accuracy, feed_dict={x: mnist.test.images, y: mnist.test.labels}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}