Basic Classification Example with TensorFlow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Classification Example with TensorFlow\n",
    "\n",
    "This notebook is a companion of [A Visual and Interactive Guide to the Basics of Neural Networks](https://jalammar.github.io/visual-interactive-guide-basics-neural-networks/).\n",
    "\n",
    "This is an example of how to do classification on a simple dataset in TensorFlow. Basically, we're building a model to help a friend choose a house to buy. She has given us the table below of houses and whether she likes them or not. We're to build a model that takes a house area and number of bathrooms as input, and outputs a prediction of whether she would like the house or not.\n",
    "\n",
    "| Area (sq ft) (x1) | Bathrooms (x2) | Label (y) |\n",
    " | --- | --- | --- |\n",
    " | 2,104 |  3 | Good |\n",
    " | 1,600 |  3 | Good |\n",
    " | 2,400 |  3 | Good |\n",
    " | 1,416 | \t2 | Bad |\n",
    " | 3,000 | \t4 | Bad |\n",
    " | 1,985 | \t4 | Good |\n",
    " | 1,534 | \t3 | Bad |\n",
    " | 1,427 | \t3 | Good |\n",
    " | 1,380 | \t3 | Good |\n",
    " | 1,494 | \t3 | Good |\n",
    " \n",
    " \n",
    " \n",
    " We'll start by loading our favorite libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd              # A beautiful library to help us work with data as tables\n",
    "import tensorflow as tf          # Fire from the gods\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll then load the house data CSV.\n",
    "Pandas is an incredible library that gives us great flexibility in dealing with table-like data.\n",
    "We load tables (or csv files, or excel sheets) into a \"data frame\", and process it however we like.\n",
    "You can think of it as a programmatic way to do a lot of the things you previously did with Excel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": "     area  bathrooms\n0  2104.0        3.0\n1  1600.0        3.0\n2  2400.0        3.0\n3  1416.0        2.0\n4  3000.0        4.0\n5  1985.0        4.0\n6  1534.0        3.0\n7  1427.0        3.0\n8  1380.0        3.0\n9  1494.0        3.0",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>area</th>\n      <th>bathrooms</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2104.0</td>\n      <td>3.0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1600.0</td>\n      <td>3.0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2400.0</td>\n      <td>3.0</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>1416.0</td>\n      <td>2.0</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>3000.0</td>\n      <td>4.0</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>1985.0</td>\n      <td>4.0</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>1534.0</td>\n      <td>3.0</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>1427.0</td>\n      <td>3.0</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>1380.0</td>\n      <td>3.0</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>1494.0</td>\n      <td>3.0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataframe = pd.read_csv(\"data.csv\") # Let's have Pandas load our dataset as a dataframe\n",
    "dataframe = dataframe.drop([\"index\", \"price\", \"sq_price\"], axis=1) # Remove columns we don't care about\n",
    "dataframe = dataframe[0:10] # We'll only use the first 10 rows of the dataset in this example\n",
    "dataframe # Let's have the notebook show us how the dataframe looks now"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "The dataframe now only has the features. Let's introduce the labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": "     area  bathrooms  y1  y2\n0  2104.0        3.0   1   0\n1  1600.0        3.0   1   0\n2  2400.0        3.0   1   0\n3  1416.0        2.0   0   1\n4  3000.0        4.0   0   1\n5  1985.0        4.0   1   0\n6  1534.0        3.0   0   1\n7  1427.0        3.0   1   0\n8  1380.0        3.0   1   0\n9  1494.0        3.0   1   0",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>area</th>\n      <th>bathrooms</th>\n      <th>y1</th>\n      <th>y2</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2104.0</td>\n      <td>3.0</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1600.0</td>\n      <td>3.0</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2400.0</td>\n      <td>3.0</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>1416.0</td>\n      <td>2.0</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>3000.0</td>\n      <td>4.0</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>1985.0</td>\n      <td>4.0</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>1534.0</td>\n      <td>3.0</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>1427.0</td>\n      <td>3.0</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>1380.0</td>\n      <td>3.0</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>1494.0</td>\n      <td>3.0</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataframe.loc[:, (\"y1\")] = [1, 1, 1, 0, 0, 1, 0, 1, 1, 1] # This is our friend's list of which houses she liked\n",
    "                                                          # 1 = good, 0 = bad\n",
    "dataframe.loc[:, (\"y2\")] = dataframe[\"y1\"] == 0           # y2 is the negation of y1\n",
    "dataframe.loc[:, (\"y2\")] = dataframe[\"y2\"].astype(int)    # Turn TRUE/FALSE values into 1/0\n",
    "# y2 means we don't like a house\n",
    "# (Yes, it's redundant. But learning to do it this way opens the door to Multiclass classification)\n",
    "dataframe # How is our dataframe looking now?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Now that we have all our data in the dataframe, we'll need to shape it in matrices to feed it to TensorFlow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "inputX = dataframe.loc[:, ['area', 'bathrooms']].values\n",
    "inputY = dataframe.loc[:, [\"y1\", \"y2\"]].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now our input matrix looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": "array([[2.104e+03, 3.000e+00],\n       [1.600e+03, 3.000e+00],\n       [2.400e+03, 3.000e+00],\n       [1.416e+03, 2.000e+00],\n       [3.000e+03, 4.000e+00],\n       [1.985e+03, 4.000e+00],\n       [1.534e+03, 3.000e+00],\n       [1.427e+03, 3.000e+00],\n       [1.380e+03, 3.000e+00],\n       [1.494e+03, 3.000e+00]])"
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And our labels matrix looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": "array([[1, 0],\n       [1, 0],\n       [1, 0],\n       [0, 1],\n       [0, 1],\n       [1, 0],\n       [0, 1],\n       [1, 0],\n       [1, 0],\n       [1, 0]])"
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's prepare some parameters for the training process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "learning_rate = 0.000001\n",
    "training_epochs = 2000\n",
    "display_step = 50\n",
    "n_samples = inputY.size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now to define the TensorFlow operations. Notice that this is a declaration step where we tell TensorFlow how the prediction is calculated. If we execute it, no calculation would be made. It would just acknowledge that it now knows how to do the operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, [None, 2])   # Okay TensorFlow, we'll feed you an array of examples. Each example will\n",
    "                                            # be an array of two float values (area, and number of bathrooms).\n",
    "                                            # \"None\" means we can feed you any number of examples\n",
    "                                            # Notice we haven't fed it the values yet\n",
    "            \n",
    "W = tf.Variable(tf.zeros([2, 2]))           # Maintain a 2 x 2 float matrix for the weights that we'll keep updating \n",
    "                                            # through the training process (make them all zero to begin with)\n",
    "    \n",
    "b = tf.Variable(tf.zeros([2]))              # Also maintain two bias values\n",
    "\n",
    "y_values = tf.add(tf.matmul(x, W), b)       # The first step in calculating the prediction would be to multiply\n",
    "                                            # the inputs matrix by the weights matrix then add the biases\n",
    "    \n",
    "y = tf.nn.softmax(y_values)                 # Then we use softmax as an \"activation function\" that translates the\n",
    "                                            # numbers outputted by the previous layer into probability form\n",
    "    \n",
    "y_ = tf.placeholder(tf.float32, [None,2])   # For training purposes, we'll also feed you a matrix of labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's specify our cost function and use Gradient Descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/javarome/anaconda3/envs/zelros-cs-nlp/lib/python3.7/site-packages/tensorflow/python/ops/math_grad.py:1205: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Cost function: Mean squared error\n",
    "cost = tf.reduce_sum(tf.pow(y_ - y, 2))/(2*n_samples)\n",
    "# Gradient descent\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/javarome/anaconda3/envs/zelros-cs-nlp/lib/python3.7/site-packages/tensorflow/python/util/tf_should_use.py:193: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.\n",
      "Instructions for updating:\n",
      "Use `tf.global_variables_initializer` instead.\n"
     ]
    }
   ],
   "source": [
    "# Initialize variabls and tensorflow session\n",
    "init = tf.initialize_all_variables()\n",
    "sess = tf.Session()\n",
    "sess.run(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Drum roll*\n",
    "\n",
    "And now for the actual training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training step: 0000 cost= 0.114958666\n",
      "Training step: 0050 cost= 0.109539941\n",
      "Training step: 0100 cost= 0.109539881\n",
      "Training step: 0150 cost= 0.109539807\n",
      "Training step: 0200 cost= 0.109539740\n",
      "Training step: 0250 cost= 0.109539680\n",
      "Training step: 0300 cost= 0.109539606\n",
      "Training step: 0350 cost= 0.109539531\n",
      "Training step: 0400 cost= 0.109539464\n",
      "Training step: 0450 cost= 0.109539390\n",
      "Training step: 0500 cost= 0.109539323\n",
      "Training step: 0550 cost= 0.109539248\n",
      "Training step: 0600 cost= 0.109539203\n",
      "Training step: 0650 cost= 0.109539129\n",
      "Training step: 0700 cost= 0.109539054\n",
      "Training step: 0750 cost= 0.109538987\n",
      "Training step: 0800 cost= 0.109538913\n",
      "Training step: 0850 cost= 0.109538853\n",
      "Training step: 0900 cost= 0.109538786\n",
      "Training step: 0950 cost= 0.109538712\n",
      "Training step: 1000 cost= 0.109538652\n",
      "Training step: 1050 cost= 0.109538577\n",
      "Training step: 1100 cost= 0.109538510\n",
      "Training step: 1150 cost= 0.109538436\n",
      "Training step: 1200 cost= 0.109538369\n",
      "Training step: 1250 cost= 0.109538309\n",
      "Training step: 1300 cost= 0.109538250\n",
      "Training step: 1350 cost= 0.109538175\n",
      "Training step: 1400 cost= 0.109538101\n",
      "Training step: 1450 cost= 0.109538034\n",
      "Training step: 1500 cost= 0.109537959\n",
      "Training step: 1550 cost= 0.109537899\n",
      "Training step: 1600 cost= 0.109537832\n",
      "Training step: 1650 cost= 0.109537773\n",
      "Training step: 1700 cost= 0.109537683\n",
      "Training step: 1750 cost= 0.109537624\n",
      "Training step: 1800 cost= 0.109537557\n",
      "Training step: 1850 cost= 0.109537482\n",
      "Training step: 1900 cost= 0.109537423\n",
      "Training step: 1950 cost= 0.109537356\n",
      "Optimization Finished!\n",
      "Training cost= 0.10953728 W= [[ 2.1414938e-04 -2.1415012e-04]\n",
      " [ 5.1274808e-05 -5.1274790e-05]] b= [ 1.19155184e-05 -1.19155275e-05] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(training_epochs):  \n",
    "    sess.run(optimizer, feed_dict={x: inputX, y_: inputY}) # Take a gradient descent step using our inputs and labels\n",
    "\n",
    "    # That's all! The rest of the cell just outputs debug messages. \n",
    "    # Display logs per epoch step\n",
    "    if (i) % display_step == 0:\n",
    "        cc = sess.run(cost, feed_dict={x: inputX, y_:inputY})\n",
    "        print(\"Training step:\", '%04d' % (i), \"cost=\", \"{:.9f}\".format(cc)) #, \\\"W=\", sess.run(W), \"b=\", sess.run(b)\n",
    "\n",
    "print(\"Optimization Finished!\")\n",
    "training_cost = sess.run(cost, feed_dict={x: inputX, y_: inputY})\n",
    "print(\"Training cost=\", training_cost, \"W=\", sess.run(W), \"b=\", sess.run(b), '\\n')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now the training is done. TensorFlow is now holding on to our trained model (Which is basically just the defined operations, plus the variables W and b that resulted from the training process).\n",
    "\n",
    "Is a cost value of 0.109537 good or bad? I have no idea. At least it's better than the first cost value of 0.114958666. Let's use the model on our dataset to see how it does, though:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": "array([[0.7112522 , 0.28874776],\n       [0.66498977, 0.33501023],\n       [0.73657656, 0.26342347],\n       [0.6471879 , 0.3528121 ],\n       [0.78335613, 0.2166439 ],\n       [0.7006948 , 0.29930523],\n       [0.6586633 , 0.34133676],\n       [0.6482863 , 0.35171375],\n       [0.6436828 , 0.35631716],\n       [0.65480113, 0.3451989 ]], dtype=float32)"
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sess.run(y, feed_dict={x: inputX })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So It's guessing they're all good houses. That makes it get 7/10 correct. Not terribly impressive. A model with a hidden layer should do better, I guess."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Btw, this is how I calculated the softmax values in the post:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": "array([0.26894143, 0.7310586 ], dtype=float32)"
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sess.run(tf.nn.softmax([1., 2.]))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "name": "python3",
   "language": "python",
   "display_name": "Python 3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}