notes-2018-01-26.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UBC Scientific Software Seminar\n",
    "\n",
    "## Hands-on Machine Learning with Scikit-Learn and TensorFlow\n",
    "\n",
    "---\n",
    "\n",
    "## Chapter 9: Up and running with TensorFlow\n",
    "\n",
    "See [Hands-on Machine Learning with Scikit-Learn and TensorfFow](https://github.com/ageron/handson-ml) (by Aurélien Géron)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.4.0'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computational graphs\n",
    "\n",
    "A [computational graph](https://www.tensorflow.org/programmers_guide/graph_viz) is a set of nodes connected by edges such that nodes represent tensor operations and edges represent the tensors flowing to and from nodes.\n",
    "\n",
    "We construct a computational graph by defining nodes for constant, variable and placeholder tensors and define nodes for operations which combine tensors by addition, matrix multiplication, etc.\n",
    "\n",
    "| Node | Description |\n",
    "| ---: | :--- |\n",
    "| `tf.constant` | tensors which do not change in the computation |\n",
    "| `tf.variable` | tensors (such as model parameters) which get updated during the computation |\n",
    "| `tf.placeholder` | tensors which get assigned input data |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Linear regression by the formula\n",
    "\n",
    "Let's construct a graph for fitting a [linear regression](https://en.wikipedia.org/wiki/Ordinary_least_squares) model. Suppoe we want to fit a linear model\n",
    "\n",
    "$$\n",
    "f(X; \\theta) = \\theta_0 + \\theta_1 X_1 + \\cdots + \\theta_m X_m\n",
    "$$\n",
    "\n",
    "to set of $n$ data points with $m$ features: $(x_{1,1},\\dots,x_{1,m}; y_1)$, $(x_{2,1},\\dots,x_{2,m}; y_2)$, ... , $(x_{n,1},\\dots,x_{n,m}; y_n)$.\n",
    "\n",
    "The model parameters $\\hat{\\theta}$ which minimize the sum of squared errors\n",
    "\n",
    "$$\n",
    "SSE(\\theta) = \\sum_i (f(x_1,x_2,\\dots,x_m; \\theta) - y_i)^2\n",
    "$$\n",
    "\n",
    "is given by the formula\n",
    "\n",
    "$$\n",
    "\\hat{\\theta} = (X^T X)^{-1} X^T y\n",
    "$$\n",
    "\n",
    "where the matrix $X$ is\n",
    "\n",
    "$$\n",
    "X = \\begin{bmatrix} 1 & x_{1,1} & \\cdots & x_{1,m} \\\\\n",
    "1 & x_{2,1} & \\cdots & x_{2,m} \\\\\n",
    "\\vdots & & & \\vdots \\\\\n",
    "1 & x_{n,1} & \\cdots & x_{n,m}\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le's import the [Califormia housing dataset](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_california_housing.html) dataset and fit a linear model to the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "\n",
    "housing = fetch_california_housing()\n",
    "m, n = housing.data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20640 8\n"
     ]
    }
   ],
   "source": [
    "print(m,n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The California housing includes 20,640 samples. Each sample is a house with 8 features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['MedInc',\n",
       " 'HouseAge',\n",
       " 'AveRooms',\n",
       " 'AveBedrms',\n",
       " 'Population',\n",
       " 'AveOccup',\n",
       " 'Latitude',\n",
       " 'Longitude']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing.feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create the computational graph to compute the model parameters and run the graph in a session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "housing_data_plus_bias = np.c_[np.ones((m, 1)), housing.data]\n",
    "\n",
    "X = tf.constant(housing_data_plus_bias, dtype=tf.float32, name=\"X\")\n",
    "y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name=\"y\")\n",
    "XT = tf.transpose(X)\n",
    "theta = tf.matmul(tf.matmul(tf.matrix_inverse(tf.matmul(XT, X)), XT), y)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    theta_value = theta.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ -3.74651413e+01],\n",
       "       [  4.35734153e-01],\n",
       "       [  9.33829229e-03],\n",
       "       [ -1.06622010e-01],\n",
       "       [  6.44106984e-01],\n",
       "       [ -4.25131839e-06],\n",
       "       [ -3.77322501e-03],\n",
       "       [ -4.26648885e-01],\n",
       "       [ -4.40514028e-01]], dtype=float32)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "theta_value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare with pure NumPy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ -3.69419202e+01]\n",
      " [  4.36693293e-01]\n",
      " [  9.43577803e-03]\n",
      " [ -1.07322041e-01]\n",
      " [  6.45065694e-01]\n",
      " [ -3.97638942e-06]\n",
      " [ -3.78654265e-03]\n",
      " [ -4.21314378e-01]\n",
      " [ -4.34513755e-01]]\n"
     ]
    }
   ],
   "source": [
    "X = housing_data_plus_bias\n",
    "y = housing.target.reshape(-1, 1)\n",
    "theta_numpy = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)\n",
    "\n",
    "print(theta_numpy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare with Scikit-Learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ -3.69419202e+01]\n",
      " [  4.36693293e-01]\n",
      " [  9.43577803e-03]\n",
      " [ -1.07322041e-01]\n",
      " [  6.45065694e-01]\n",
      " [ -3.97638942e-06]\n",
      " [ -3.78654265e-03]\n",
      " [ -4.21314378e-01]\n",
      " [ -4.34513755e-01]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "lin_reg = LinearRegression()\n",
    "lin_reg.fit(housing.data, housing.target.reshape(-1, 1))\n",
    "\n",
    "print(np.r_[lin_reg.intercept_.reshape(-1, 1), lin_reg.coef_.T])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Linear regression using batch gradient descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient Descent requires scaling the feature vectors first. We could do this using TF, but let's just use Scikit-Learn for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "scaled_housing_data = scaler.fit_transform(housing.data)\n",
    "scaled_housing_data_plus_bias = np.c_[np.ones((m, 1)), scaled_housing_data]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 MSE = 2.75443\n",
      "Epoch 100 MSE = 0.632222\n",
      "Epoch 200 MSE = 0.57278\n",
      "Epoch 300 MSE = 0.558501\n",
      "Epoch 400 MSE = 0.549069\n",
      "Epoch 500 MSE = 0.542288\n",
      "Epoch 600 MSE = 0.537379\n",
      "Epoch 700 MSE = 0.533822\n",
      "Epoch 800 MSE = 0.531243\n",
      "Epoch 900 MSE = 0.529371\n",
      "Best theta:\n",
      "[[  2.06855249e+00]\n",
      " [  7.74078071e-01]\n",
      " [  1.31192386e-01]\n",
      " [ -1.17845066e-01]\n",
      " [  1.64778143e-01]\n",
      " [  7.44078017e-04]\n",
      " [ -3.91945094e-02]\n",
      " [ -8.61356676e-01]\n",
      " [ -8.23479772e-01]]\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "n_epochs = 1000\n",
    "learning_rate = 0.01\n",
    "\n",
    "X = tf.constant(scaled_housing_data_plus_bias, dtype=tf.float32, name=\"X\")\n",
    "y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name=\"y\")\n",
    "theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name=\"theta\")\n",
    "y_pred = tf.matmul(X, theta, name=\"predictions\")\n",
    "error = y_pred - y\n",
    "mse = tf.reduce_mean(tf.square(error), name=\"mse\")\n",
    "\n",
    "gradients = tf.gradients(mse, [theta])[0]\n",
    "training_op = tf.assign(theta, theta - learning_rate * gradients)\n",
    "# Or use TensorFlow's GradientDescentOptimizer\n",
    "# optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
    "# training_op = optimizer.minimize(mse)\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    for epoch in range(n_epochs):\n",
    "        if epoch % 100 == 0:\n",
    "            print(\"Epoch\", epoch, \"MSE =\", mse.eval())\n",
    "        sess.run(training_op) \n",
    "    best_theta = theta.eval()\n",
    "\n",
    "print(\"Best theta:\")\n",
    "print(best_theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Linear regression using mini-batch gradient descent\n",
    "\n",
    "In the previous example, we compute the gradient of the mean squared error function using the **all** the data points. This can become computational expensive! Instead, mini-batch gradient descent computes gradients using a small portion (a mini batch) of the data for each step in gradient descent.\n",
    "\n",
    "We define placeholder nodes in the graph so that we can input different data sets for each step through the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_epochs = 1000\n",
    "learning_rate = 0.01\n",
    "\n",
    "tf.reset_default_graph()\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=(None, n + 1), name=\"X\")\n",
    "y = tf.placeholder(tf.float32, shape=(None, 1), name=\"y\")\n",
    "\n",
    "theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0, seed=42), name=\"theta\")\n",
    "y_pred = tf.matmul(X, theta, name=\"predictions\")\n",
    "error = y_pred - y\n",
    "mse = tf.reduce_mean(tf.square(error), name=\"mse\")\n",
    "\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\n",
    "training_op = optimizer.minimize(mse)\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "n_epochs = 10\n",
    "\n",
    "batch_size = 100\n",
    "n_batches = int(np.ceil(m / batch_size))\n",
    "\n",
    "def fetch_batch(epoch, batch_index, batch_size):\n",
    "    np.random.seed(epoch * n_batches + batch_index)  # not shown in the book\n",
    "    indices = np.random.randint(m, size=batch_size)  # not shown\n",
    "    X_batch = scaled_housing_data_plus_bias[indices] # not shown\n",
    "    y_batch = housing.target.reshape(-1, 1)[indices] # not shown\n",
    "    return X_batch, y_batch\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "        for batch_index in range(n_batches):\n",
    "            X_batch, y_batch = fetch_batch(epoch, batch_index, batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})\n",
    "\n",
    "    best_theta = theta.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2.07001591],\n",
       "       [ 0.82045609],\n",
       "       [ 0.1173173 ],\n",
       "       [-0.22739051],\n",
       "       [ 0.31134021],\n",
       "       [ 0.00353193],\n",
       "       [-0.01126994],\n",
       "       [-0.91643935],\n",
       "       [-0.87950081]], dtype=float32)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_theta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the graph with TensorBoard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mse_summary = tf.summary.scalar('MSE', mse)\n",
    "file_writer = tf.summary.FileWriter('graph', graph=tf.get_default_graph())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_epochs = 10\n",
    "batch_size = 100\n",
    "n_batches = int(np.ceil(m / batch_size))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.Session() as sess:                                                        # not shown in the book\n",
    "    sess.run(init)                                                                # not shown\n",
    "\n",
    "    for epoch in range(n_epochs):                                                 # not shown\n",
    "        for batch_index in range(n_batches):\n",
    "            X_batch, y_batch = fetch_batch(epoch, batch_index, batch_size)\n",
    "            if batch_index % 10 == 0:\n",
    "                summary_str = mse_summary.eval(feed_dict={X: X_batch, y: y_batch})\n",
    "                step = epoch * n_batches + batch_index\n",
    "                file_writer.add_summary(summary_str, step)\n",
    "            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})\n",
    "\n",
    "    best_theta = theta.eval()                                                     # not shown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "file_writer.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2.07001591],\n",
       "       [ 0.82045609],\n",
       "       [ 0.1173173 ],\n",
       "       [-0.22739051],\n",
       "       [ 0.31134021],\n",
       "       [ 0.00353193],\n",
       "       [-0.01126994],\n",
       "       [-0.91643935],\n",
       "       [-0.87950081]], dtype=float32)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_theta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enter the following command into a terminal to open Tensorboard:\n",
    "\n",
    "```\n",
    "python -m tensorboard.main --logdir graph\n",
    "```\n",
    "\n",
    "[Tensorboard](https://www.tensorflow.org/programmers_guide/summaries_and_tensorboard) is a web application to visualize Tensorflow graphs and sessions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  },
  "nav_menu": {
   "height": "603px",
   "width": "616px"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}