supporting_notebooks/Classifier Experiment 1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following notebook supports a study of classifier methods used in contemporary machine learning. This specific experiment provides Python implementations as a basis for comparisons of performance using the same dataset and evaluation criteria. This is by no means exhaustive, but illustrative in the differences between each method, and informative in which perform better on different types of data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data being used for this experiment is part of the sci-kit package, the MNIST images of handwritten digits. It is often used in machine learning as a validation set as it is a known quantity, comes cleansed, labeled, and packaged with sci-kit. It contains 60,000 training images and 10,000 test images.<br><br>\n",
    "More information on the dataset can be found here:<br>\n",
    "<a href=\"http://yann.lecun.com/exdb/mnist/\">http://yann.lecun.com/exdb/mnist/</a><br>\n",
    "<a href=\"https://srdata.nist.gov/gateway/gateway?keyword=handwriting+recognition\">https://srdata.nist.gov/gateway/gateway?keyword=handwriting+recognition</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:09.800828Z",
     "start_time": "2017-12-05T03:11:08.992644Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Data setup\n",
    "from sklearn.datasets import fetch_mldata\n",
    "mnist_dataset = fetch_mldata('MNIST original')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:13.562890Z",
     "start_time": "2017-12-05T03:11:13.552627Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['COL_NAMES', 'DESCR', 'data', 'target']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir(mnist_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:15.017590Z",
     "start_time": "2017-12-05T03:11:15.012417Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(70000, 784)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X, y = mnist_dataset['data'], mnist_dataset['target']\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:15.797364Z",
     "start_time": "2017-12-05T03:11:15.792411Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(70000,)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:17.493268Z",
     "start_time": "2017-12-05T03:11:17.482515Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,  86, 131, 225, 225, 225,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,  13,  73, 197, 253, 252, 252, 252, 252,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         4,  29,  29, 154, 187, 252, 252, 253, 252, 252, 233, 145,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,  29, 252, 253, 252, 252, 252, 252, 253, 204, 112,  37,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0, 169, 253, 255, 253, 228, 126,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,  98, 243, 252, 253, 252, 246, 130,  38,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,  98, 240, 252, 252, 253, 252, 252,\n",
       "       252, 221,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0, 225, 252, 252, 236, 225,\n",
       "       223, 230, 252, 252,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0, 146, 252, 157,\n",
       "        50,   0,   0,  25, 205, 252,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,  26, 207, 253,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,  29,  19,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,  73, 205, 252,  79,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0, 120, 215, 209,\n",
       "       175,   0,   0,   0,   0,   0,   0,   0,  19, 209, 252, 220,  79,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0, 174,\n",
       "       252, 252, 239, 140,   0,   0,   0,   0,   0,  29, 104, 252, 249,\n",
       "       177,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0, 174, 252, 252, 223,   0,   0,   0,   0,   0,   0, 174, 252,\n",
       "       252, 223,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0, 141, 241, 253, 146,   0,   0,   0,   0, 169, 253,\n",
       "       255, 253, 253,  84,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0, 178, 252, 154,  85,  85, 210, 225,\n",
       "       243, 252, 215, 121,  27,   9,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,  66, 208, 220, 252, 253,\n",
       "       252, 252, 214, 195,  31,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,  19,  37,\n",
       "        84, 146, 223, 114,  28,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "         0,   0,   0,   0], dtype=uint8)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Let us examine the raw data of a single image\n",
    "X[36000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:19.094260Z",
     "start_time": "2017-12-05T03:11:19.082573Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,  86, 131, 225, 225, 225,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,  13,  73, 197, 253, 252, 252, 252, 252,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   4,\n",
       "         29,  29, 154, 187, 252, 252, 253, 252, 252, 233, 145,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,  29,\n",
       "        252, 253, 252, 252, 252, 252, 253, 204, 112,  37,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0, 169,\n",
       "        253, 255, 253, 228, 126,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,  98, 243,\n",
       "        252, 253, 252, 246, 130,  38,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,  98, 240, 252,\n",
       "        252, 253, 252, 252, 252, 221,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0, 225, 252, 252,\n",
       "        236, 225, 223, 230, 252, 252,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0, 146, 252, 157,\n",
       "         50,   0,   0,  25, 205, 252,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,  26, 207, 253,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,  29,  19,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,  73, 205, 252,  79,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0, 120, 215, 209, 175,   0,   0,   0,   0,   0,\n",
       "          0,   0,  19, 209, 252, 220,  79,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0, 174, 252, 252, 239, 140,   0,   0,   0,   0,\n",
       "          0,  29, 104, 252, 249, 177,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0, 174, 252, 252, 223,   0,   0,   0,   0,   0,\n",
       "          0, 174, 252, 252, 223,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0, 141, 241, 253, 146,   0,   0,   0,   0, 169,\n",
       "        253, 255, 253, 253,  84,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0, 178, 252, 154,  85,  85, 210, 225, 243,\n",
       "        252, 215, 121,  27,   9,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,  66, 208, 220, 252, 253, 252, 252, 214,\n",
       "        195,  31,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,  19,  37,  84, 146, 223, 114,  28,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "          0,   0]], dtype=uint8)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_digit = X[36000]\n",
    "#Reshape to 28 x 28 for the 784 available features\n",
    "random_digit_image = random_digit.reshape(28,28)\n",
    "random_digit_image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:21.049427Z",
     "start_time": "2017-12-05T03:11:20.602664Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x11593ef98>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADa9JREFUeJzt3X+IHPUZx/HPk9gqmCDJieFMo4kSC/VXCkfwjyANrWLP\nagxoUBCvVnqCbamg0mANFUshlGhQECHB0EttTSoqxlgqGkptoYgxpGrUxjS54CUxqZeG2vhHavL0\nj52UM958Z7M7s7OX5/2C43bn2dl52ORzM7Pf3fmauwtAPJPqbgBAPQg/EBThB4Ii/EBQhB8IivAD\nQRF+ICjCDwRF+IGgTuvkxsyMjxMCFXN3a+Zxbe35zewaM/u7me0ws6XtPBeAzrJWP9tvZpMlbZd0\nlaQRSW9IusXd302sw54fqFgn9vzzJe1w953ufkTSOkmL2ng+AB3UTvhnSvpwzP2RbNnnmNmgmW02\ns81tbAtAySp/w8/dV0laJXHYD3STdvb8eyTNGnP/K9kyABNAO+F/Q9JcM5tjZl+WdLOkDeW0BaBq\nLR/2u/tnZvZDSS9LmixpjbtvK60zAJVqeaivpY1xzg9UriMf8gEwcRF+ICjCDwRF+IGgCD8QFOEH\ngiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBTh\nB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QVMtTdEuSmQ1L+kTSUUmfuXtfGU0BzViyZEmyfuml\nl+bWbrvttrLb+Zzzzz+/0ucvQ1vhzyx0949LeB4AHcRhPxBUu+F3Sa+a2ZtmNlhGQwA6o93D/gXu\nvsfMzpH0ipm97+6vjX1A9keBPwxAl2lrz+/ue7LfByQ9L2n+OI9Z5e59vBkIdJeWw29mZ5rZ1OO3\nJV0t6Z2yGgNQrXYO+2dIet7Mjj/Pb939D6V0BaBy5u6d25hZ5zaGrjB16tTc2oIFC5LrPvDAA8n6\nFVdckax38v/2iU47rYxR9Na4uzXzOIb6gKAIPxAU4QeCIvxAUIQfCIrwA0HVNx6Bjigacurt7W3r\n+ZctW5asz5kzJ7e2cOHCtrZdpdHR0WR93bp1HeqkOuz5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAo\nxvlPcUXj+Lt27UrWs+s15Krza7NFtm7dmlt76qmnkutu3LgxWd+xY0dLPXUT9vxAUIQfCIrwA0ER\nfiAowg8ERfiBoAg/EBTj/Ke4FStWJOtF4/iTJqX3D8eOHUvW9+7dm1tbvXp1ct2HHnooWUd72PMD\nQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCF4/xmtkbSdyQdcPdLsmXTJa2XNFvSsKQl7v6v6tpEyu23\n355b6+/vT65b9H38onH8ovV7enpya0VzClx00UXJ+vbt25N1pDWz5/+VpGtOWLZU0iZ3nytpU3Yf\nwARSGH53f03SwRMWL5I0lN0eknRDyX0BqFir5/wz3H1fdvsjSTNK6gdAh7T92X53dzPLPfEzs0FJ\ng+1uB0C5Wt3z7zezXknKfh/Ie6C7r3L3Pnfva3FbACrQavg3SBrIbg9IeqGcdgB0SmH4zexpSX+V\n9FUzGzGzOyQtl3SVmX0g6VvZfQATiHXyuuup9waQLzWOL0krV67MrU2ZMqWtbdd53f7du3cn6xde\neGFl257I3D39j5bhE35AUIQfCIrwA0ERfiAowg8ERfiBoBjqmwCKhrxmzpzZ8nMfOnQoWT98+HCy\nXvSV39NPPz23ds455yTXLVL0leCoGOoDkET4gaAIPxAU4QeCIvxAUIQfCIrwA0ExUDoBvPBC+lop\nd911V25taGgotyZJjz/+eLK+ZcuWZL1Ib29vbm3jxo3JdS+//PK2to009vxAUIQfCIrwA0ERfiAo\nwg8ERfiBoAg/EBTf50elzj333NzaSy+9lFz3sssuS9YnT57cUk+nOr7PDyCJ8ANBEX4gKMIPBEX4\ngaAIPxAU4QeCKvw+v5mtkfQdSQfc/ZJs2YOSvi/pn9nD7nf331fVZCfMmjUrWf/0009za6Ojo2W3\nc8pIfSe/6Jr/RZ9BWbRoUbJedB2E6JrZ8/9K0jXjLF/p7vOynwkdfCCiwvC7+2uSDnagFwAd1M45\n/4/M7C0zW2Nm00rrCEBHtBr+JyRdIGmepH2SHs57oJkNmtlmM9vc4rYAVKCl8Lv7fnc/6u7HJK2W\nND/x2FXu3ufufa02CaB8LYXfzMZeknWxpHfKaQdApzQz1Pe0pG9IOtvMRiT9TNI3zGyeJJc0LOnO\nCnsEUIHC8Lv7LeMsfrKCXiq1dOnSZH1gYCBZP3LkSG5t586dyXUXL16crE9kPT09yfry5ctzaxdf\nfHFy3eHh4VZaQpP4hB8QFOEHgiL8QFCEHwiK8ANBEX4gqDBTdM+fn/shREnS3LlzW37u8847L1lf\nsWJFsn7vvfe2vO2qFX3V+dprr03WU8N5R48eTa67bdu2ZJ2v7LaHPT8QFOEHgiL8QFCEHwiK8ANB\nEX4gKMIPBBVmnL9Khw4dSta7eRy/yKOPPpqsX3/99S0/9969e5P1oktzoz3s+YGgCD8QFOEHgiL8\nQFCEHwiK8ANBEX4gqDDj/Lt27UrWDx8+nKxPmTIlt/biiy+21FMnrF+/Plm/8cYbk/VJk9L7h6Jp\ntlPuu+++ltdF+9jzA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQheP8ZjZL0lpJMyS5pFXu/qiZTZe0\nXtJsScOSlrj7v6prtT333HNPsl503f7+/v7c2hlnnJFc96abbkrWiyxbtixZP+uss3Jr06dPT67r\n7sn6+++/n6yvXbu25frBgweT66Jazez5P5N0j7t/TdIVkn5gZl+TtFTSJnefK2lTdh/ABFEYfnff\n5+5bstufSHpP0kxJiyQNZQ8bknRDVU0CKN9JnfOb2WxJX5f0uqQZ7r4vK32kxmkBgAmi6c/2m9kU\nSc9Kutvd/21m/6+5u5vZuCePZjYoabDdRgGUq6k9v5l9SY3g/8bdn8sW7zez3qzeK+nAeOu6+yp3\n73P3vjIaBlCOwvBbYxf/pKT33P2RMaUNkgay2wOSmDIVmECsaKjHzBZI+rOktyUd//7m/Wqc9/9O\n0nmSdqsx1Jccu8k7NegGCxcuTNZXrlyZW0t93VeSZs+enayPPYUaT9G/Ucr27duT9dHR0WT91ltv\nTdZ379590j2hWu6e/g+VKTznd/e/SMp7sm+eTFMAugef8AOCIvxAUIQfCIrwA0ERfiAowg8EVTjO\nX+rGunicv0hPT09ubdq0acl1582bl6xfeeWVyfp1112XrD/22GO5tWeeeSa57sjISLKOiafZcX72\n/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOP8wCmGcX4ASYQfCIrwA0ERfiAowg8ERfiBoAg/EBTh\nB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QVGH4zWyWmf3RzN41s21m9uNs+YNmtsfMtmY//dW3\nC6AshRfzMLNeSb3uvsXMpkp6U9INkpZI+o+7r2h6Y1zMA6hcsxfzOK2JJ9onaV92+xMze0/SzPba\nA1C3kzrnN7PZkr4u6fVs0Y/M7C0zW2Nm485ZZWaDZrbZzDa31SmAUjV9DT8zmyLpT5J+4e7PmdkM\nSR9Lckk/V+PU4HsFz8FhP1CxZg/7mwq/mX1J0kZJL7v7I+PUZ0va6O6XFDwP4QcqVtoFPM3MJD0p\n6b2xwc/eCDxusaR3TrZJAPVp5t3+BZL+LOltSceyxfdLukXSPDUO+4cl3Zm9OZh6Lvb8QMVKPewv\nC+EHqsd1+wEkEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Iq\nvIBnyT6WtHvM/bOzZd2oW3vr1r4kemtVmb2d3+wDO/p9/i9s3Gyzu/fV1kBCt/bWrX1J9Naqunrj\nsB8IivADQdUd/lU1bz+lW3vr1r4kemtVLb3Ves4PoD517/kB1KSW8JvZNWb2dzPbYWZL6+ghj5kN\nm9nb2czDtU4xlk2DdsDM3hmzbLqZvWJmH2S/x50mrabeumLm5sTM0rW+dt0243XHD/vNbLKk7ZKu\nkjQi6Q1Jt7j7ux1tJIeZDUvqc/fax4TN7EpJ/5G09vhsSGb2S0kH3X159odzmrv/pEt6e1AnOXNz\nRb3lzSz9XdX42pU543UZ6tjzz5e0w913uvsRSeskLaqhj67n7q9JOnjC4kWShrLbQ2r85+m4nN66\ngrvvc/ct2e1PJB2fWbrW1y7RVy3qCP9MSR+OuT+i7pry2yW9amZvmtlg3c2MY8aYmZE+kjSjzmbG\nUThzcyedMLN017x2rcx4XTbe8PuiBe4+T9K3Jf0gO7ztSt44Z+um4ZonJF2gxjRu+yQ9XGcz2czS\nz0q6293/PbZW52s3Tl+1vG51hH+PpFlj7n8lW9YV3H1P9vuApOfVOE3pJvuPT5Ka/T5Qcz//5+77\n3f2oux+TtFo1vnbZzNLPSvqNuz+XLa79tRuvr7petzrC/4akuWY2x8y+LOlmSRtq6OMLzOzM7I0Y\nmdmZkq5W980+vEHSQHZ7QNILNfbyOd0yc3PezNKq+bXruhmv3b3jP5L61XjH/x+SflpHDzl9XSDp\nb9nPtrp7k/S0GoeB/1XjvZE7JPVI2iTpA0mvSpreRb39Wo3ZnN9SI2i9NfW2QI1D+rckbc1++ut+\n7RJ91fK68Qk/ICje8AOCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/ENT/AN78Xa555jiCAAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10e167c50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Let's plot to see what the actual image looks like\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.imshow(random_digit_image, cmap='gray')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:22.887876Z",
     "start_time": "2017-12-05T03:11:22.882856Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Check the label\n",
    "y[36000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the dataset is established, we now split into training and test sets. There are a variety of means of handling this including more advanced techniques like stratified sampling. For the sake of simplicity and abstraction we use the split capabilities of sklearn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:25.124368Z",
     "start_time": "2017-12-05T03:11:25.051678Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(mnist_dataset.data, mnist_dataset.target, \n",
    "                                                    test_size=0.1, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:26.146830Z",
     "start_time": "2017-12-05T03:11:26.142136Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(63000, 784)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:26.977018Z",
     "start_time": "2017-12-05T03:11:26.972455Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7000, 784)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:27.826492Z",
     "start_time": "2017-12-05T03:11:27.822497Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7000,)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:29.256559Z",
     "start_time": "2017-12-05T03:11:29.252283Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(63000,)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logistic Regression is a versatile classifier that can be used for binary and multiclass categorization problems. At its heart is a decision function that assess a probability that will determine which categorization is selected by the algorithm. In a multinomial case it will take a \"one vs rest\" evaluation approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:11:31.572279Z",
     "start_time": "2017-12-05T03:11:31.522435Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "logregress = LogisticRegression(solver = 'lbfgs')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:12:13.084313Z",
     "start_time": "2017-12-05T03:11:32.912394Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='lbfgs', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Train the model\n",
    "logregress.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:12:19.732121Z",
     "start_time": "2017-12-05T03:12:19.693117Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Now run predictions using the test set\n",
    "logregress_predictions = logregress.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:12:22.401433Z",
     "start_time": "2017-12-05T03:12:22.373080Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.916\n"
     ]
    }
   ],
   "source": [
    "#The mean accuracy on given test data and labels\n",
    "#In multi-label classification, this is the  accuracy for each label's correct prediction\n",
    "logregress_score = logregress.score(X_test, y_test)\n",
    "print (logregress_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:12:27.617349Z",
     "start_time": "2017-12-05T03:12:27.613045Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Confirming that a multinomial model is being used\n",
    "logregress.classes_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examine a subset of the predicted values against the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:15:51.491632Z",
     "start_time": "2017-12-05T03:15:51.486707Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 3.,  8.,  5.,  8.,  1.,  2.,  3.,  7.,  5.,  5.])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logregress_predictions[20:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:16:02.341482Z",
     "start_time": "2017-12-05T03:16:02.336469Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 9.,  8.,  5.,  8.,  1.,  2.,  3.,  7.,  5.,  5.])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test[20:30]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some inaccuracies are seen. Let's look at the performance mesasures to determine which labels were recognized better than others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:27:59.207452Z",
     "start_time": "2017-12-05T03:27:59.196971Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        0.0       0.96      0.97      0.96       660\n",
      "        1.0       0.95      0.97      0.96       779\n",
      "        2.0       0.91      0.90      0.91       688\n",
      "        3.0       0.89      0.89      0.89       733\n",
      "        4.0       0.92      0.92      0.92       640\n",
      "        5.0       0.90      0.84      0.87       651\n",
      "        6.0       0.95      0.98      0.96       738\n",
      "        7.0       0.93      0.93      0.93       724\n",
      "        8.0       0.86      0.86      0.86       672\n",
      "        9.0       0.89      0.89      0.89       715\n",
      "\n",
      "avg / total       0.92      0.92      0.92      7000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Additional performance measures\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "print(classification_report(y_test, logregress_predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T00:39:34.459482Z",
     "start_time": "2017-12-05T00:39:34.445487Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix:\n",
      "[[640   0   2   2   2   3   5   1   5   0]\n",
      " [  0 755   5   3   1   2   3   0   9   1]\n",
      " [  1   6 622  10   5   3   7  11  23   0]\n",
      " [  0   6  20 655   0  19   2   8  16   7]\n",
      " [  5   2   3   1 588   2   7   3   4  25]\n",
      " [  4   4   9  28   9 548  13   4  24   8]\n",
      " [  3   2   4   0   3   4 720   0   2   0]\n",
      " [  4   1  10   2   7   0   0 671   3  26]\n",
      " [  4  14  10  20   3  26   3   4 578  10]\n",
      " [  7   2   1  14  24   5   0  20   7 635]]\n"
     ]
    }
   ],
   "source": [
    "#Confusion matrix. Higher values along the diagonal are better\n",
    "print(\"Confusion matrix:\\n%s\" % confusion_matrix(y_test, logregress_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mean accuracy: 91.6% <br>\n",
    "Precision: 92% <br>\n",
    "Recall: 92% <br>\n",
    "F1-Score: 92% <br>\n",
    "Labels with the most errors: \"3\", \"8\", \"9\" <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stochastic Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stochastic Gradient Descent is in the family of Gradient Descent algorithms. It takes a sample of feature vectors, hence its stochastic nature, and performs an incremental gradient descent by trying to find a minimum. It is generally faster in its execution than mini-batch gradient descent, and is faster than full-batch gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:27:20.508021Z",
     "start_time": "2017-12-05T03:27:20.505242Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import SGDClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:27:27.235904Z",
     "start_time": "2017-12-05T03:27:23.056166Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGDClassifier(alpha=0.0001, average=False, class_weight=None, epsilon=0.1,\n",
       "       eta0=0.0, fit_intercept=True, l1_ratio=0.15,\n",
       "       learning_rate='optimal', loss='hinge', n_iter=5, n_jobs=1,\n",
       "       penalty='l2', power_t=0.5, random_state=42, shuffle=True, verbose=0,\n",
       "       warm_start=False)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_clf = SGDClassifier(random_state=42)\n",
    "sgd_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:27:40.023371Z",
     "start_time": "2017-12-05T03:27:39.996454Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sgd_predictions = sgd_clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:27:41.903142Z",
     "start_time": "2017-12-05T03:27:41.875723Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.889142857143\n"
     ]
    }
   ],
   "source": [
    "sgd_score = sgd_clf.score(X_test, y_test)\n",
    "print(sgd_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:28:04.706901Z",
     "start_time": "2017-12-05T03:28:04.697645Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        0.0       0.91      0.97      0.94       660\n",
      "        1.0       0.95      0.95      0.95       779\n",
      "        2.0       0.89      0.90      0.89       688\n",
      "        3.0       0.87      0.88      0.87       733\n",
      "        4.0       0.92      0.89      0.90       640\n",
      "        5.0       0.85      0.78      0.81       651\n",
      "        6.0       0.97      0.92      0.94       738\n",
      "        7.0       0.88      0.92      0.90       724\n",
      "        8.0       0.78      0.85      0.81       672\n",
      "        9.0       0.87      0.81      0.84       715\n",
      "\n",
      "avg / total       0.89      0.89      0.89      7000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, sgd_predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:28:08.529907Z",
     "start_time": "2017-12-05T03:28:08.516332Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix:\n",
      "[[642   0   4   3   2   1   2   2   4   0]\n",
      " [  0 742   9   4   1   7   0   0  14   2]\n",
      " [  0   8 620  14   5   2   5  13  18   3]\n",
      " [  4   2  15 643   1  32   1  10  17   8]\n",
      " [  5   0   2   1 570   2   4   7  22  27]\n",
      " [ 13   5  10  24  12 509   8   7  57   6]\n",
      " [ 18   1  17   0   2  16 679   0   5   0]\n",
      " [  4   0   5   4  10   1   1 669   0  30]\n",
      " [  3  18  15  26   1  19   1  10 570   9]\n",
      " [ 13   1   1  23  17  13   0  43  24 580]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Confusion matrix:\\n%s\" % confusion_matrix(y_test, sgd_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mean accuracy: 88.9% <br>\n",
    "Precision: 89% <br>\n",
    "Recall: 89% <br>\n",
    "F1-Score: 89% <br>\n",
    "Labels with the most errors: \"5\", \"3\", \"8\", \"7\" <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scalable Vector Machine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scalable Vector Machine works by expressing observations into multidimensional space. The algorithm forms \"roads\" separating the different categories by creating hyperplanes for their separation and classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:34:38.631343Z",
     "start_time": "2017-12-05T03:32:30.052689Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearSVC(C=1, class_weight=None, dual=True, fit_intercept=True,\n",
       "     intercept_scaling=1, loss='hinge', max_iter=1000, multi_class='ovr',\n",
       "     penalty='l2', random_state=None, tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "scalable_vec_machine = LinearSVC(C=1, loss=\"hinge\")\n",
    "scalable_vec_machine.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:38:02.842060Z",
     "start_time": "2017-12-05T03:38:02.805587Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "svm_predictions = scalable_vec_machine.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:38:03.693490Z",
     "start_time": "2017-12-05T03:38:03.665691Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.865\n"
     ]
    }
   ],
   "source": [
    "svn_score = scalable_vec_machine.score(X_test, y_test)\n",
    "print(svn_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adjusting hyperparameters to try coaxing better performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:40:17.350422Z",
     "start_time": "2017-12-05T03:38:07.545708Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearSVC(C=4, class_weight=None, dual=True, fit_intercept=True,\n",
       "     intercept_scaling=1, loss='hinge', max_iter=1000, multi_class='ovr',\n",
       "     penalty='l2', random_state=None, tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scalable_vec_machinel2 = LinearSVC(C=4, loss=\"hinge\")\n",
    "scalable_vec_machinel2.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:40:21.164376Z",
     "start_time": "2017-12-05T03:40:21.137677Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "svm_predictions2 = scalable_vec_machinel2.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:40:22.594444Z",
     "start_time": "2017-12-05T03:40:22.567602Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.867142857143\n"
     ]
    }
   ],
   "source": [
    "svn_score2 = scalable_vec_machinel2.score(X_test, y_test)\n",
    "print(svn_score2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:42:44.386915Z",
     "start_time": "2017-12-05T03:40:34.757390Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearSVC(C=0.5, class_weight=None, dual=True, fit_intercept=True,\n",
       "     intercept_scaling=1, loss='hinge', max_iter=1000, multi_class='ovr',\n",
       "     penalty='l2', random_state=None, tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scalable_vec_machinel3 = LinearSVC(C=0.5, loss=\"hinge\")\n",
    "scalable_vec_machinel3.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:42:51.639006Z",
     "start_time": "2017-12-05T03:42:51.610749Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.865\n"
     ]
    }
   ],
   "source": [
    "svn_score3 = scalable_vec_machinel3.score(X_test, y_test)\n",
    "print(svn_score3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decrease in mean accuracy. Selecting the second model for performance measures and evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:43:00.345750Z",
     "start_time": "2017-12-05T03:43:00.337216Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        0.0       0.84      0.98      0.91       660\n",
      "        1.0       0.97      0.95      0.96       779\n",
      "        2.0       0.71      0.92      0.80       688\n",
      "        3.0       0.93      0.74      0.82       733\n",
      "        4.0       0.93      0.88      0.90       640\n",
      "        5.0       0.82      0.79      0.81       651\n",
      "        6.0       0.93      0.96      0.94       738\n",
      "        7.0       0.81      0.93      0.87       724\n",
      "        8.0       0.86      0.72      0.78       672\n",
      "        9.0       0.91      0.79      0.85       715\n",
      "\n",
      "avg / total       0.87      0.87      0.87      7000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, svm_predictions2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:43:10.662242Z",
     "start_time": "2017-12-05T03:43:10.649904Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix:\n",
      "[[646   0   1   2   1   2   3   3   2   0]\n",
      " [  1 738  10   1   1  11   3   4   9   1]\n",
      " [  3   2 633   5   2   4  12  13  14   0]\n",
      " [ 15   4 100 542   0  28   4  25   9   6]\n",
      " [ 17   2  12   0 562   6   4   8   8  21]\n",
      " [ 32   3  24  11   5 515  24   6  22   9]\n",
      " [ 13   1   7   0   2   1 711   1   2   0]\n",
      " [  8   0  22   0   7   0   0 676   3   8]\n",
      " [ 18   8  68  14   5  47   7  14 483   8]\n",
      " [ 14   1   9   9  18  11   0  80   9 564]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Confusion matrix:\\n%s\" % confusion_matrix(y_test, svm_predictions2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-03T03:21:48.219818Z",
     "start_time": "2017-12-03T03:21:48.213851Z"
    }
   },
   "source": [
    "Mean accuracy: 88.3% <br>\n",
    "Precision: 88% <br>\n",
    "Recall: 88% <br>\n",
    "F1-Score: 88% <br>\n",
    "Labels with the most errors: \"3\", \"8\", \"5\" <br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Perceptron model is a fundamental building block of Artificial Neural Networks, it is closely related to Stochastic Gradient Descent and follows the same iterative procedure of minimizing an error function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:53:32.507777Z",
     "start_time": "2017-12-05T03:53:28.392377Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Perceptron(alpha=0.0001, class_weight=None, eta0=1.0, fit_intercept=True,\n",
       "      n_iter=5, n_jobs=1, penalty=None, random_state=0, shuffle=True,\n",
       "      verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import Perceptron\n",
    "perceptron_clf = Perceptron()\n",
    "perceptron_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:54:03.326443Z",
     "start_time": "2017-12-05T03:54:03.301299Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "perceptron_predictions = perceptron_clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:57:51.305293Z",
     "start_time": "2017-12-05T03:57:51.277840Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.89\n"
     ]
    }
   ],
   "source": [
    "perceptron_score = perceptron_clf.score(X_test, y_test)\n",
    "print(perceptron_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:54:05.983622Z",
     "start_time": "2017-12-05T03:54:05.972936Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        0.0       0.97      0.94      0.96       660\n",
      "        1.0       0.96      0.94      0.95       779\n",
      "        2.0       0.89      0.88      0.88       688\n",
      "        3.0       0.84      0.89      0.87       733\n",
      "        4.0       0.88      0.92      0.90       640\n",
      "        5.0       0.85      0.84      0.84       651\n",
      "        6.0       0.97      0.92      0.95       738\n",
      "        7.0       0.93      0.86      0.90       724\n",
      "        8.0       0.76      0.85      0.81       672\n",
      "        9.0       0.86      0.83      0.85       715\n",
      "\n",
      "avg / total       0.89      0.89      0.89      7000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, perceptron_predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T03:54:10.355953Z",
     "start_time": "2017-12-05T03:54:10.341357Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix:\n",
      "[[623   0   5   5   2   9   4   2  10   0]\n",
      " [  0 736   2   9   1  10   0   1  18   2]\n",
      " [  1   8 605  21   5   3   5   6  31   3]\n",
      " [  0   1  21 656   1  24   1   5  19   5]\n",
      " [  4   1   7   1 587   1   4   2  14  19]\n",
      " [  2   2  10  29   9 547   7   3  36   6]\n",
      " [  3   2  11   0  11  15 681   0  15   0]\n",
      " [  1   0   8  12  10   1   0 626   9  57]\n",
      " [  2  13   8  30   6  27   1   6 572   7]\n",
      " [  6   2   3  17  36   7   0  22  25 597]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Confusion matrix:\\n%s\" % confusion_matrix(y_test, perceptron_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multilayer Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Multilayer Perceptron is a type of Artifical Neural Network. It differs from the Perceptron model in that it can contain one or many hidden layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T00:46:26.611914Z",
     "start_time": "2017-12-05T00:46:26.599487Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.neural_network import MLPClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T01:02:32.143286Z",
     "start_time": "2017-12-05T01:02:32.139891Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp_clf = MLPClassifier(solver='sgd', activation='relu', hidden_layer_sizes=(100), alpha=0.0001,\n",
    "                       learning_rate_init=0.001, learning_rate='adaptive', verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T01:03:55.276623Z",
     "start_time": "2017-12-05T01:02:35.215290Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 1, loss = 2.02638774\n",
      "Iteration 2, loss = 1.45543766\n",
      "Iteration 3, loss = 1.29047532\n",
      "Iteration 4, loss = 1.09103654\n",
      "Iteration 5, loss = 0.91984541\n",
      "Iteration 6, loss = 0.77203245\n",
      "Iteration 7, loss = 0.73428491\n",
      "Iteration 8, loss = 0.64824450\n",
      "Iteration 9, loss = 0.61352885\n",
      "Iteration 10, loss = 0.58031264\n",
      "Iteration 11, loss = 0.55977070\n",
      "Iteration 12, loss = 0.56601589\n",
      "Iteration 13, loss = 0.53656548\n",
      "Iteration 14, loss = 0.52808469\n",
      "Iteration 15, loss = 0.53156162\n",
      "Iteration 16, loss = 0.51933094\n",
      "Iteration 17, loss = 0.49848795\n",
      "Iteration 18, loss = 0.45651464\n",
      "Iteration 19, loss = 0.42979631\n",
      "Iteration 20, loss = 0.41488604\n",
      "Iteration 21, loss = 0.40217192\n",
      "Iteration 22, loss = 0.39249995\n",
      "Iteration 23, loss = 0.38874476\n",
      "Iteration 24, loss = 0.38057261\n",
      "Iteration 25, loss = 0.37540719\n",
      "Iteration 26, loss = 0.36900653\n",
      "Iteration 27, loss = 0.36740460\n",
      "Iteration 28, loss = 0.35793926\n",
      "Iteration 29, loss = 0.36097530\n",
      "Iteration 30, loss = 0.35740798\n",
      "Iteration 31, loss = 0.34999102\n",
      "Iteration 32, loss = 0.35177729\n",
      "Iteration 33, loss = 0.34350552\n",
      "Iteration 34, loss = 0.33764916\n",
      "Iteration 35, loss = 0.32831460\n",
      "Iteration 36, loss = 0.32627591\n",
      "Iteration 37, loss = 0.31965222\n",
      "Iteration 38, loss = 0.31389076\n",
      "Iteration 39, loss = 0.31257104\n",
      "Iteration 40, loss = 0.30521022\n",
      "Iteration 41, loss = 0.30521157\n",
      "Iteration 42, loss = 0.29642763\n",
      "Iteration 43, loss = 0.29004672\n",
      "Iteration 44, loss = 0.28786145\n",
      "Iteration 45, loss = 0.28627437\n",
      "Iteration 46, loss = 0.27921789\n",
      "Iteration 47, loss = 0.28060206\n",
      "Iteration 48, loss = 0.27779030\n",
      "Iteration 49, loss = 0.27549928\n",
      "Iteration 50, loss = 0.27372181\n",
      "Iteration 51, loss = 0.27179756\n",
      "Iteration 52, loss = 0.27064230\n",
      "Iteration 53, loss = 0.26752518\n",
      "Iteration 54, loss = 0.26692129\n",
      "Iteration 55, loss = 0.26618607\n",
      "Iteration 56, loss = 0.26463946\n",
      "Iteration 57, loss = 0.26338946\n",
      "Iteration 58, loss = 0.26414689\n",
      "Iteration 59, loss = 0.26129096\n",
      "Iteration 60, loss = 0.25973335\n",
      "Iteration 61, loss = 0.26004826\n",
      "Iteration 62, loss = 0.25478196\n",
      "Iteration 63, loss = 0.25650022\n",
      "Iteration 64, loss = 0.25646323\n",
      "Iteration 65, loss = 0.26063907\n",
      "Training loss did not improve more than tol=0.000100 for two consecutive epochs. Setting learning rate to 0.000200\n",
      "Iteration 66, loss = 0.22046218\n",
      "Iteration 67, loss = 0.21112130\n",
      "Iteration 68, loss = 0.20845135\n",
      "Iteration 69, loss = 0.20773587\n",
      "Iteration 70, loss = 0.20704736\n",
      "Iteration 71, loss = 0.20607154\n",
      "Iteration 72, loss = 0.20587947\n",
      "Iteration 73, loss = 0.20551674\n",
      "Iteration 74, loss = 0.20499631\n",
      "Iteration 75, loss = 0.20506154\n",
      "Iteration 76, loss = 0.20427319\n",
      "Iteration 77, loss = 0.20383603\n",
      "Iteration 78, loss = 0.20392356\n",
      "Iteration 79, loss = 0.20397949\n",
      "Iteration 80, loss = 0.20330823\n",
      "Iteration 81, loss = 0.20295067\n",
      "Iteration 82, loss = 0.20327632\n",
      "Iteration 83, loss = 0.20261323\n",
      "Iteration 84, loss = 0.20188432\n",
      "Iteration 85, loss = 0.20160500\n",
      "Iteration 86, loss = 0.20189259\n",
      "Iteration 87, loss = 0.20166229\n",
      "Iteration 88, loss = 0.20203002\n",
      "Training loss did not improve more than tol=0.000100 for two consecutive epochs. Setting learning rate to 0.000040\n",
      "Iteration 89, loss = 0.19141139\n",
      "Iteration 90, loss = 0.19059451\n",
      "Iteration 91, loss = 0.19031843\n",
      "Iteration 92, loss = 0.19028549\n",
      "Iteration 93, loss = 0.19007643\n",
      "Iteration 94, loss = 0.19022295\n",
      "Iteration 95, loss = 0.18980815\n",
      "Iteration 96, loss = 0.19011408\n",
      "Iteration 97, loss = 0.18994743\n",
      "Iteration 98, loss = 0.18970348\n",
      "Iteration 99, loss = 0.18970683\n",
      "Iteration 100, loss = 0.18972812\n",
      "Iteration 101, loss = 0.18965875\n",
      "Training loss did not improve more than tol=0.000100 for two consecutive epochs. Setting learning rate to 0.000008\n",
      "Iteration 102, loss = 0.18722194\n",
      "Iteration 103, loss = 0.18700946\n",
      "Iteration 104, loss = 0.18699765\n",
      "Iteration 105, loss = 0.18690006\n",
      "Iteration 106, loss = 0.18687618\n",
      "Training loss did not improve more than tol=0.000100 for two consecutive epochs. Setting learning rate to 0.000002\n",
      "Iteration 107, loss = 0.18649513\n",
      "Iteration 108, loss = 0.18634850\n",
      "Iteration 109, loss = 0.18631616\n",
      "Iteration 110, loss = 0.18632223\n",
      "Iteration 111, loss = 0.18630603\n",
      "Training loss did not improve more than tol=0.000100 for two consecutive epochs. Setting learning rate to 0.000000\n",
      "Iteration 112, loss = 0.18618479\n",
      "Iteration 113, loss = 0.18617728\n",
      "Iteration 114, loss = 0.18617068\n",
      "Iteration 115, loss = 0.18617344\n",
      "Training loss did not improve more than tol=0.000100 for two consecutive epochs. Learning rate too small. Stopping.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "MLPClassifier(activation='relu', alpha=0.0001, batch_size='auto', beta_1=0.9,\n",
       "       beta_2=0.999, early_stopping=False, epsilon=1e-08,\n",
       "       hidden_layer_sizes=100, learning_rate='adaptive',\n",
       "       learning_rate_init=0.001, max_iter=200, momentum=0.9,\n",
       "       nesterovs_momentum=True, power_t=0.5, random_state=None,\n",
       "       shuffle=True, solver='sgd', tol=0.0001, validation_fraction=0.1,\n",
       "       verbose=True, warm_start=False)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlp_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T01:05:19.022564Z",
     "start_time": "2017-12-05T01:05:18.977124Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp_predictions = mlp_clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T01:06:12.758718Z",
     "start_time": "2017-12-05T01:06:12.711021Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.926142857143\n"
     ]
    }
   ],
   "source": [
    "mlp_score = mlp_clf.score(X_test, y_test)\n",
    "print(mlp_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T01:06:51.188474Z",
     "start_time": "2017-12-05T01:06:51.179740Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        0.0       0.96      0.97      0.96       660\n",
      "        1.0       0.97      0.96      0.96       779\n",
      "        2.0       0.93      0.92      0.92       688\n",
      "        3.0       0.90      0.89      0.90       733\n",
      "        4.0       0.91      0.94      0.92       640\n",
      "        5.0       0.90      0.89      0.90       651\n",
      "        6.0       0.96      0.96      0.96       738\n",
      "        7.0       0.94      0.93      0.93       724\n",
      "        8.0       0.91      0.91      0.91       672\n",
      "        9.0       0.88      0.88      0.88       715\n",
      "\n",
      "avg / total       0.93      0.93      0.93      7000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, mlp_predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-05T01:07:17.232322Z",
     "start_time": "2017-12-05T01:07:17.220017Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix:\n",
      "[[639   0   1   2   2   1   7   0   6   2]\n",
      " [  0 749   5   8   0   3   0   0  11   3]\n",
      " [  2   6 635  11   7   0   8  10   8   1]\n",
      " [  1   2  19 652   0  27   3   4  16   9]\n",
      " [  1   1   2   0 600   7   3   4   2  20]\n",
      " [  5   4   6  15   5 581   8   2  13  12]\n",
      " [  5   2   5   0   8   2 711   0   2   3]\n",
      " [  3   2   9   5   5   2   1 672   0  25]\n",
      " [  7   7   4  14   2  14   1   1 614   8]\n",
      " [  2   2   0  15  29  10   0  25   2 630]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Confusion matrix:\\n%s\" % confusion_matrix(y_test, mlp_predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the following implementation is sourced from the TensorFlow project (https://www.tensorflow.org/get_started/mnist/beginners). It uses the APIs and libraries available in TensorFlow to construct a Deep Neural Network in a similar fashion to the Multilayer Perceptron."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-03T21:48:42.593191Z",
     "start_time": "2017-12-03T21:48:41.144750Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-03T21:48:45.862877Z",
     "start_time": "2017-12-03T21:48:45.835730Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#setup and initialize weight and bias to 0 values\n",
    "W = tf.Variable(tf.zeros([784,10]))\n",
    "b = tf.Variable(tf.zeros([10]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-03T21:49:01.412978Z",
     "start_time": "2017-12-03T21:49:01.401911Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#setup and initialize feature vector - x_tf, yhat or true distribution y_, and y or predicted distribution - y_tf \n",
    "x_tf = tf.placeholder(tf.float32, [None, 784])\n",
    "y_tf = tf.nn.softmax(tf.matmul(x_tf, W) + b)\n",
    "y_ = tf.placeholder(tf.float32, [None, 10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-03T21:49:05.552668Z",
     "start_time": "2017-12-03T21:49:05.539042Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_tf), reduction_indices=[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-03T21:49:06.474946Z",
     "start_time": "2017-12-03T21:49:06.385112Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-03T21:49:10.512336Z",
     "start_time": "2017-12-03T21:49:10.508894Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Execution\n",
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-03T21:49:14.284835Z",
     "start_time": "2017-12-03T21:49:14.250784Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.global_variables_initializer().run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-03T21:49:24.027831Z",
     "start_time": "2017-12-03T21:49:22.244169Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-03T21:49:27.536017Z",
     "start_time": "2017-12-03T21:49:26.603893Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for _ in range(1000):\n",
    "  batch_xs, batch_ys = mnist.train.next_batch(100)\n",
    "  sess.run(train_step, feed_dict={x_tf: batch_xs, y_: batch_ys})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-03T21:49:29.106164Z",
     "start_time": "2017-12-03T21:49:29.095700Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "correct_prediction = tf.equal(tf.argmax(y_tf,1), tf.argmax(y_,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-03T21:49:33.336616Z",
     "start_time": "2017-12-03T21:49:33.329306Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-12-03T21:49:34.698554Z",
     "start_time": "2017-12-03T21:49:34.659512Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9207\n"
     ]
    }
   ],
   "source": [
    "print(sess.run(accuracy, feed_dict={x_tf: mnist.test.images, y_: mnist.test.labels}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Citations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This experiment and notebook leverage libraries and packages made available through the scikit-learn project. Necessary citation is made here: Scikit-learn: Machine Learning in Python, Pedregosa et al., JMLR 12, pp. 2825-2830, 2011.<br> <br>\n",
    "Additional reference and citation is also made for TensorFlow and using its neural network code and libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}