Part09_Hither_to_Train_Thither_to_Test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 9: Hither to Train, Thither to Test\n",
    "OK, now we know a bit about perceptrons. We'll return to that subject again. But now let's do a couple of things with our 48 colors from lesson 7:\n",
    "\n",
    "* We're going to wiggle some more - perturb the color data - in order to generate even more data.\n",
    "* But now we're going to randomly split the data into two parts, 80% for training and 20% for testing.\n",
    "\n",
    "Why split for training and testing?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Repeating the Same Things Too Much Makes Jack a Dull Network\n",
    "It's possible to *overtrain* your network, to provide it with so much similar data and so many epoch repetitions that it only learns the data you give it, so that if you give it something new it can't deal with it very well and its guesses come out wrong. A network that can make good predictions is a *generalized* network. \n",
    "\n",
    "So if you have a lot of data - enough not to worry about *undertraining* by not providing enough examples - you can keep aside some of it as a test for after all your epochs are done, to see, if you give it data it was not trained on, it can still produce similar loss and provide accuracy.\n",
    "\n",
    "This testing is called *scoring* the network against test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## But why weren't we splitting data before?\n",
    "In the beginning we had no colors, then 3, then 11, then 24, then 48. Splitting with so little data does not do much good, as you'll keep important information about some colors out of training, and the network won't know what they are since it never saw them as an example. When we started perturbing - wiggling - the original colors to multiply how much data we have, splitting started to become possible.\n",
    "\n",
    "What we're going to do below is increase the amount of data even more, by adding more wiggle points. And then we're going to keep 20% of the data aside for testing the network to see whether it's becoming too focused - not generalized enough - and can't figure out what to do with the test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slightly Different Network\n",
    "There are a bunch of differences in the network code below, based on what we learned in lesson 7:\n",
    "\n",
    "* We use only 4 perceptrons per color, used to be 8.\n",
    "* We use a batch size of 8 to avoid waiting too long for training. As we increase the data we can usually increase the batch size without making things much worse.\n",
    "* We split the data into training and test, then train on the training data.\n",
    "* Then after training we score the network against the test data which the network hasn't seen yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.layers import Activation, Dense, Dropout\n",
    "from keras.models import Sequential\n",
    "import keras.optimizers, keras.utils, numpy\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelBinarizer\n",
    "\n",
    "def train(rgbValues, colorNames, epochs = 3, perceptronsPerColorName = 4, batchSize = 8):\n",
    "    \"\"\"\n",
    "    Trains a neural network to understand how to map color names to RGB triples.\n",
    "    The provided lists of RGB triples must be floating point triples with each\n",
    "    value in the range [0.0, 1.0], and the number of color names must be the same length.\n",
    "    Different names are allowed to map to the same RGB triple.\n",
    "    Returns a trained model that can be used for recognize().\n",
    "    \"\"\"\n",
    "\n",
    "    # Convert the Python map RGB values into a numpy array needed for training.\n",
    "    rgbNumpyArray = numpy.array(rgbValues, numpy.float64)\n",
    "    \n",
    "    # Convert the color labels into a one-hot feature array.\n",
    "    # Text labels for each array position are in the classes_ list on the binarizer.\n",
    "    labelBinarizer = LabelBinarizer()\n",
    "    oneHotLabels = labelBinarizer.fit_transform(colorNames)\n",
    "    numColors = len(labelBinarizer.classes_)\n",
    "    colorLabels = labelBinarizer.classes_\n",
    "    \n",
    "    # Partition the data into training and testing splits using 80% of\n",
    "    # the data for training and the remaining 20% for testing.\n",
    "    (trainingColors, testColors, trainingOneHotLabels, testOneHotLabels) = train_test_split(\n",
    "        rgbNumpyArray, oneHotLabels, test_size=0.2)\n",
    "\n",
    "    # Hyperparameters to define the network shape.\n",
    "    numFullyConnectedPerceptrons = numColors * perceptronsPerColorName\n",
    "    \n",
    "    model = Sequential([\n",
    "        # Layer 1: Fully connected layer with ReLU activation.\n",
    "        Dense(numFullyConnectedPerceptrons, activation='relu', kernel_initializer='TruncatedNormal', input_shape=(3,)),\n",
    "\n",
    "        # Outputs: SoftMax activation to get probabilities by color.\n",
    "        Dense(numColors, activation='softmax')\n",
    "    ])\n",
    "\n",
    "    print(model.summary())\n",
    "\n",
    "    # Compile for categorization.\n",
    "    model.compile(\n",
    "        optimizer = keras.optimizers.SGD(lr = 0.01, momentum = 0.9, decay = 1e-6, nesterov = True),\n",
    "        loss = 'categorical_crossentropy',\n",
    "        metrics = [ 'accuracy' ])\n",
    "\n",
    "    history = model.fit(trainingColors, trainingOneHotLabels, epochs=epochs, batch_size=batchSize)\n",
    "\n",
    "    print(\"\")\n",
    "    print(\"Scoring result against test data after training with training data:\")\n",
    "    score = model.evaluate(testColors, testOneHotLabels, batch_size=batchSize)\n",
    "    \n",
    "    print(\"\")\n",
    "    print(\"Score: loss=%1.4f, accuracy=%1.4f\" % (score[0], score[1]))\n",
    "    return (model, colorLabels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's our createMoreTrainingData() function, mostly the same but we've doubled the number of perturbValues by adding points in between the previous ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def createMoreTrainingData(colorNameToRGBMap):\n",
    "    # The incoming color map is not typically going to be oversubscribed with e.g.\n",
    "    # extra 'red' samples pointing to slightly different colors. We generate a\n",
    "    # training dataset by perturbing each color by a small amount positive and\n",
    "    # negative. We do this for each color individually, by pairs, and for all three\n",
    "    # at once, for each positive and negative value, resulting in dataset that is\n",
    "    # many times as large.\n",
    "    perturbValues = [ 0.0, 0.005, 0.01, 0.015, 0.02, 0.025, 0.03 ]\n",
    "    rgbValues = []\n",
    "    labels = []\n",
    "    for colorName, rgb in colorNameToRGBMap.items():\n",
    "        reds = []\n",
    "        greens = []\n",
    "        blues = []\n",
    "        for perturb in perturbValues:\n",
    "            if rgb[0] + perturb <= 1.0:\n",
    "                reds.append(rgb[0] + perturb)\n",
    "            if perturb != 0.0 and rgb[0] - perturb >= 0.0:\n",
    "                reds.append(rgb[0] - perturb)\n",
    "            if rgb[1] + perturb <= 1.0:\n",
    "                greens.append(rgb[1] + perturb)\n",
    "            if perturb != 0.0 and rgb[1] - perturb >= 0.0:\n",
    "                greens.append(rgb[1] - perturb)\n",
    "            if rgb[2] + perturb <= 1.0:\n",
    "                blues.append(rgb[2] + perturb)\n",
    "            if perturb != 0.0 and rgb[2] - perturb >= 0.0:\n",
    "                blues.append(rgb[2] - perturb)\n",
    "        for red in reds:\n",
    "            for green in greens:\n",
    "                for blue in blues:\n",
    "                    rgbValues.append((red, green, blue))\n",
    "                    labels.append(colorName)\n",
    "    return (rgbValues, labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And our previous 48 crayon colors, and let's try training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From c:\\users\\erik\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 192)               768       \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 48)                9264      \n",
      "=================================================================\n",
      "Total params: 10,032\n",
      "Trainable params: 10,032\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "WARNING:tensorflow:From c:\\users\\erik\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\tensorflow\\python\\ops\\math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Epoch 1/3\n",
      "74224/74224 [==============================] - 14s 185us/step - loss: 1.0514 - acc: 0.7338\n",
      "Epoch 2/3\n",
      "74224/74224 [==============================] - 12s 166us/step - loss: 0.2417 - acc: 0.9323\n",
      "Epoch 3/3\n",
      "74224/74224 [==============================] - 13s 169us/step - loss: 0.1757 - acc: 0.9456\n",
      "\n",
      "Scoring result against test data after training with training data:\n",
      "18557/18557 [==============================] - 2s 86us/step\n",
      "\n",
      "Score: loss=0.1417, accuracy=0.9565\n"
     ]
    }
   ],
   "source": [
    "def rgbToFloat(r, g, b):  # r, g, b in 0-255 range\n",
    "    return (float(r) / 255.0, float(g) / 255.0, float(b) / 255.0)\n",
    "\n",
    "# http://www.jennyscrayoncollection.com/2017/10/complete-list-of-current-crayola-crayon.html\n",
    "colorMap = {\n",
    "    # 8-crayon box colors\n",
    "    'red': rgbToFloat(238, 32, 77),\n",
    "    'yellow': rgbToFloat(252, 232, 131),\n",
    "    'blue': rgbToFloat(31, 117, 254),\n",
    "    'brown': rgbToFloat(180, 103, 77),\n",
    "    'orange': rgbToFloat(255, 117, 56),\n",
    "    'green': rgbToFloat(28, 172, 20),\n",
    "    'violet': rgbToFloat(146, 110, 174),\n",
    "    'black': rgbToFloat(35, 35, 35),\n",
    "\n",
    "    # Additional for 16-count box\n",
    "    'red-violet': rgbToFloat(192, 68, 143),\n",
    "    'red-orange': rgbToFloat(255, 117, 56),\n",
    "    'yellow-green': rgbToFloat(197, 227, 132),\n",
    "    'blue-violet': rgbToFloat(115, 102, 189),\n",
    "    'carnation-pink': rgbToFloat(255, 170, 204),\n",
    "    'yellow-orange': rgbToFloat(255, 182, 83),\n",
    "    'blue-green': rgbToFloat(25, 158, 189),\n",
    "    'white': rgbToFloat(237, 237, 237),\n",
    "\n",
    "    # Additional for 24-count box\n",
    "    'violet-red': rgbToFloat(247, 83 ,148),\n",
    "    'apricot': rgbToFloat(253, 217, 181),\n",
    "    'cerulean': rgbToFloat(29, 172, 214),\n",
    "    'indigo': rgbToFloat(93, 118, 203),\n",
    "    'scarlet': rgbToFloat(242, 40, 71),\n",
    "    'green-yellow': rgbToFloat(240, 232, 145),\n",
    "    'bluetiful': rgbToFloat(46, 80, 144),\n",
    "    'gray': rgbToFloat(149, 145, 140),\n",
    "    \n",
    "    # Additional for 32-count box\n",
    "    'chestnut': rgbToFloat(188, 93, 88),\n",
    "    'peach': rgbToFloat(255, 207, 171),\n",
    "    'sky-blue': rgbToFloat(128, 215, 235),\n",
    "    'cadet-blue': rgbToFloat(176, 183, 198),\n",
    "    'melon': rgbToFloat(253, 188, 180),\n",
    "    'tan': rgbToFloat(250, 167, 108),\n",
    "    'wisteria': rgbToFloat(205, 164, 222),\n",
    "    'timberwolf': rgbToFloat(219, 215, 210),\n",
    "\n",
    "    # Additional for 48-count box\n",
    "    'lavender': rgbToFloat(252, 180, 213),\n",
    "    'burnt-sienna': rgbToFloat(234, 126, 93),\n",
    "    'olive-green': rgbToFloat(186, 184, 108),\n",
    "    'purple-mountains-majesty': rgbToFloat(157, 129, 186),\n",
    "    'salmon': rgbToFloat(255, 155, 170),\n",
    "    'macaroni-and-cheese': rgbToFloat(255, 189, 136),\n",
    "    'granny-smith-apple': rgbToFloat(168, 228, 160),\n",
    "    'sepia': rgbToFloat(165, 105, 79),\n",
    "    'mauvelous': rgbToFloat(239, 152, 170),\n",
    "    'goldenrod': rgbToFloat(255, 217, 117),\n",
    "    'sea-green': rgbToFloat(159, 226, 191),\n",
    "    'raw-sienna': rgbToFloat(214, 138, 89),\n",
    "    'mahogany': rgbToFloat(205, 74, 74),\n",
    "    'spring-green': rgbToFloat(236, 234, 190),\n",
    "    'cornflower': rgbToFloat(154, 206, 235),\n",
    "    'tumbleweed': rgbToFloat(222, 170, 136),\n",
    "}\n",
    "\n",
    "(rgbValues, colorNames) = createMoreTrainingData(colorMap)\n",
    "(colorModel, colorLabels) = train(rgbValues, colorNames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad: We quickly got our loss down to 0.17 in only 3 epochs, but the larger batch size kept it from taking a really long time.\n",
    "\n",
    "But let's examine our new addition, the test data scoring result. From my machine:\n",
    "\n",
    " `Score: loss=0.1681, accuracy=0.9464`\n",
    "\n",
    "Note that we trained with 74,000 data points, but we kept aside an additional 18,000 data points as test data the network was not allowed to train with. And when we ask the network to predict with the test data, the loss we get - 0.168 on my machine - is pretty close to the 0.172 loss I got on training.\n",
    "\n",
    "This is good news! It means our network is well generalized: not overtrained, not too focused to deal with making predictions on new data.\n",
    "\n",
    "Try it out to make sure it still seems like a good result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c814e76c6dd146c090f2d0a8851e1e69",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(FloatSlider(value=0.5, description='r', max=1.0, step=0.01), FloatSlider(value=0.5, desc…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ipywidgets import interact\n",
    "from IPython.core.display import display, HTML\n",
    "\n",
    "def displayColor(r, g, b):\n",
    "    rInt = min(255, max(0, int(r * 255.0)))\n",
    "    gInt = min(255, max(0, int(g * 255.0)))\n",
    "    bInt = min(255, max(0, int(b * 255.0)))\n",
    "    hexColor = \"#%02X%02X%02X\" % (rInt, gInt, bInt)\n",
    "    display(HTML('<div style=\"width: 50%; height: 50px; background: ' + hexColor + ';\"></div>'))\n",
    "\n",
    "numPredictionsToShow = 5\n",
    "@interact(r = (0.0, 1.0, 0.01), g = (0.0, 1.0, 0.01), b = (0.0, 1.0, 0.01))\n",
    "def getTopPredictionsFromModel(r, g, b):\n",
    "    testColor = numpy.array([ (r, g, b) ])\n",
    "    predictions = colorModel.predict(testColor, verbose=0)  # Predictions shape (1, numColors)\n",
    "    predictions *= 100.0\n",
    "    predColorTuples = []\n",
    "    for i in range(0, len(colorLabels)):\n",
    "        predColorTuples.append((predictions[0][i], colorLabels[i]))\n",
    "    predAndNames = numpy.array(predColorTuples, dtype=[('pred', float), ('colorName', 'U50')])\n",
    "    sorted = numpy.sort(predAndNames, order=['pred', 'colorName'])\n",
    "    sorted = sorted[::-1]  # reverse rows to get highest on top\n",
    "    for i in range(0, numPredictionsToShow):\n",
    "        print(\"%2.1f\" % sorted[i][0] + \"%\", sorted[i][1])\n",
    "    displayColor(r, g, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In my opinion the extra perturbation data made quite a bit of difference. It guesses over 70% for gray at (0.5, 0.5, 0.5), better than before.\n",
    "\n",
    "Here's the hyperparameter slider version so you can try out different epochs, batch sizes, and perceptrons:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5c188004fb094f53b76b286b7ce635fd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=4, description='epochs', max=10, min=1), IntSlider(value=3, description=…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "@interact(epochs = (1, 10), perceptronsPerColorName = (1, 12), batchSize = (1, 50))\n",
    "def trainModel(epochs=4, perceptronsPerColorName=3, batchSize=16):\n",
    "    global colorModel\n",
    "    global colorLabels\n",
    "    (colorModel, colorLabels) = train(rgbValues, colorNames, epochs=epochs, perceptronsPerColorName=perceptronsPerColorName, batchSize=batchSize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fca05e42ef0d4c56be6d5036cffa139f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(FloatSlider(value=0.5, description='r', max=1.0, step=0.01), FloatSlider(value=0.5, desc…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function __main__.getTopPredictionsFromModel(r, g, b)>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interact(getTopPredictionsFromModel, r = (0.0, 1.0, 0.01), g = (0.0, 1.0, 0.01), b = (0.0, 1.0, 0.01))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Coming up...\n",
    "We'll begin understanding why neurons and perceptrons by themselves are not enough, it's the connections that matter too. And we'll begin to learn how training works to create weights and biases that let ask the network for new predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}