Tutorials/CNTK_101_LogisticRegression.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# CNTK 101: Logistic Regression and ML Primer\n",
    "\n",
    "This tutorial is targeted to individuals who are new to CNTK and to machine learning. In this tutorial, you will train a simple yet powerful machine learning model that is widely used in industry for a variety of applications. The model trained below scales to massive data sets in the most expeditious manner by harnessing computational scalability leveraging the computational resources you may have (one or more CPU cores, one or more GPUs, a cluster of CPUs or a cluster of GPUs), transparently via the CNTK library.\n",
    "\n",
    "The following notebook users Python APIs. If you are looking for this example in BrainScript, please look [here](https://github.com/Microsoft/CNTK/tree/v2.0/Tutorials/HelloWorld-LogisticRegression). \n",
    "\n",
    "## Introduction\n",
    "\n",
    "**Problem**:\n",
    "A cancer hospital has provided data and wants us to determine if a patient has a fatal [malignant][] cancer vs. a benign growth. This is known as a classification problem. To help classify each patient, we are given their age and the size of the tumor. Intuitively, one can imagine that younger patients and/or patient with small tumor size are less likely to have malignant cancer. The data set simulates this application where the each observation is a patient represented as a dot (in the plot below) where red color indicates malignant and blue indicates benign disease. Note: This is a toy example for learning, in real life there are large number of features from different tests/examination sources and doctors'  experience that play into the diagnosis/treatment decision for a patient.\n",
    "[malignant]: https://en.wikipedia.org/wiki/Malignancy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://www.cntk.ai/jup/cancer_data_plot.jpg\" width=\"400\" height=\"400\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure 1\n",
    "Image(url=\"https://www.cntk.ai/jup/cancer_data_plot.jpg\", width=400, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Goal**:\n",
    "Our goal is to learn a classifier that automatically can label any patient into either benign or malignant category given two features (age and tumor size). In this tutorial, we will create a linear classifier that is a fundamental building-block in deep networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://www.cntk.ai/jup/cancer_classify_plot.jpg\" width=\"400\" height=\"400\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure 2\n",
    "Image(url= \"https://www.cntk.ai/jup/cancer_classify_plot.jpg\", width=400, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the figure above, the green line represents the learnt model from the data and separates the blue dots from the red dots. In this tutorial, we will walk you through the steps to learn the green line. Note: this classifier does make mistakes where couple of blue dots are on the wrong side of the green line. However, there are ways to fix this and we will look into some of the techniques in later tutorials. \n",
    "\n",
    "**Approach**: \n",
    "Any learning algorithm has typically five stages. These are Data reading, Data preprocessing, Creating a model, Learning the model parameters, and Evaluating (a.k.a. testing/prediction) the model. \n",
    "\n",
    ">1. Data reading: We generate simulated data sets with each sample having two features (plotted below) indicative of the age and tumor size.\n",
    ">2. Data preprocessing: Often the individual features such as size or age needs to be scaled. Typically one would scale the data between 0 and 1. To keep things simple, we are not doing any scaling in this tutorial (for details look here: [feature scaling][]).\n",
    ">3. Model creation: We introduce a basic linear model in this tutorial. \n",
    ">4. Learning the model: This is also known as training. While fitting a linear model can be done in a variety of ways ([linear regression][]), in CNTK we use Stochastic Gradient Descent a.k.a. [SGD][].\n",
    ">5. Evaluation: This is also known as testing where one takes data sets with known labels (a.k.a. ground-truth) that was not ever used for training. This allows us to assess how a model would perform in real world (previously unseen) observations.\n",
    "\n",
    "## Logistic Regression\n",
    "[Logistic regression][] is fundamental machine learning technique that uses a linear weighted combination of features and generates the probability of predicting different classes. In our case the classifier will generate a  probability in [0,1] which can then be compared with a threshold (such as 0.5) to produce a binary label (0 or 1). However, the method shown can be extended to multiple classes easily. \n",
    "[feature scaling]: https://en.wikipedia.org/wiki/Feature_scaling\n",
    "[SGD]: https://en.wikipedia.org/wiki/Stochastic_gradient_descent\n",
    "[linear regression]: https://en.wikipedia.org/wiki/Linear_regression\n",
    "[logistic regression]: https://en.wikipedia.org/wiki/Logistic_regression\n",
    "[softmax]: https://en.wikipedia.org/wiki/Multinomial_logistic_regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://www.cntk.ai/jup/logistic_neuron.jpg\" width=\"300\" height=\"200\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure 3\n",
    "Image(url= \"https://www.cntk.ai/jup/logistic_neuron.jpg\", width=300, height=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the figure above, contributions from different input features are linearly weighted and aggregated. The resulting sum is mapped to a 0-1 range via a [sigmoid][] function. For classifiers with more than two output labels, one can use a [softmax][] function.\n",
    "[sigmoid]: https://en.wikipedia.org/wiki/Sigmoid_function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the relevant components\n",
    "from __future__ import print_function\n",
    "import numpy as np\n",
    "import sys\n",
    "import os\n",
    "from cntk import *\n",
    "\n",
    "import cntk as C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the block below, we check if we are running this notebook in the CNTK internal test machines by looking for environment variables defined there. We then select the right target device (GPU vs CPU) to test this notebook. In other cases, we use CNTK's default policy to use the best available device (GPU, if available, else CPU)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Select the right target device when this notebook is being tested:\n",
    "if 'TEST_DEVICE' in os.environ:\n",
    "    if os.environ['TEST_DEVICE'] == 'cpu':\n",
    "        C.device.try_set_default_device(C.device.cpu())\n",
    "    else:\n",
    "        C.device.try_set_default_device(C.device.gpu(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Generation\n",
    "Let us generate some synthetic data emulating the cancer example using `numpy` library. We have two features (represented in two-dimensions)  each either being to one of the two classes (benign:blue dot or malignant:red dot). \n",
    "\n",
    "In our example, each observation in the training data has a label (blue or red) corresponding to each observation (set of features - age and size). In this example, we have two classes represented by labels 0 or 1, thus a  binary classification task. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define the network\n",
    "input_dim = 2\n",
    "num_output_classes = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input and Labels\n",
    "\n",
    "In this tutorial we are generating synthetic data using `numpy` library. In real world problems, one would use a [reader][], that would read feature values (`features`: *age* and *tumor size*) corresponding to each obeservation (patient). The simulated *age* variable is scaled down to have similar range as the other variable. This is a key aspect of data pre-processing that we will learn more in later tutorials. Note, each observation can reside in a higher dimension space (when more features are available) and will be represented as a [tensor][] in CNTK. More advanced tutorials shall introduce the handling of high dimensional data.\n",
    "\n",
    "[reader]: https://github.com/Microsoft/CNTK/search?p=1&q=reader&type=Wikis&utf8=%E2%9C%93\n",
    "[tensor]: https://en.wikipedia.org/wiki/Tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ensure we always get the same amount of randomness\n",
    "np.random.seed(0)\n",
    "\n",
    "# Helper function to generate a random data sample\n",
    "def generate_random_data_sample(sample_size, feature_dim, num_classes):\n",
    "    # Create synthetic data using NumPy. \n",
    "    Y = np.random.randint(size=(sample_size, 1), low=0, high=num_classes)\n",
    "\n",
    "    # Make sure that the data is separable \n",
    "    X = (np.random.randn(sample_size, feature_dim)+3) * (Y+1)\n",
    "    \n",
    "    # Specify the data type to match the input variable used later in the tutorial \n",
    "    # (default type is double)\n",
    "    X = X.astype(np.float32)    \n",
    "    \n",
    "    # converting class 0 into the vector \"1 0 0\", \n",
    "    # class 1 into vector \"0 1 0\", ...\n",
    "    class_ind = [Y==class_number for class_number in range(num_classes)]\n",
    "    Y = np.asarray(np.hstack(class_ind), dtype=np.float32)\n",
    "    return X, Y   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the input variables denoting the features and the label data. Note: the input \n",
    "# does not need additional info on number of observations (Samples) since CNTK creates only \n",
    "# the network topology first \n",
    "mysamplesize = 32\n",
    "features, labels = generate_random_data_sample(mysamplesize, input_dim, num_output_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us visualize the input data.\n",
    "\n",
    "**Note**: If the import of `matplotlib.pyplot` fails, please run `conda install matplotlib` which will fix the `pyplot` version dependencies. If you are on a python environment different from Anaconda, then use `pip install`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEPCAYAAACgFqixAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XecFdX9//HXZzt3YRUFBClSFZVuRUTWhgUFsSJGVMAW\nS9T4i5omSdRoErAgaoydaDQgitgAAysKEVAhgAVFDEVhBSnCFtjy+f1xB7Pw3YW75TJ7d9/Px2Mf\n3pmdO/NG8X7uOWfmHHN3RESkfksKO4CIiIRPxUBERFQMRERExUBERFAxEBERVAxERIQ4FwMze9LM\ncs1sUZl9jc1smpktNbOpZrZPPDOIiMiexbtl8DRw2i77bgfecfdDgBnAHXHOICIie2DxfujMzA4C\nprh7t2D7c6Cfu+eaWXMgx907xzWEiIjsVhhjBs3cPRfA3dcCzULIICIiZdSGAWTNhyEiErKUEK6Z\na2YHlOkm+q6iA81MhUJEpArc3Spz/N5oGVjws8NrwOXB68uAybt7s7sn7M+dd94Zeob6mj+Rsyt/\n+D+Jnr8q4n1r6QvAHOBgM1tpZlcA9wKnmtlS4ORgW0REQhTXbiJ3H1rBr06J53VFRKRyasMAcp2V\nnZ0ddoRqSeT8iZwdlD9siZ6/KuL+nEF1mJnX5nwiIrWRmeG1cABZRERqORUDERFRMRARERUDERFB\nxUBERFAxEBERVAxERAQVAxERQcVARERQMRAREVQMREQEFQMREUHFQEREUDEQERHCWQNZRKRaZs+e\nzYIFC+jQoQOnn346ZpWarVnKofUMRCShjL73Xh76wx84s7SU95KT6XPuuTz27LMqCGVUZT0DFQMR\nSRibN2+mVbNmfL59Oy2BPOCwSIRX33+fnj17hh2v1tDiNiJSp23YsIHGqam0DLYzgY6pqaxbty7M\nWHWCioGIJIzWrVuTlpXFI2YUAW8Ai0tK6NGjR9jREp6KgYgkjJSUFN6YOZMnO3Uiw4wbmzdn0ltv\n0axZs7CjJTyNGYhIQiotLSUpSd9ny6MxAxGpN1QIapb+bYqIiIqBiIioGIiICCoGIiKCioGIiKBi\nICIiqBiIiAgqBiIigoqBiIigYiAiIqgYiIgIKgYiIkKIxcDMbjazJWa2yMyeN7O0sLKIiNR3oRQD\nMzsQuAHo5e7dgBRgSBhZREQk+iEclmQg08xKgQjwbYhZRETqtVBaBu7+LTAaWAl8A2xy93fCyCIi\nIiG1DMxsX2AQcBCwGZhoZkPd/YVdjx01atSPr7Ozs8nOzt5LKUVEEkNOTg45OTnVOkcoy16a2fnA\nae5+ZbB9KXCMu1+/y3Fa9lJEpJISadnLlcCxZpZhZgacDHwWUhYRkXovrDGDecBEYAHwH8CAx8PI\nIiIiIXUTxUrdRCIilZdI3UQiIlKLqBiIiIiKgYiIqBiIiAgqBiIigoqBiIigYiAiIqgYiIgIKgYi\nIoKKgYiIoGIgIiKoGIiICCoGIiKCioGIiKBiICIiqBiIiAiQEnYAqZsKCwsZP348ubm59OvXj759\n+4YdSUR2I+ZiYGZZQAugwN1Xxi+SJLpt27ZxSu/eNPriC7oXFnJxRga/f/BBho8cGXY0EanAbpe9\nNLNGwLXAUKAhsB7IAPYH3gcecff34hZOy14mpJdeeolHR4xgZl4eBnwCnJCZyfotWzCr1Ep8IlIF\nVVn2ck8tg1eA54GT3f37MhdKAo4CLjWzTu7+VKXTSp21ceNGOpaWsuNvYkdgS2EhpaWlJCcnhxlN\nRCqw25ZB2NQySExLly7l+F69eD4/nx7Ab9PSWN27N6/n5IQdLaEVFxfzlz/+kfemTqXFQQcx6r77\naNWqVdixpBaqSssg5mJgZocBbSnTmnD31ypzscpSMUhc06dP58bhw/lu40ZO7NuXx59/nv322y/s\nWAntmssv54sJE7gpP5/5ycmM339/Fnz+OY0bNw47mtQycSsGZvY34EjgU6A02O3uPqzSKStBxUAk\nqri4mMyMDL4rKWGfYN/Ahg0Z8te/MnTo0FCzSe0TjzGDHY4HDtMns8jetWDBApYtW0bnzp3L/b0G\n5KWmxFoM5gIHA0vjmEVEyvjDr3/NX++/n6NTUphdVMRxRx3FOYsW/dhNtDgS4dnTTw87ptQRsXYT\n9QWmAN8A2wAj2k3UK67h1E0k9dTSpUvp17MniwsKaAp8BfRKT+eWW29l/rvv0uKgg7jz3ns1gCzl\nimc30VPAcGAx/xszEJE4Wb16NZ3T0mhaUABAB6BJaioXDxvGnXfdFW44qZNiLQbfu/ukuCYRkR8d\nfvjhfFJczBzgOGAyUJCSQps2bUJOJnVVrMXgQzN7jmhX0bYdO+N9a6lIfdW8eXOemzCBsy+4AIqL\naZCZyStvvklGRkbY0aSOinXMYHw5u3VrqUicFRcXs2HDBpo0aUJSkiYZltjE9aGzMKgYSCLZsmUL\nv7j+ej547z3atGvHXx57jE6dOoUdS+qhqhSDmL5qmNmTZrZvme3GwYNoIhK46KyzyHvpJR7/+mtO\nyMnhpN692bBhQ9ixqmzFihXcfuut3Hj11bz77rthx5E4i7WbaIG799xl38e6tVQkatOmTbRu1oyN\nRUU/DsSdnpXFtc89x6BBg0LNVhUrVqzg2O7d+cmWLRxQWsroBg145PnnGTx4cNjRJAZxaxkASWa2\n4yl4zKwxkFqZC4nUZampqZS4szXYdmCTO+np6WHGqrLHH3mEoVu38ufSUm4Fni4o4I933BF2LImj\nWIvBA8C/zexOM7sTmA2Mjl8skZpTWlrK119/zdq1a+N2jczMTK4cPpzTIhEeA36Sno63acOJJ54Y\nt2vGU0FeHk1LSn7cbgoUBM88SN0UUzFw96eBIcDm4GeIuz9TnQub2T5mNsHMPjOzT8zsmOqcT6Q8\n69at49iuXenbpQuHtm3LVZdeSmlpfJ6bvP/RRxk5ZgwfDR3KIbfdxr8++CBhWwbnXXwxD0QiTAHm\nAddFIlx42WVhx5I4Cu1uIjN7BnjX3Z82sxQg4u4/7HKMxgykWi4eOJAWb7/N6KIi8oD+kQjDH3yQ\nkVqCc49ef/117rn9dvLz87lg2DDu+O1vdXtrgkiYW0uD9ZQXuHuHPRynYiDVcmirVkz45hu6BNsP\nAl+OGMHDTzwRZiyRuIrnAHJNawesN7OnzexjM3vczBqElEXqsA4dO/Jm8G22GJjWoAEdDjss3FAi\ntVBYLYMjgA+A3u7+oZk9AGx29zt3OU4tA6mW5cuXc8pxx9G8oIANpaW069mTye+8Q1paWtjRROIm\nbrOWmtkg4F7gQKLTV++Ywjqr0imjVgOr3P3DYHsicFt5B44aNerH19nZ2WRnZ1fxklIftW/fnv98\n+SUfffQRDRo04MgjjyQ5OTnsWCI1Kicnh5xqrjEe60Nny4DB7r64Wlfb+ZzvAle6+xfB7aoRd79t\nl2PUMhARqaR4rmeQW5OFIHAj8LyZpQLLgStq+PwiIhKjWFsGDxB97uRV9uIU1moZiIhUXjxbBvsT\nXeFsYJl9Dmg9AxGROkBTWIvUQu7OxIkTWbZsGd26dWPAgAFhR5IEUuMtAzP7ubuPNrMx5f3e3W+p\nzMVEZM/cnRFDh7JoyhROKizk1owM5lxzDXf/5S9hRwuVu/P4o4/yrylTaHrggdzxu9/RqlWrsGPV\nGbttGZjZOe7+qpmNKO/37v5k3JKhloHUT4sWLeKs3r35PD+fCPA90CE9nS9WrqRZs2ZhxwvNr3/x\nC94aN46f5+ezJDmZvzduzMeffUaTJk3Cjlbr1HjLwN1fDf4Z1w99EfmfjRs30jo1lUiwvT+wf2oq\nGzdurLfFwN156KGH+GzbNloClJTwVX4+r7zyCldeeWXY8eqE3U5HYWaPmtmhFfyugZkNM7OL4xNN\npH7q3r07X5sxHtgA3J+URPI++9C+ffuwo4Wq1J2yz42nu8dtBtr6aE9zEz0B3G1mS8zsH2b2UDCP\n0EyiM9s2BV6Je0qRemTfffflzZkzeeDgg2mbns6krl156913SU2tv+tJmRnDL7uMCyIRpgGjzZie\nlsbZZ58ddrQ6I9bnDLKAo4EWQAHwmbt/EudsGjMQkR8VFxdz3113MeP112lywAH8fswYDjnkkLBj\n1UoJM4V1rFQMREQqL5GmsBap1WbMmMGIIUO45rLLWLRoUdhxROJOLQORXbz++utcedFF/Do/ny3A\n6MxM/jVnDt26dQs7mkhM4t4yMLPEXNBVpBLuHzWKcfn5XAfcDtySl8djDzwQdiwA8vLyGDl0KK33\n248eHTowbdq0sCNJHRFTMTCzo81sMfBlsN3dzMbGNZlISIqKimhYZrsRsL2wMKw4O7n60kvZOmkS\nszZu5J7ly7lk8GCWLFkSdiypA2JtGTwEnEX0YUjc/T/AifEKJRKmS6+9lhuDWxhfBu5q0IBLasmD\nTa+9+Sbjtm2jHXAmMKS4mOnTp4cdS+qAWGctTXL3FWY7dUGVxCGPSOhGXn01lpTEXePGkZaWxpOj\nRnHiibXju09WJMLKbdvYP9hemZJC90aNQs0kdUOszxm8DNwHPAYcBdwA9HH3C+IaTgPIIjt59umn\n+dX11zMyP5/PMzL4tGVL5ixcSMOGDff8Zqk34vacgZk1I9pVdEqw6x3gendfX+mUlaBiUHlLlizh\n/fffp2nTpgwaNIiUlFgbf5IocnJyeGfaNPZv2pSRI0fSSC0D2UU8i0ELd1+zy74mKga1y6RJr/CT\nn1wNDCQ5+RN69tyXGTOmqCCI1DPxvLV0tZmN3+XWUt3TVsuMGHEdBQWvUVDwBFu3vs+CBZt55RVN\nHSUiexZrMVgEzAdmm1m7YF+lqo7EV2lpKT/88B3QM9iTTHFxN3Jzc8OMJSIJItZi4O7+EHAL8IaZ\nnUF0DWSpJZKSkjjyyH6kpPwWKAIWYPYKxx9/fNjRRCQBxFoMDMDdZwGnAr8GNF1gLTN58vP07Dmf\npKQGNGzYn6eeeogePXqEHUtEEkCsA8it3H11me1UoK+7z4hrOA0gV0lJSQnJyclhxxCRkNT4spdm\ndrG7/wM4d5cHznaIazGQqlEhkB1WrlzJsmXL6NChAwcddFDYcaQW21M3UePgn00r+BGRWurZp56i\nV+fO/O7cczni0EN56m9/CzuS1GKawlqkDlq3bh2HtGnDvwsLOYToDJPHZGTw6ddf07x587DjSZzF\n7TkDM/ujmWWZWYqZTTWzXDMbWrWYIhJvK1eupE1a2o93eXQC2qelsWLFimqfe/v27fz+N7/h7BNO\n4PqRI1m/Pq7PnspeEuvdRGe4+w9EZy5dAxwK3Ba3VCJSLe3bt2d1cTFzg+35wH+Li+nQoUO1z33Z\nBRcwd/RoRrz3HsnPPUe/o44iPz+/2ueVcMVaDHYMNJ8J/NPdN6DnDERqrcaNG/PMiy9yZiRCp4YN\nOS0S4akXXqBJkybVOu+mTZt4/a23mFhQwDnAA0VFNP7+e2bNmlUzwSU0sU5a85aZLSE6bfV1ZtYE\n2Ba/WCJSXWedfTYrcnNZvXo1rVq1qpGZTXeM4e3ojDai3yg1tpf4Yh5ADmYu3eDuxWbWENjH3b+J\nazgNIIvUOheedRbbZszgyoICclJSeOPAA/nw00/JzMwMO5oE4roGsrt/5+7Fweut8S4EIlI5X375\nJZdfeCED+/XjkbFj4/ZtffzLL9PthhsY17s3Wy6+mJx581QI6gDdWipSB6xevZqjunThxi1bOKS0\nlLsjEc7+2c8Ydc89YUeTEMSlZWBRLaoeSyR8kydP5pRTzuWMMy6sk4OdL7/8MgMKC7mjtJRzgX/m\n5/Poww+HHUsSyB4HkN3dzWw60GUv5BGpcRMmTOTyy28mP/9eoIBZs85n+vRXOe6448KOVmPcfac5\n5ZPQ7X5SObGOGSw0s557Pkyk9vnzn/9Kfv5Y4BJgJPn5v+ahh54IO1aNOu+885iSns6fzJgMXBiJ\ncM1Pfxp2LEkgsd5a2hOYb2ZfAXlE7yhzd+9VnYubWRLwIbDa3QdW51wiFYmOO5X93mx17lbI1q1b\n8+68edx1xx28+913DDvvPK6/6aawY0kCiXUK63IfW3T3r6p1cbObgSOArPKKgQaQpSb8r5voT0A+\nkcgdTJv2Cn369Ak7mkhc1PgU1ju4+1dmdjjQN9j1nrt/UtmAZZlZK6JPNN9NdAU1kbi44ILzSU1N\nYezYZ0lNTeGOOyaoEIjsItaWwfXAT4FXg12DgHHu/kiVL2w2gWgh2Af4uVoGIiI1I24tA+Aq4Gh3\n3xpc6B5gDlClYmBmA4Bcd19oZtns3KErIiJ7WazFwIDtZbaLqN4HeB9goJmdCTQAGpnZc+4+bNcD\nR40a9ePr7OxssrOzq3FZEZG6Jycnh5ycnGqdI9Zuol8AFwMvB7sGA/9w979U6+rRc/dD3UShWLdu\nHcuWLaNNmza0bNky7DgiUkPiNjeRu/8JuBrID36uqYlCIOGZPPk12rY9lDPO+BkdO3bjkUceDzuS\niISoMrOWZgGtKNO15O6L4pRrxzXVMoiDvLw8mjVrTX7+28DRwHIaNDiGTz6ZR7t27cKOJyLVFLcB\nZDO7k+gg8tf87yl3B06oVELZibvz2muv8cUXX9ClSxfOOOOMvXLdb7/9lqSkfYkWAoD2pKV1Zdmy\nZSoGe1BaWsq3335LJBJhv/32CzuOSI2JdQB5KNDe3bWgTQ0aPvw6JkyYzfbtJ5OW9iRXX/0eo0fH\nf5bJli1bEl3FdDbRsfwv2L59MZ06dYr7tRNZbm4uZ590Eiu//pq84mKuuuoq/jJ2LGa6GU4SX6xz\nE30CNIpnkPrms88+45//nExe3myKisaQlzebceMeZc2aNXG/diQSYcKE8WRmDiIrqxsZGccwduyf\nadu2bdyvnciuvfRSsr/4gjUFBawsKmLGM8/w0ksvhR1LpEbE2jK4G1hgZosos9ylu58bl1T1wIYN\nG0hNbQ3sWIpwf9LSmrFhwwZatIj/jOFnnHEGq1cvY/ny5bRu3ZqmTZvG/ZqJ7uMFC/hzcTEGNAYu\nzMvj43nzGDJkSNjRRKot1mLwLHA/sBgojV+c+qNLly4kJa0C/g4MxOzvRCJFdOzYca9l2HfffenV\nq1pzDdYr7Q46iOnff08Hd4qBmZEIg9W1JnVErM8ZzHf3o/ZCnl2vW6fvJlq4cCEXXHAFK1YspVOn\nrkya9ByHHHJI2LHi6quvvmLq1KlkZmZy/vnnJ9RyiZ9++in9+/bl4OJi1paW0u6II3h1+nRSU1PD\njiayk6rcTRRrMRhN9PmC19i5m0i3ltZhubm5jB8/noKCQgYPPocuXaq3vtHs2bM57bRzKCkZRHLy\nGg44YBULFrxPVlZWDSWOv40bNzJv3jwaNWrEscceS1JSzMuIi+w18SwG75Wz2909rreWqhiE55tv\nvqFHj9788EN/Skoak57+DG+/PYm+ffvu+c0V6Nr1OJYsuQm4EID09EsYNaobt99+W7nHFxUVMWvW\nLPLz8+nTp49u5RSJUTynsK76J4AkpDFjxrJx4/mUlIwBID+/Fzfd9Fs++mhmlc+5bt06oOuP29u2\ndWPNmnXlHltQUMAJJ5zB559vJimpKSkp1zJ79jt07ty5ytcXkYrF+tDZL8vb7+7xvyleQvH995sp\nKTmszJ72bNq0uVrn7N//JCZMGEVh4RPAGiKRv9K//9hyjx037hGWLNmXwsIZQBJmDzN8+M+YM2dq\ntTKISPli7fAsKfOTCpwD6DaKOuz88wcQiYwB5gNfEYncxvnnn1Wtcz766BhOOy2ZlJSmRCLHcvfd\nNzNgwIByj/3yyxUUFvZjx19R92xWrlxZreuLSMVinptopzeZZQBvu3t2jSfa+ToaMwjR448/wW9/\ney/btxdyySVDuP/+e0lJifVu5Iq5+x6f2h0/fjzXXvsAeXnvAFmkpd3AgAFbmDRpfLnHb968mYkT\nJ1JYWMiZZ56paTWkXovbAHI5F9oH+Mjd43pTvIpB/eXu3HDDrTz++GMkJaXRtWt3pk6dVO4g8vff\nf0/v7t3psnEj+5eWMjklhTdmzOCoo/b63dAitUKNFwMzS3H3YjNbwP8mqEsGWgD3uPsDVU4bSzgV\ng3pvy5YtFBYW0qRJkwpbE7+54w6+Gz2avxYVAdEnJMcffTTvzJ27F5NKfbRt2zZuufZaXn3lFRpl\nZvKHMWO44MILw44Vl7uJ5gG9gPPL7CsG1mrSOtkbGjVqRKNGu58Wa/2aNXQJCgFAF2D9uvLvUhKp\nSbdedx0rXnyR9wsKWLVpExddfjktDjyQ448/PuxolbanAWQDcPevyvysUCGQ2uSUs89mbCTCl8AG\nYFRGBifvpenApX6bMnky9xcU0I7ofP7XFBby1uuvhx2rSvbUMmhqZrdU9Et3H1PDeUQq7bzzzmPF\nV19x7O9/T+H27QwZPJh7xuivpsTfPo0a8d/163+8tfLr1FQOS9CHI/c0ZrAGeJSghbArd/9dnHLt\nuL7GDESk1poyZQojL7qIEdu2sSo1lQ/235+5ixeH/rR8PAaQP3b30Ka1VDEQkdpu/vz5vPnGGzRs\n1Igrrrgi9EIA8SkGC9y9Z7WTVZGKQdW5O2vXriU1NZUmTZqEHUdE9qKqFIM9DSCfXI08EpItW7Zw\n/PGn0b59V1q27MCQIVdQUlISdiwRqcV2WwzcfcPeCiI158Ybb+Ojjw6ksDCX7du/ZcqU//Lggw/v\nlWv/8MMPrFq1SsVHJMFoMvY66IMPPmbbtiuJPh+YSX7+MN5//6O4X/fuu/9E06Yt6dz5GDp06Mry\n5cvjfk0RqRkqBnVQx45tSU5+J9hy0tP/RefONTNXz1dffUW/fgNo2bIzZ511Ed999x0AM2fO5J57\nHmP79qXk53/LqlXDOffcYTVyTRGJvyrNTbS3aAC5alatWsUxx5xIXt6BuG+lbdt05syZTsOGDat1\n3q1bt9KhQ1fWr/8ppaVnkpr6FJ06vceHH77Lfffdxz33bKSo6MEdR5OS0pSiooLq/4FEpFLitriN\nJJbWrVuzdOkC/v3vf5OWlsZxxx1HWlpatc/74YcfUljYgtLS/wdAUdFfWLasBfvu2wT3NIqLHRgO\ndAfeoUULzRwqkihUDOqoRo0a0b9//xo9ZyQSoaTke6LTU6UAW9m+fQswFTgeeASzfjRqdDSlpQt5\n8cVXa/T6IhI/6iaSmJWWlnLiiWcxfz4UFJxKWtoLlJTkUVLyaXCEk5SUwYQJ/6BPnz4ccMABoeYV\nqa/i8ZyB1GPr16/npJMGkpGRxYEHdmLatGlMn/4qd9/dnyuvXM4NN/QjPb0EyAveMZdIpBGDBw9W\nIRBJMOomkgoNGjSU+fMPpajoGdasWcB55w3h44/f5+abbwKiTzmvW7eVl1/uTnLy4RQXz+Ef/3hm\nj6uYiUjto24iKVdxcTHp6Q0oLc0nuuw1RCLDuf/+Y7nqqqt+PM7d+eCDD1izZg29evWibdu24QQW\nkR/pbiKpMcnJyWRkNCQ//0vgMKCUpKSl7LffmTsdZ2b07t07lIwiUnM0ZiDlMjMeemgMkcippKT8\nnMzMUznssHQGDRoUdjQRiQN1E8luzZkzh/fee48DDjiAoUOH1sjzCiISXzU+hXXYVAxERCpPt5aK\niEiVhDKAbGatgOeAA4BS4G/u/lAYWaR+WLZsGTNnziQrK4tzzjmH9PT0sCOJ1CqhdBOZWXOgubsv\nNLOGwEfAIHf/fJfj1E0k1ZaTk8MFAwYwAPivGdvat2fG3Lk0aNAg7GgicZEw3UTuvtbdFwavtwKf\nAS3DyCJ1300jRvBkfj7P5OczMy+PJsuW8dRTT4UdS6RWCX3MwMzaAj2AueEmkboqd/16dizkbUDP\nggJy164NM5JIrRPqQ2dBF9FE4GdBC+H/GDVq1I+vs7Ozyc7O3ivZpO7od8IJ/H7aNB7evp0VwLOR\nCE/p75HUITk5OeTk5FTrHKHdWmpmKcDrwFvu/mAFx2jMQKpt06ZNXHruuUydNYsGqanc++c/c+31\n14cdSyRuEuo5AzN7Dljv7rfs5hgVA6mW3Nxc5s2bx3777cfRRx9NSkqKJtKTOi9hioGZ9QFmAYsB\nD35+6e5v73KcioFU2dy5czn11IGY9aKk5Gv69evKa6+9SHJyctjRROIqYYpBrFQMpDo6dOjO8uW/\nBi4AtpOZmc3jj1/P0KFDw44mElcJc2upyN6wZs0K4KRgK43CwuNZsWJFmJFEai0VA6mzunU7kuTk\ncUR7IdeQnj6JI444IuxYIrWSioHUWRMmPE27dpPIyDiA1NRO/OIXI+jfv3/YsURqJY0ZSJ1WWlrK\n2rVrycrKomHDhmHHEdkrNIAsIiIaQBYRkapRMRARERUDEREJeaI6SQwFBQXMmDGDoqIi+vXrR+PG\njcOOJCI1TAPIslubN2/mqKOyWbs2E2hIevpnzJ07k/bt24cdTUQqoAFkqXF33/0nVqzoyZYt77Fl\ny9ts2HAt1113W9ixRKSGqRjIbi1btort2/sSXRYGSkv78t//rgo3lIjUOBUD2a0TTzyWSOQJYDOw\nnYyMsfTrd2zYsUSkhqkYyG5dd901XHxxT1JSmpOS0pjjjy9g9Oi7w44lIjVMA8gSk/z8fIqLi8nK\nygo7iojsgaajEBER3U0kIiJVo2IgIiIqBiIiomIgIiKoGIiICCoGIiKCioGIiKBiICIiqBiIiAgq\nBiIigoqBiIigYiAiIqgYiIgIKgYiIoKKgYiIoGIgIiKoGIiICCoGIiKCioGIiBBiMTCz083sczP7\nwsxuCyuHiIiEVAzMLAl4GDgNOBy42Mw6h5ElnnJycsKOUC2JnD+Rs4Pyhy3R81dFWC2Do4Ev3X2F\nuxcBLwKDQsoSN4n+FyqR8ydydlD+sCV6/qoIqxi0BFaV2V4d7BMRkRBoAFlERDB33/sXNTsWGOXu\npwfbtwPu7vftctzeDyciUge4u1Xm+LCKQTKwFDgZWAPMAy5298/2ehgRESEljIu6e4mZXQ9MI9pV\n9aQKgYhIeEJpGYiISO1S6weQzexPZvaZmS00s5fNLCvsTHuSyA/UmVkrM5thZp+Y2WIzuzHsTFVh\nZklm9rEWjfSTAAAHkUlEQVSZvRZ2lsoys33MbELw9/4TMzsm7EyVYWY3m9kSM1tkZs+bWVrYmXbH\nzJ40s1wzW1RmX2Mzm2ZmS81sqpntE2bGilSQvUqfmbW+GBDtSjrc3XsAXwJ3hJxnt+rAA3XFwC3u\nfjjQG7guwfLv8DPg07BDVNGDwJvufijQHUiYLlQzOxC4Aejl7t2IdkUPCTfVHj1N9P/Xsm4H3nH3\nQ4AZ1N7PnfKyV+kzs9YXA3d/x91Lg80PgFZh5olBQj9Q5+5r3X1h8Hor0Q+ihHoGxMxaAWcCT4Sd\npbKCb3F93f1pAHcvdvcfQo5VWclAppmlABHg25Dz7Ja7vw9s3GX3IODZ4PWzwDl7NVSMyste1c/M\nWl8MdjEceCvsEHtQZx6oM7O2QA9gbrhJKu1+4P8BiTgg1g5Yb2ZPB91cj5tZg7BDxcrdvwVGAyuB\nb4BN7v5OuKmqpJm750L0CxLQLOQ8VRXzZ2atKAZmNj3oX9zxszj459lljvkVUOTuL4QYtd4ws4bA\nROBnQQshIZjZACA3aN1Y8JNIUoBewDh37wXkE+2ySAhmti/Rb9UHAQcCDc1saLipakTCfbGo7Gdm\nKLeW7srdT93d783scqLN/pP2SqDq+QZoU2a7VbAvYQTN+4nAeHefHHaeSuoDDDSzM4EGQCMze87d\nh4WcK1argVXu/mGwPRFIpJsQTgGWu/sGADObBBwHJNqXuFwzO8Ddc82sOfBd2IEqoyqfmbWiZbA7\nZnY60Sb/QHffFnaeGMwHOprZQcFdFEOARLuj5SngU3d/MOwgleXuv3T3Nu7enui/+xkJVAgIuiZW\nmdnBwa6TSayB8JXAsWaWYWZGNH8iDIDv2op8Dbg8eH0ZUJu/FO2UvaqfmbX+OQMz+xJIA74Pdn3g\n7j8NMdIeBf8xHuR/D9TdG3KkmJlZH2AWsJho09iBX7r726EGqwIz6wf83N0Hhp2lMsysO9HB71Rg\nOXCFu28ON1XszOxOooW4CFgAjAxupqiVzOwFIBvYH8gF7gReBSYArYEVwIXuvimsjBWpIPsvqcJn\nZq0vBiIiEn+1vptIRETiT8VARERUDERERMVARERQMRAREVQMREQEFQOpBczsV8GUx/8J5uM5qgrn\nOMjMFlfyPU+b2bmVvVZ1mNm/gqk+MLP39+J1B5jZ7/bW9STxqBhIqIL1sM8Eerh7d6LTGaza/bsq\nVKsfmgmmyFi4Y64ndz8+Dtco9/9pd38DOMvMMmr6mlI3qBhI2FoA6929GMDdNwSzRGJmR5nZ7GCR\njg/MLDNoAcwysw+Dn2N3PWGwsM2fzGxu8N4ry/zu4WDhj2lUMBOlmY00s3lmtiBYZCYj2N/ezP4d\ntGD+YGZbyrzn1uA9C4MncMtzCWWmNdjxfjPrZ2YzyyxoM76cTO3N7KMy2x13bJvZ12Z2r5l9CJxv\nZjdYdFGchcETqjvkAGdVkE3qORUDCds0oI1FV4YbZ2YnAJhZKtG1IG4IFuk4BSgg+sj9Ke5+JNEp\nD8aWc84RRKdOPobo+hJXBUVkMNApWDTmMqITqJXnZXc/2t17Ap8H54PoFCP3By2Y1QQtETM7NTjv\n0UBP4EgzK+9bfx/gozLbZVsyPYAbgcOADma2UzZ3Xw5sMrNuwa4rgCfLHLLe3Y90938SneW0R/Dv\n7Zoyx3wE9K3gzyz1nIqBhMrd84hO2XwVsA540cyGAYcA37r7x8FxW4MFO9KAJyy6zN8E4NByTtsf\nGGZmC4iuxbAf0Ak4AfhHcL41RFewKk/XoPWxCBhKdMU6iK78NjF4XfYbd3/gVDP7GPg4yN6pnPM2\nDv685Znn7ms8Oj/MQqBtOcc8CVwRdAVdtEuGl8q8/g/wgpldApSU2f8d0WmlRf6PWjGFtdRvwQfg\nLGBWMAg8jOiHanlrEdwMrHX3bmaWTLS1sCsj2qKYvtPO6FoHsXiG6IyPS8zsMqDfjqi7XKPs6z+6\n+9/2cN7i3fyu7OySJZT//+bLRCcimwl8uMvEaWWLzACihW8g8Csz6xIU0gzK//clopaBhMvMDjaz\njmV29SA6S+RSoLmZHREc1zD48N8HWBMcO4zoEou7mgr8NFiXATPrZGYRogXnomBMoQVwYgWxGgJr\ng66qS8rs/wA4P3hddl3fqcBwM8sMrnegmTUt57xLzax92T9+BdcvVzAd8VTgUaJr3/4fwbTRbdz9\nXaLdRVnBnwfgYGBJZa4p9YdaBhK2hsBYM9uH6DfnZcBV7l5kZhcBD1t02cd8ouMGjwAvB11Jb7Pz\nN+IdniDazfJx8OH4HXCOu79iZicBnxCdd39OBZl+A8wL3jcXaBTsvxn4u5n9kuiH8mYAd59uZp2B\nf0cvxxbgJ0S7vcp6g2gBWh5sV3T30+7uinqe6Hq80yo4PjnImEW02DxYZg3lE0mgVdNk79IU1iIx\nMrMG7l4QvL4IGOLugyvx/ubAs+5+WjUy/BzIcveK7liq6H3NgOf3tKqg1F9qGYjE7ggze5joN+6N\nRBcbj5m7rzWzv5lZw6qsK23RJSTbU7XlX9sAP6/C+6SeUMtAREQ0gCwiIioGIiKCioGIiKBiICIi\nqBiIiAgqBiIiAvx/kC4r2gd/J1MAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1d824b1e4e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the data \n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# given this is a 2 class () \n",
    "colors = ['r' if l == 0 else 'b' for l in labels[:,0]]\n",
    "\n",
    "plt.scatter(features[:,0], features[:,1], c=colors)\n",
    "plt.xlabel(\"Scaled age (in yrs)\")\n",
    "plt.ylabel(\"Tumor size (in cm)\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Creation\n",
    "\n",
    "A logistic regression (a.k.a. LR) network is the simplest building block but has been powering many ML \n",
    "applications in the past decade. LR is a simple linear model that takes as input, a vector of numbers describing the properties of what we are classifying (also known as a feature vector, $\\bf{x}$, the blue nodes in the figure) and emits the *evidence* ($z$) (output of the green node, a.k.a. as activation). Each feature in the input layer is connected with a output node by a corresponding weight w (indicated by the black lines of varying thickness). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://www.cntk.ai/jup/logistic_neuron2.jpg\" width=\"300\" height=\"200\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Figure 4\n",
    "Image(url= \"https://www.cntk.ai/jup/logistic_neuron2.jpg\", width=300, height=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step is to compute the evidence for an observation. \n",
    "\n",
    "$$z = \\sum_{i=1}^n w_i \\times x_i + b = \\textbf{w} \\cdot \\textbf{x} + b$$ \n",
    "\n",
    "where $\\bf{w}$ is the weight vector of length $n$ and $b$ is known as the [bias][] term. Note: we use **bold** notation to denote vectors. \n",
    "\n",
    "The computed evidence is mapped to a 0-1 scale using a [`sigmoid`][] (when the outcome can take one of two values) or a `softmax` function (when the outcome can take one of more than 2 classes value).\n",
    "\n",
    "Network input and output: \n",
    "- **input** variable (a key CNTK concept): \n",
    ">An **input** variable is a user-code facing container where user-provided code fills in different observations (data point or sample, equivalent to a blue/red dot in our example) as inputs to the model function during model learning (a.k.a.training) and model evaluation (a.k.a. testing). Thus, the shape of the `input` must match the shape of the data that will be provided.  For example, when data are images each of  height 10 pixels  and width 5 pixels, the input feature dimension will be 2 (representing image height and width). Similarly, in our example the dimensions are age and tumor size, thus `input_dim` = 2. More on data and their dimensions to appear in separate tutorials. \n",
    "\n",
    "[bias]: https://www.quora.com/What-does-the-bias-term-represent-in-logistic-regression\n",
    "\n",
    "[`sigmoid`]: https://en.wikipedia.org/wiki/Sigmoid_function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature = C.input_variable(input_dim, np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network setup\n",
    "\n",
    "The `linear_layer` function is a straight forward implementation of the equation above. We perform two operations:\n",
    "0. multiply the weights ($\\bf{w}$)  with the features ($\\bf{x}$) using CNTK `times` operator and add individual features' contribution,\n",
    "1. add the bias term $b$.\n",
    "\n",
    "These CNTK operations are optimized for execution on the available hardware and the implementation hides the complexity away from the user. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a dictionary to store the model parameters\n",
    "mydict = {\"w\":None,\"b\":None} \n",
    "\n",
    "def linear_layer(input_var, output_dim):\n",
    "    \n",
    "    input_dim = input_var.shape[0]\n",
    "    weight_param = C.parameter(shape=(input_dim, output_dim))\n",
    "    bias_param = C.parameter(shape=(output_dim))\n",
    "    \n",
    "    mydict['w'], mydict['b'] = weight_param, bias_param\n",
    "\n",
    "    return C.times(input_var, weight_param) + bias_param"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`z` will be used to represent the output of a network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_dim = num_output_classes\n",
    "z = linear_layer(feature, output_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning model parameters\n",
    "\n",
    "Now that the network is setup, we would like to learn the parameters $\\bf w$ and $b$ for our simple linear layer. To do so we convert, the computed evidence ($z$) into a set of predicted probabilities ($\\textbf p$) using a `softmax` function.\n",
    "\n",
    "$$ \\textbf{p} = \\mathrm{softmax}(z)$$ \n",
    "\n",
    "The `softmax` is an activation function that maps the accumulated evidences to a probability distribution over the classes (Details of the [softmax function][]). Other choices of activation function can be [found here][].\n",
    "\n",
    "[softmax function]: https://www.cntk.ai/pythondocs/cntk.ops.html#cntk.ops.softmax\n",
    "\n",
    "[found here]: https://github.com/Microsoft/CNTK/wiki/Activation-Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "The output of the `softmax` is a probability of observations belonging to the respective classes. For training the classifier, we need to determine what behavior the model needs to mimic. In other words, we want the generated probabilities to be as close as possible to the observed labels. This function is called the *cost* or *loss* function and shows what is the difference between the learnt model vs. that generated by the training set.\n",
    "\n",
    "[`Cross-entropy`][] is a popular function to measure the loss. It is defined as:\n",
    "\n",
    "$$ H(p) = - \\sum_{j=1}^C y_j \\log (p_j) $$  \n",
    "\n",
    "where $p$ is our predicted probability from `softmax` function and $y$ represents the label. This label provided with the data for training is also called the ground-truth label. In the two-class example, the `label` variable has dimensions of two (equal to the `num_output_classes` or $C$). Generally speaking, if the task in hand requires classification into $C$ different classes, the label variable will have $C$ elements with 0 everywhere except for the class represented by the data point where it will be 1.  Understanding the [details][] of this cross-entropy function is highly recommended.\n",
    "\n",
    "[`cross-entropy`]: http://cntk.ai/pythondocs/cntk.ops.html#cntk.ops.cross_entropy_with_softmax\n",
    "[details]: http://colah.github.io/posts/2015-09-Visual-Information/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "label = C.input_variable((num_output_classes), np.float32)\n",
    "loss = C.cross_entropy_with_softmax(z, label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluation\n",
    "\n",
    "In order to evaluate the classification, one can compare the output of the network which for each observation emits a vector of evidences (can be converted into probabilities using `softmax` functions) with dimension equal to number of classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "eval_error = C.classification_error(z, label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure training\n",
    "\n",
    "The trainer strives to reduce the `loss` function by different optimization approaches, [Stochastic Gradient Descent][] (`sgd`) being one of the most popular one. Typically, one would start with random initialization of the model parameters. The `sgd` optimizer would calculate the `loss` or error between the predicted label against the corresponding ground-truth label and using [gradient-decent][] generate a new set model parameters in a single iteration. \n",
    "\n",
    "The aforementioned model parameter update using a single observation at a time is attractive since it does not require the entire data set (all observation) to be loaded in memory and also requires gradient computation over fewer datapoints, thus allowing for training on large data sets. However, the updates generated using a single observation sample at a time can vary wildly between iterations. An intermediate ground is to load a small set of observations and use an average of the `loss` or error from that set to update the model parameters. This subset is called a *minibatch*.\n",
    "\n",
    "With minibatches we often sample observation from the larger training dataset. We repeat the process of model parameters update using different combination of training samples and over a period of time minimize the `loss` (and the error). When the incremental error rates are no longer changing significantly or after a preset number of maximum minibatches to train, we claim that our model is trained.\n",
    "\n",
    "One of the key parameter for optimization is called the `learning_rate`. For now, we can think of it as a scaling factor that modulates how much we change the parameters in any iteration. We will be covering more details in later tutorial. \n",
    "With this information, we are ready to create our trainer. \n",
    "\n",
    "[optimization]: https://en.wikipedia.org/wiki/Category:Convex_optimization\n",
    "[Stochastic Gradient Descent]: https://en.wikipedia.org/wiki/Stochastic_gradient_descent\n",
    "[gradient-decent]: http://www.statisticsviews.com/details/feature/5722691/Getting-to-the-Bottom-of-Regression-with-Gradient-Descent.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate the trainer object to drive the model training\n",
    "learning_rate = 0.5\n",
    "lr_schedule = C.learning_rate_schedule(learning_rate, C.UnitType.minibatch) \n",
    "learner = C.sgd(z.parameters, lr_schedule)\n",
    "trainer = C.Trainer(z, (loss, eval_error), [learner])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let us create some helper functions that will be needed to visualize different functions associated with training. Note these convinience functions are for understanding what goes under the hood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "# Define a utility function to compute the moving average sum.\n",
    "# A more efficient implementation is possible with np.cumsum() function\n",
    "def moving_average(a, w=10):\n",
    "    if len(a) < w: \n",
    "        return a[:]    \n",
    "    return [val if idx < w else sum(a[(idx-w):idx])/w for idx, val in enumerate(a)]\n",
    "\n",
    "\n",
    "# Defines a utility that prints the training progress\n",
    "def print_training_progress(trainer, mb, frequency, verbose=1):\n",
    "    training_loss, eval_error = \"NA\", \"NA\"\n",
    "\n",
    "    if mb % frequency == 0:\n",
    "        training_loss = trainer.previous_minibatch_loss_average\n",
    "        eval_error = trainer.previous_minibatch_evaluation_average\n",
    "        if verbose: \n",
    "            print (\"Minibatch: {0}, Loss: {1:.4f}, Error: {2:.2f}\".format(mb, training_loss, eval_error))\n",
    "        \n",
    "    return mb, training_loss, eval_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the trainer\n",
    "\n",
    "We are now ready to train our Logistic Regression model. We want to decide what data we need to feed into the training engine.\n",
    "\n",
    "In this example, each iteration of the optimizer will work on 25 samples (25 dots w.r.t. the plot above) a.k.a. `minibatch_size`. We would like to train on say 20000 observations. If the number of samples in the data is only 10000, the trainer will make 2 passes through the data. This is represented by `num_minibatches_to_train`. Note: In real world case, we would be given a certain amount of labeled data (in the context of this example, observation (age, size) and what they mean (benign / malignant)). We would use a large number of observations for training say 70% and set aside the remainder for evaluation of the trained model.\n",
    "\n",
    "With these parameters we can proceed with training our simple feedforward network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the parameters for the trainer\n",
    "minibatch_size = 25\n",
    "num_samples_to_train = 20000\n",
    "num_minibatches_to_train = int(num_samples_to_train  / minibatch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch: 0, Loss: 0.6931, Error: 0.32\n",
      "Minibatch: 50, Loss: 1.4795, Error: 0.36\n",
      "Minibatch: 100, Loss: 1.1093, Error: 0.32\n",
      "Minibatch: 150, Loss: 0.3336, Error: 0.20\n",
      "Minibatch: 200, Loss: 0.1325, Error: 0.08\n",
      "Minibatch: 250, Loss: 0.1334, Error: 0.08\n",
      "Minibatch: 300, Loss: 0.1012, Error: 0.04\n",
      "Minibatch: 350, Loss: 0.1103, Error: 0.04\n",
      "Minibatch: 400, Loss: 0.3099, Error: 0.08\n",
      "Minibatch: 450, Loss: 0.3240, Error: 0.12\n",
      "Minibatch: 500, Loss: 0.3922, Error: 0.20\n",
      "Minibatch: 550, Loss: 0.6709, Error: 0.24\n",
      "Minibatch: 600, Loss: 0.3018, Error: 0.12\n",
      "Minibatch: 650, Loss: 0.1677, Error: 0.12\n",
      "Minibatch: 700, Loss: 0.2782, Error: 0.12\n",
      "Minibatch: 750, Loss: 0.2313, Error: 0.04\n"
     ]
    }
   ],
   "source": [
    "# Run the trainer and perform model training\n",
    "training_progress_output_freq = 50\n",
    "\n",
    "plotdata = {\"batchsize\":[], \"loss\":[], \"error\":[]}\n",
    "\n",
    "for i in range(0, num_minibatches_to_train):\n",
    "    features, labels = generate_random_data_sample(minibatch_size, input_dim, num_output_classes)\n",
    "    \n",
    "    # Specify input variables mapping in the model to actual minibatch data to be trained with\n",
    "    trainer.train_minibatch({feature : features, label : labels})\n",
    "    batchsize, loss, error = print_training_progress(trainer, i, \n",
    "                                                     training_progress_output_freq, verbose=1)\n",
    "    \n",
    "    if not (loss == \"NA\" or error ==\"NA\"):\n",
    "        plotdata[\"batchsize\"].append(batchsize)\n",
    "        plotdata[\"loss\"].append(loss)\n",
    "        plotdata[\"error\"].append(error)\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAACfCAYAAAAMJSWPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH9dJREFUeJzt3Xm8lGX5x/HPF2RVQNwlEVc0NbefW0qCmqbmFuYCrmRp\npFimhfyycOtXVpZpai5oQamZC4JLSormhguIyiYIiIiAiIIsSizX74/7Hs4wzJwz55yZeZ4553q/\nXvM6Z56553muGQ5zzX3fz309MjOcc865fFokHYBzzrn08iThnHOuIE8SzjnnCvIk4ZxzriBPEs45\n5wryJOGcc64gTxLNnKRbJf2svm0l9ZQ0u7zRrT3uTEmHV+JYTUH8t3m71G0bEMfzks4ux75d5WyQ\ndACuPCS9B2wFdDGzT7K2vwHsBWxnZu+bWf9i95mnbYMW2UjqBswENjCzNQ3ZR1MhqQfwBOG9bAG0\nB5YCitt2M7MP6rNPM3sO+Eqp27rmyXsSTZcRPoj7ZDZI2gNoRwM/3Eso8wGosh9IalnuYzSGmb1g\nZh3MrCOwO+F96ZTZlpsgFCUSrGuWPEk0bcOAc7LunwP8NbuBpLslXR1/7ylptqQfS5ovaY6kc/O1\nrdmkQZIWSJohqW/WA8dKGidpsaRZkgZnPe+5+HORpM8kHRif8z1Jk+K2CZL2znrOPpLelPSppHsl\ntc73giWdI+kFSb+XtAAYLGmwpGFZbbpJWiOpRbw/WtLV8XmfSfqXpE0K7H+SpGOz7reU9JGkvSW1\nkTRM0scxzlckbZ5vP3VYJwnEYZurJb1E6GV0lXRe1ns1TdJ5We2PkDQz6/5sSZdIeivG9XdJrerb\nNj4+SNLc2O678X3cts4XFPxC0nuS5km6S1KH+Fi7eJzM+zYm8/7H1zkzvs53JZ3agPfTNYIniaZt\nDNBB0i7xA/E04G/U/g1+K6AD0AX4LnCzpE61tN0ktj0XuF3SzvGxpcBZZtYJ+CbwfUknxMcOjT87\nxm/Lr0g6BfgFcGb8Vn0CsDDrWKcARwHbE4bLzq3lNRwIvAtsCfwybsvtPeXe70NIopsDbYDLCuz7\nHqBv1v2jgQVmNj4+vyPwJcL78n3g81rirI8zCa+5IzAHmAccE9+r7wE3xZ5iRu7rOwU4AtgB2A84\nq75tJR0HXAj0BLoDh+d5biHfI7xvhwI7Et6fG+Jj/Qg93C5x+w+AL2ISuR44Ir7OQ4C3ijyeKxFP\nEk1fpjdxJDAZ+LCO9v8FrjGz1Wb2BOHDfpcCbQ34uZmtNLP/AI8BpwKY2X/MbGL8fQJwH+HDJVt2\nsjoP+I2ZjYvPmWFm2RPjfzSz+Wa2CBgJZPcycs0xs1vMbI2Zrajj9WbcbWbTY/v7a9n/vcAJktrG\n+33iNoCVwKZAdwveMLOlRR6/LneZ2dT477LazB4zs1kAZvYs8DTwtVqe/wczW2BmnwKPUvv7V6jt\nKcCQGMfnwFX1iL8v8Ls4D7YM+F9qku1KYDNq3rdxZrY8PrYG+IqkNvHff0o9julKwJNE0/c3wn/G\nc4GhRbRfmDOZvBzYqEDbT83si6z7swjfBpF0oKRn4lDMIuACwgdBIV2B6bU8Pr/ImAAactbVvGL2\nb2bTgUnA8ZLaEXo898SHhwFPAvdJ+kDSr1W6OZF1XpOk4+KwzEJJnxK+BNT2/tbn/SvUtktOHLMp\nfl6pC+HvI2MW0CYOx/0F+DdwfxzG+j9JLcxsCSEJXwTMkzQiq6fqKsSTRBNnZu8TJrCPAR4q8e47\nxw/KjG2p6an8HRgOfMnMNgZuo+YDJd8QxWzCMEQp5O5/GeGsoYytG7n/+wiJ90RgopnNADCzVWZ2\njZntDhwMHA+U6hTQta8p9mL+SRhK29zMOgOjKP+JAHOBbbLub0vxw00fAt2y7ncDVsQey0ozu9rM\ndgN6AL2BMwDM7EkzO5IwtDmd8HfkKsiTRPPwHeDwOERQSgKuktRK0tcIcw/3x8c2IvQ0Vko6gHXH\n8RcQhhGyk8KdwGWS9gWQtKOkriWKczxwqKSucX7l8kbu7z7C/Eh/anoRSOolaY84/7OUMIxS31N8\ni/mgbwO0Aj4GLM4VHFHP4zTE/cB5krpLag9cUY/n3gv8OJ400AG4lvjeSTpM0u6SRNb7Jmmr2GNq\nB6wiJPvVpXxBrm6eJJqutd/wzGxmZqw/97H67CePucCnhG+Jw4ALzGxafOwHwDWSFhM+TP6RFc/n\nhG/BL0r6RNIBZvZA3HaPpM+AhwmTmPWNd/0XYPbvePy3gNcIcxrrNKnn/uYBLwMHkfW6CN92HwAW\nAxOB0YT3JbMQ8ZZidl/XNjNbDFxC6KktJHzzzn1Nde2z3m3N7FHgVuA/wDvAC/GhQvM+2fu6g/Be\nPU84qWAx8KP4WBdCL3cx8DbwFCGBtAR+Qvj7WgB8lTBx7ipI5bzokKQhwHHAfDPbs0CbXsAfCN+M\nFpjZYWULyDlXMvFsqrFm1ibpWFz5lDtJ9CB0H4fmSxKx6/8ScJSZzZG0mZl9XLaAnHONIukkwlls\nHQgnQiwzs9OSjcqVU1mHm8zsBcJwRCF9gQfNbE5s7wnCuXS7kDAXMpVw5tNFyYbjyi3p2k3dgVaS\nRhMmOm80s2F1PMc5l5B4ppFrRpJOEhsA+xJWbm4IvCzpZTN7N7ehpKTrDTnnXFUyswafHp302U0f\nAE+a2RdmtpBw1sRehRqbWepvgwcPTjwGj9PjrNYYPc7S3xqrEklCFD73+xGgh0KRtPaEmjuTKxCT\nc865IpR1uEnSPUAvYFNJ7wODgdaAmdntZjZF0pOE89dXA7eb2aRyxuScc654ZU0SZta3iDa/A35X\nzjgqqVevXkmHUBSPs7SqIc5qiBE8zrQp6zqJUpJk1RKrc86lhSSsiieunXPOpVizTRIrir3KgHPO\nNWPNNkl873swtJirKzjnXDNW1iQhaYjCtZJrveSgpP0lrZTUu5zxZHz0EYwcCccdV4mjOedc9Sp3\nT+Ju4Bu1NYi1939NuKJXRQwZAr17wyZ5L3XvnHMuo9ynwL4gqVsdzQYQavDvX85YMlatgltvheHD\nK3E055yrbonOSUjqApxkZrdS/ksvAvDoo7DNNrDvvutunzcPTj0Vli/P/zznnGuOki7wdwMwMOt+\nrYniyiuvXPt7r169GrSYZcUKGDRo/e1bbgnt2sHpp8NDD8EGSb8zzjnXAM8++yzPPvtsyfZX9sV0\ncbhppOW/6NCMzK/AZoRr2J5vZiPytC37YrqVK8NkdrducNttoIr0bZxzrnyqYTFdwQJ/ZrZDvG1P\nmJf4Qb4EUSmtWsEDD8DYsXDNNUlF4Zxz6ZFogb+c5qmoudGhAzz2GBxyCBx/POyzT9IROedccrx2\nUwFLloSE4Zxz1ayxw03NIkmYhVuLZru+3DnXXFXDnETinnsOTjwx6Sicc676NIsk8ac/wTHHJB2F\nc85VnyafJD74AJ55Bs46q3H7mTMnTGQvXVqauJxzrhokWuBPUl9Jb8bbC5K+UuoYbrsNzjij8ZPQ\nXbqEBXennBLWUzjnXHNQ1olrST2ApcDQAovpDgImm9liSUcDV5rZQQX2Ve+J6xUrwsK40aPhy19u\nwAvIsWpVmNvYYgu46y5fbOecS79UT1yb2QvAp7U8PsbMFse7Y4AvlfL4c+bAt79dmgQBoVTH/ffD\nxInw85+XZp/OOZdmiZblyGl3GdDdzM4v8HhqrnG9YAH06BFWZ3+l5ANkzjlXOo3tSaSijJ2kw4B+\nQI/a2pWiwF8pbL45jB8fCgI651yaNKkCf/HxPYEHgaPNbHot+0lNT8I556pFquckooIF/iRtS0gQ\nZ9WWIJxzziWj3Gc3rS3wB8wnp8CfpDuA3sAsQiJZaWYHFNhX0T0Js2TOPErquM45V4jXbsqxZk2o\n4Prgg2FtQ6W8/z706wcPPwwdO1buuM45V5tqGG6qqH/9Kyx223rryh63a1fYYQe47LLKHtc558qp\nyfUkvvnNsDaiX78KBJXjo49gl13g3Xdh000rf3znnMvlPYks06fDq6+G61QnYYst4IQTYMiQZI7v\nnHOllmjtptjmRknTJI2XtHdjjnfrraEHkeT6hQED4JZbYPXq5GJwzrlSKXdP4m7gG4UelHQMsKOZ\n7QxcAPy5MQf78EPo378xe2i8/faDk0+GhQuTjcM550oh0cV0kv4MjDazf8T7k4FeZjY/T1tfTOec\nc/VU7XMSXwJmZ92fQ4mL/DnnnGu4VNRuKlZaajc551xaNanaTXmGm6YAPX24yTnnSqMahpsK1m4C\nRgBnw9oLEC3KlyCq2apVSUfgnHMNV+5TYO8BXgK6S3pfUj9JF0g6H8DMHgdmSnoXuA34QX2Pcf31\nMHRoScMumenTYe+9Q6kQ55yrRlW94nrVKthuO3jsMdhrr2Tiqo1ZOCX2mmvg2GOTjsY51xxVZLhJ\n0o6S2sTfe0m6WNLGDT1oqTzyCGy/fToTBISKsAMGwE03JR2Jc841TLHDTQ8CqyXtBNwOdAXuKVtU\nRbr5ZrjwwqSjqN3pp8PYsTB1atKROOdc/RWbJNaY2SrgW8BNZvYToMJ1Vtc1cSJMmQK9eycZRd3a\ntoXzzgsJzTnnqk2xSWKlpD7AOcCjcVur8oRUnNdfh4sugtatk4yiOP37h/kJ55yrNkVNXEvaDfg+\n8LKZ3Stpe+BUM7uuiOceDdxASEhDcp8jqSPwN2BboCVwvZn9Jc9+fJ2Ec87VU8WvTCepM9DVzApW\nds1q2wKYChwBfAi8BpxuZlOy2gwCOprZIEmbAe8AW8bhrex9eZJwzrl6qtTZTc9K6ihpE2AccIek\n3xfx1AOAaWY2y8xWAvcBJ+a0MaBD/L0DsDA3QTjnnEtGsXMSnczsM6A3MNTMDgS+XsTzcgv4fcD6\nBfz+BOwm6UPgTeCHRcbknHOuzIot8LeBpK2BU4GflTiGbwBvmNnhknYERkna08yW5jZsKgX+zMIa\nCuecK7VECvxJOgX4OfCimfWXtAPwWzM7uY7nHQRcaWZHx/uXA5Y9eS3pUeBXZvZivP80MNDMXs/Z\nl82aZVxxRXrLcBTjzjth2jS4rs4pf+eca7yKzEmY2T/NbE8z6x/vz6grQUSvATtJ6iapNXA6oahf\ntlnEoStJWwLdgRn5dnbbbdC5czERp9fhh4drYC9fnnQkzjlXt2InrreR9LCkj+LtQUnb1PU8M1sN\nXAQ8BUwE7jOzydlF/oBrgYPjdbBHAT81s0/y7e/OO+EH9S4BmC477AAHHwx//3vSkTjnXN2KHW4a\nRSjDMSxuOhM4w8yOLGNsuTHYkUcaTz1VqSOWz6hRcOml8OabPjfhnCuvSl1PYnMzu9vMVsXbX4DN\nG3rQhkp7naZiff3rsHIlPP980pE451ztik0SCyWdKallvJ0JLCxnYPkcd1ylj1geElx+OcyeXXdb\n55xLUrHDTd2Am4CvEha/vQQMMLOKfcz5imvnnKu/ipflyDrwj8zshoYeuAHH8yThnHP1lOQ1rn9c\nTCNJR0uaImmqpIEF2vSS9IakCZJGNyIm55xzJdSYnsRsM+taR5tiCvx1IgxfHWVmcyRtZmYf59mX\n9yScc66ekuxJFPOJXUyBv77Ag2Y2ByBfgmjqzPx6E865dKo1SUhaIumzPLclQJci9l9Mgb/uwCaS\nRkt6TdJZ9XoFTcBpp8GYMUlH4Zxz66u1wJ+Zdajt8RLGsC9wOLAh8LKkl83s3dyGTaXAX66vfhVu\nuin8dM65xkikwF+Dd15cgb+BQFszuyrevxN4wswezNlXk52TWLQItt8eJk2CrRO9crhzrqlJck6i\nGMUU+HsE6BEX6bUHDgQmlzmuVNl4Yzj99FDA0Dnn0qSsPQlYe43rP1JzjetfS7qA0KO4Pba5DOgH\nrAbuMLOb8uynyfYkACZODOU6Zs2C1q2TjsY511Qktpiu0pp6kgD4zndg4EDYZZekI3HONRWeJJxz\nzhWU9jkJ55xzVcyThHNN3Jo1MHVq/seWLoX33oPPP69oSK6KlD1JFFO7KbbbX9JKSb3LHZNzzcEX\nX8Add8Duu4drseQbrR03Dnr2DGfYdewI3bvD174G116bf58rVnhCaW7KvU6iztpNWe1GAZ8Dd5nZ\nQ3n25XMSzhVhwQK45Ra49VbYbz+47LKQCGq7CqIZLF4M8+fDvHnQti0ceOD67YYPDxUC2raFrbaC\nrl3D5XhPOCEcy6VPY+ckal1xXQJrazcBSMrUbpqS024A8ACwf5njqQqffgonngjPPAMblPtfyDU5\nAwaEXsHo0fDlLxf3HCn0JjbeuPaz6046KfRQFi0KCWXGDHjhBRg71pNEU1Xuj6B8tZsOyG4gqQtw\nkpkdJmmdx5qrzp3DN7vhw+Hb3046Gldt7r23vNdOl8LfaOfOsOuucOyxhdvefDPMnBl6Mj16hOe4\n6pKG76k3ANlzFQX/vJtq7aZ8BgwI9Zw8Sbh8Vq2CN96A/fP0vcuZIOrr4INDz/jGG6FvX9hxx5Aw\nfvSjUIrGlV5TrN00I/MrsBmwDDjfzEbk7KtZzUmsXBn+Ez3+OOy5Z9LRuLRYsgSGDIEbboCdd4Yn\nn4QWVXKO4n//GybKn3sulKHp1i3piJqHVC+mk9QSeIcwcT0XeBXoY2Z5azNJuhsY6RPXwbXXhjId\nd9yRdCQuaR98EL6N33UXHHEEXHopHNDEBmfNQs9jjz1Cb6NnzzAx7hon1YvpzGw1cBHwFDARuM/M\nJku6QNL5+Z5SzniqzfnnhySxZk3Skbik/fa3oXf5+uvwj380vQSRcdttoec8fDj8z/+E3vT55/tF\nuZLkZTmcS5BZOFto2TJYvjxs23bbZGNKCzOYMgUmT4beeVZPrV4dhtrSNAeTRqkebiolTxIuzebO\nDd/wly2r+cBftiwMl/ziF+u3f+WVUPV3+XJo1Qo23DDcevSAe+6pfPzV6Mkn4eyz4dBDw9DUoYeG\noapqmaOpFE8SzdAXX4TFTC493nsP/vAHaN++5gN/ww1DkjjmmPXbr1wZVi63b+9rYRrj/ffDRPh/\n/hN+LlwIV1wBl1ySdGTp4UmiGTr0UNhrL7jmmrD4yTkXzJ0bku8OO6z/2DPPhBpWHTrARhuFnx06\nhLabblr5WGfOhDlzanqfS5eG1ext2pT2OGlfce3K4OGHYdAg2G23MKHZt6+Py1aCWRgKatEC+vRJ\nOhqXT22X//3447C2ZMmSdW8DB8Kpp67ffvBgGDmyJplkbueeG87CynXzzeGU9cyHfuZ23XXhlN9c\nd98NTz8depwbbRR+nnRS6ZNEY1XqynQ3UHNluutyHu9LzWK6JUB/M3s7z368J5FjzBjo3z/0Jm65\npfgSDK7+Jk0KRfIWLYI//zl/XSPXtHz4YeiZ5CaVHj1C0cRcr78e6l5lhhozH/ybbx6GFZOS6uGm\nYgr8xQV3k81scUwoV5rZQXn25Ukij1WrQoLYbrtQZM2V1tKlcPXV4Vvf4MEhKbdsmXRUzhUv7cNN\ndRb4M7MxWe3HEOo9uSJtsAFcfHHSUTRdZ5wBnTrBhAmw5ZZJR+Nc5SVe4C/Hd4EnyhqRc/Vw333Q\nrl3SUTiXnNRMXEs6DOgH9CjUpjkV+CuF3/8+nC576aXpmwyrFp4gXLVpcgX+4vY9gQeBo81seoF9\n+ZxEPb33XhiKmjo1nHlxxBFJR5ReI0eGM1aSOBXSuXJKde0mwkT1TpK6SWoNnA7kVnfdlpAgziqU\nIFzDbLcdjBgBv/kNnHdeOFV27tyko0qXGTPg+OPhJz8JZ7M459ZVqVNg/0jNKbC/lnQBoUdxu6Q7\ngN7ALEK58JVmtt68hfckGmfZMvjlL0PvIl/Zh08+Caf3QVgPkHmrN900XOUs1/z54ToBmfYQJni3\n2KI6VhB/8UVInjfeGC7v+eMfQ+vWSUflXOml+hTYUvIkURpm+RfeXXEFDB1a85gUblddFerj5Lrq\nqpBssve1aFHYfsEF67d/8UX46KOw2GnrrcP1kZOaJ1m+HPbZJ5zrfsMNXlDPNW2eJFxVGDIkjPvP\nnRtu8+aFxUZDhsC3vrV++2nTQkJr0QJWrKi5de+ef97g8cfhnXdCDyG7/bnn5r9o04QJoRicc02d\nJwlXldasCcXY2rULySLXT38KDz0UEkWbNjW3664LFT9z3X57WBXdtu267U88EXbaqfyvx7m08iTh\nnHOuoLSf3eScc66KlT1JSDpa0hRJUyUNLNDmRknTJI2XtHe5YyqnUi5iKSePs7SqIc5qiBE8zrQp\na5KIBf7+BHwD2B3oI2nXnDbHADua2c7ABcCfyxlTuVXLH47HWVrVEGc1xAgeZ9qUuyextsCfma0E\nMgX+sp0IDAUws1eATpK8lJpzzqVAuZNEvgJ/uVVec9vMydPGOedcAspdu+lk4Btmdn68fyZwgJld\nnNVmJPArM3sp3v838FMzG5ezLz+1yTnnGiDN15OYA2SvZ90mbstt07WONo16kc455xom8QJ/8f7Z\nsLZq7CIzm1/muJxzzhWhrD0JM1st6SLgKWoK/E3OLvBnZo9LOlbSu8AywjUlnHPOpUDVrLh2zjlX\neVWx4rqYBXkVjGWIpPmS3sra1lnSU5LekfSkpE5Zjw2KCwUnSzqqQjFuI+kZSRMlvS3p4pTG2UbS\nK5LeiHEOTmOc8bgtJI2TNCKtMcZjvyfpzfievprGWCV1kvTPeMyJkg5MYYzd43s4Lv5cLOnitMUZ\nj3uJpAmS3pL0d0mtSxqnmaX6Rkhk7wLdgFbAeGDXBOPpAewNvJW17TrCGVkAA4Ffx993A94gDOtt\nF1+HKhDjVsDe8feNgHeAXdMWZzx2+/izJTCGsLYmjXFeAvwNGJHGf/OsOGcAnXO2pSpW4C9Av/j7\nBkCntMWYE28L4EPCCTapihPoEv/NW8f7/wDOKWWcFXujG/EmHAQ8kXX/cmBgwjF1Y90kMQXYMv6+\nFTAlX6zAE8CBCcQ7HPh6muME2gOvA/unLU7CGXejgF7UJIlUxZh1vJnApjnbUhMr0BGYnmd7amLM\nE9tRwPNpjJOQJGYBneMH/4hS/1+vhuGmYhbkJW0Li2dkmdk8YIu4PfGFgpK2I/R8xhD+aFIVZxzG\neQOYB4wys9dSGOcfgJ8A2RN4aYsxw4BRkl6T9N24LU2xbg98LOnuOJRzu6T2KYsx12lA5nqOqYrT\nzD4Ergfej8dcbGb/LmWc1ZAkqlEqzgaQtBHwAPBDM1vK+nElHqeZrTGzfQjf1g+QtDspilPSN4H5\nZjaecHndQhJ/L6NDzGxf4FjgQklfI0XvJ+Hb7r7AzTHOZYRvt2mKcS1JrYATgH/GTamKU9LGhNJG\n3Qi9ig0lnZEnrgbHWQ1JopgFeUmbr1hvStJWwEdxe1ELBctB0gaEBDHMzB5Ja5wZZvYZ8CxwNOmK\n8xDgBEkzgHuBwyUNA+alKMa1zGxu/LmAMMx4AOl6Pz8AZpvZ6/H+g4SkkaYYsx0DjDWzj+P9tMX5\ndWCGmX1iZquBh4GDSxlnNSSJYhbkVZpY91vlCODc+Ps5wCNZ20+PZxtsD+wEvFqhGO8CJpnZH9Ma\np6TNMmddSGoHHAlMTlOcZva/Zratme1A+Nt7xszOAkamJcYMSe1j7xFJGxLG0t8mXe/nfGC2pO5x\n0xHAxDTFmKMP4ctBRtrifB84SFJbSSK8n5NKGmclJ4AaMTlzNOEMnWnA5QnHcg/hTIcV8R+oH2HS\n6N8xxqeAjbPaDyKcQTAZOKpCMR4CrCacCfYGMC6+h5ukLM6vxNjGA28BP4vbUxVn1rF7UjNxnboY\nCeP9mX/ztzP/V9IWK7AX4cvfeOAhwtlNqYoxHrc9sADokLUtjXEOjsd8C/gr4SzQksXpi+mcc84V\nVA3DTc455xLiScI551xBniScc84V5EnCOedcQZ4knHPOFeRJwjnnXEGeJFzqSFojaWjW/ZaSFqim\nTPfxkn5axz62lnR//P0cSTfVM4ZBRbS5W1Lv+uy3lCSNlrRvUsd3zYMnCZdGy4A9JLWJ948kqyiZ\nmY00s9/UtgMzm2tmp2ZvqmcM/1vP9lVFUsukY3DVwZOES6vHgW/G39cpjZDdM4jf5v8o6UVJ72a+\n2ccyLm9n7W/b+M37HUm/yNrXw7Fi6tuZqqmSfgW0i1VKh8VtZ6vmYj5/zdpvz9xjZ4txTIrVTidI\n+lcm+WX3BCRtKmlm1ut7OF40ZoakCxUuLDNO0kuxqFvG2TGmtyTtH5/fXuHiWGMkjZV0fNZ+H5H0\nNGE1rnN18iTh0siA+4A+8QN1T+CVPG0ytjKzQ4DjCRdbyddmf+BbhJIQp2QN0/Qzs/3j4z+U1NnM\nBgHLzWxfMztL0m6EnkUvCxVrf1jEsbPtBNxkZnsAi4GTa3ndGbsDJxEK9P0SWGqhauoY4Oysdu1i\nTBcS6nUB/Ax42swOAg4HfhdrYwHsA/Q2s8MKxODcOjxJuFQyswmEK2f1AR6j9jLdw+NzJlNTNz/X\nKDNbZGZfEOoF9YjbfyRpPOHDdxtg57g9+3iHA/80s0/jcRbV89gzzSzTqxkbX1ddRpvZcgvVRxcB\nj8btb+c8/954/OeBDpI6Egr7Xa5wnY5ngdbUVFIeZWaLizi+c0Co7e5cWo0Afku4ItxmtbRbkfV7\noWSyXn19ST0JCeBAM1shaTTQtp4xFnPs7Dars46xipovarnHzX6OZd1fw7r/b/NdN0DAyWY2LfsB\nSQcR5nucK5r3JFwaZT5s7wKuMrOJDXhuriMlbRyHXU4CXiRUH/00JohdCZfKzfhv1uTuM4Qhqk0A\nJHWu57ELbX8P2C/+fkqBNnU5LcbUg3BVsiXAk8DFaw8u7d3AfTvnScKlkgGY2Rwz+1MxbWu5n/Eq\nYZhpPGHoaBzwL6CVpInA/wEvZ7W/HXhb0jAzmxQffy4O4Vxfz2MX2v47oL+ksYTSzoXUtt8vJI0D\nbgG+E7dfQ3hdb0maAFxdy76dq5WXCnfOOVeQ9yScc84V5EnCOedcQZ4knHPOFeRJwjnnXEGeJJxz\nzhXkScI551xBniScc84V9P+NdwBVwihTLwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1d824b93a90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAACfCAYAAADXno+tAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXeYVdXVh98fVlDELlaiCLZIEAWN+CEYJSixJLGhEcVP\nJcZEY2LDhjUaNUajIZbYe8UW9ZNEsRekCCIISokCGrEgBAWE9f2x92XOXO7cOffOzC0z632e+8wp\nu/zuuWfOOnvvtdeWmeE4juM4hdKq3AIcx3Gc6sQNiOM4jlMUbkAcx3GconAD4jiO4xSFGxDHcRyn\nKNyAOI7jOEXhBqTKkPQ3SecUmlbSnpI+alp1y+udLmmvUtRVbUhaJmmrUudtKMm6C7kHc5QzX9L3\nGlObUz7cgFQIkmZI+lbSulnHx8Z/3i0AzOxEM7s0TZk50hY16UdSh6jB7xcafD0aMvGqzrySRkr6\nRtLXkv4j6RFJGzWgrjrrTnsPSnpB0rG1CjFra2YzGlFXpq4ZkhbG7z8//v1LY9fj1MYfCJWDAdOB\nAZkDkr4PtKZhD53GQFGDmrwiaaWmrqORKPY3acg1zJfXgF+Z2VpAZ2Bt4M85CynO8DX5b99ADOhv\nZmtFI7WWmZ2cK2Gue6zQ+66K7tMmxQ1IZXEXcHRi/2jgjmQCSbdJuihu7ynpI0m/k/SppFmSjsmV\ntuaQhkj6TNI0SUckTuwnaYykeZJmShqayPdi/PtVfLPbNeY5XtJ78di7krom8uwk6R1JX0q6T9Kq\nub6wpKMlvSLpakmfAUMlDZV0VyJNrTf++GZ7Ucz3taRns1tuibzvSdovsb9SfEPvKmk1SXdJmht1\nvilpg1zlpEVSd0mvxfJmSbpO0spZyfpL+jDquCIr/7FR8+eSnsm0PNNWD2BmXwGPAN+PZd4maZik\nf0iaD/SWtKqkq+JvPSeeXy2h43RJsyV9LGkQCYOZfV9JOjC2lOdJmiqpr6RLgP8Brk+2BlS7K2wt\nSXfG6zBdiW6xeF+8LOlKSV/E69Uvzfdf4WDte2wu4R7LdUySzlVozXwi6XZJa8UyMvfgsZJmAv9K\n+6M0Z9yAVBZvAG0lbRMflocBd5P/7a890BbYBDgO+KukdnnSrhvTHgPcJKlTPLcAOMrM2gH9gV9K\nOiCe6xX/rhXf7N6UdAhwPvCL+NZ7APB5oq5DgL7AlsAPYn11sSvwAbARkOkayX7Dz94fQDCwGwCr\nAafVUfa9wBGJ/X7AZ2Y2LuZfC9iUcF1+CXyTR2calgK/jeX9ENgL+FVWmoOAbvFzoGI3j6QDgbPi\n+Q2Al4H7ChUgaX3g58CYxOEBwMVm1hZ4FfgjsDXQJf7dlPB7Eh/UvwN+BHQC9s5TVw/CS87v473T\nC5hhZudG/b/Oag0kf8frCffu94DewMBorDL0ACYB6wFXArcUch2yyNxjG1Jzj2UfGwQMBPYEtora\nrs8qpxewLfDjBmhpNrgBqTwyrZB9CP88s+tJv5jwYFhqZs8QDME2daQ14DwzW2JmLwH/AA4FMLOX\nzGxi3H4XuJ/wj5Qkacj+F7jCzMbEPNPMLDlIf62ZfRrfhp8Ekq2TbGaZ2TAzW2Zmi+r5vhluM7MP\nY/oH85R/H3CApNXj/gBqHspLCA+nzhYYa2YLUtafEzMbY2ZvxfL+DdzEitfxcjObZ2YfA9dQ0205\nGLjMzKaY2TLgcqCrpM1TVn+dpC+AsYT75veJc4+b2RtR4yLgeODUqOO/sa6MjkMI13eSmX0DXJCn\nzmOBW8zs+Vj2HDObkie9YHk32mHAWWa20MxmAn8CjkqknWlmt1oI2HcH0F7ShnnKfiy2Vr6Mf/83\ncS7XPZZ97AjgajObaWYLgSHA4arp8jNgqJl9U8B92qzJblo75edu4CXCm/udKdJ/Hh82GRYCa9aR\n9ksz+zaxP5PQGkGhW+oyQrfHqvHzUJ56Nwc+zHP+0yxNG+dJW4x32CdZ5ef8zmb2oaT3gP0lPUVo\nKZ0fT98FbAbcH1ttdwPnmNnSIvQAEFt0VwO7EMavVgZGZyX7OLG9/DcAOgDXSvpTpjjCQ2tT0l2j\n35jZrXWcW54/dtO1AUZLy98JWlHzgrAJ8HaWxrpawZsTXkQKZX3Ctfl3Vj2bJvaX/8Zm9o2C2DWB\n/9RR5oFm9kId53Jdv+xjm0QNST0rE1rGGT7GWY63QCqM+NY6HdgXeLSRi19HUuvE/hbUtHDuAR4D\nNjWztYEbqXlo5Bow/gjo2Ei6ssv/L+EBlyGf8UnD/YS3ywOBiWY2DcDMvjOzi81sB2B3YH9CF0ZD\n+Buh5dgxXsdzWPHhm2xRdKDmN/gIGGxm68bPOma2Zqbl0ECS13guwejukKhr7dgFBTAnh8a6nAby\n3Qf5HA3mElqAHbLqmZUnT33U52RQ37HZOfQsofbLULkdWioKNyCVybHAXrH7oDERcKGkVST9D2Gs\n48F4bk1CC2VJ7NdOjht8Biyj9oPi78BpkroBSOpYQFdLfYwDeknaPLYMzmpgefcTxmNOJIyJACCp\nt6Tvxy6KBYSHxbLcRayAgNXjQHzmI0K/+ddmtlDStrHObE6XtHa8XidHfQA3AGdL2j7qayfp4MK/\nbn5il9DNwDWxNYKkTSX1jUkeBI6RtJ2kNtS02HJxCzBIUp84CL2JpEwX6qeEsYRcGpbFei6VtKak\nDsCphFZhubgPOFXS9yStSRgXuT/Rwq90T7SS4wakckj62U/PjC1knyuknBzMAb4kvGndRXjbnRrP\n/Qq4WNI84FzggYSebwj/TK/GvuUeZvZwPHavpK+B4YSB40L1rvgFzP4Z6x8PjCKModRKUmB5nwCv\nA7uR+F4Ep4KHgXnAROAF4gNMYbLcsHzFAvMJb/LfxL99COMOR8ZrciM1xiGZ73FCt9aY+N1ujTof\nI4xF3C/pK8L375eVN5+eQs6dSRhAfiPW9RzB/Rcze5YwNvM8MIU8HkdmNoow+HwN4TqOJLRsAa4F\nDlHwKLsmh5aTCddtGqHb9m4zu63A75Hkyejxlfk8Uk/6bG4l/P4vEbpnF0aNaetvcaipF5SKHh3X\nEIzVLWb2xzrSdQdeAw4zs0cLyes4juOUniY1ILFrYArBHXA24W3ycDObnCPdCMKb3K1m9mjavI7j\nOE55aOourB7A1OgWt4TQnD8wR7rfELoS/lNEXsdxHKcM5DUgCrN263KLS0O2++HH1HbTQ9ImwEFm\n9jdqD1LVm9dxHMcpH3nngZjZ0jh9v52ZzWsiDdcQBvSKRpIPbjmO4xSImTXIsyxNF9YCYIKkWyT9\nJfNJWf4sajwyIEzayvbz3oXgdTIdOBgYFkNopMm7HDOr6M/QoUPLrsF1uk7X6Tozn8YgzUz0Ryl+\nQtsoYOvo4z0HOJxEtFkAM1vuJy7pNuBJM3tCIdpl3ryO4zhO+ajXgJjZHQqRVDvHQ+9bGNSuFwtd\nYL8m+JhnXHEnSRocTttN2Vnqy5umXsdxHKfpqdeASOpNCGQ2gzDIvbmkoy0E46sXC5OStsk6dmMd\nabMXn1khb7XSu3fvcktIhetsXFxn4+I6K4t654FIGg0cYWbvx/3OwH1mtnMJ9KVCkjVWn57jOE5L\nQBJWgkH0VTLGA8BCqOZVGlKp4ziOU/2kGUR/W9LfCaGuAY6kdqhnx3EcpwWSpgtrNeAkYI946GVg\nmFXQgireheU4jlMYjdGFldeARFfaO83syKIrqCcgYpzzcTEhjPYSwippr8ZzMwgRPpcBS8ysRx11\nuAFxHMcpgCY3ILGSVwhrUywuuPAUAREltbGwfCSSdgQeNLPt4v40YGcz+7KeehrfgNx8M/TtCx06\n1J/WcRynymgMA5JmDGQaYR2IJwgrxQFgZlenyLs8ICKApExAxOUGJGM8ImtSe0EfUY41S8xg+nTo\n1g0OOgiGDIGtty65DMdxnEomzcP5Q+CpmLZt4pOGVAERJR0kaRJhcZ3kXBADRkgaJen4lHWmZ+JE\nuOGGFY9L8Ic/wNSpsPnm8MMfwi9+Ae+91+gSHMdxqpW8LZA4BtLWzE5rShEWVmJ7TNIewCXAPvFU\nTzObE5fdHCFpkpm9kquMCy64YPl2796980/kGTsWLrkEXnkFTsvz1dZdFy64AH73Oxg2DB5/HLbf\nvqDv5jiOUwmMHDmSkSNHNmqZacZAXjezHxZVuLQbcIGZ9Yv7ZxFCmNS5sqCkD4HuZvZF1vGhwPxc\nXWepx0DefBMuvjgYkNNPh+OPhzXWKOxLOY7jNANKNQYyLo5/PETtMZA0ARbrDaYoqaOZfRi3uwGr\nmtkXktoArcxsgaQ1gL7AhWm+VJ089xz07w8PPwyrr96gopZjBqNHw847h64vx3GcFkKaFkiuRe4t\nO25Vnvz9gGupceO9PBlMUdIZwEBgMWFJ29PM7HVJWwLDCeMgKwP3mNnlddRRPjfezz6D3XeH9u3h\n3HOD55YbEsdxKpySuPFWA7UMyLJl8NprsMce+TM1JkuXwoMPwqWXQuvWwZDsvz+0Kr0DmeM4Thqa\nNBaWpAcT29mT/55rSKVNwtKl8MAD0LUrnHIKLFhQurpXWgkGDIDx4+Hss+Gii+DWW0tXv+M4Thmo\nswUiaayZ7RS3x5hZt1znKgFJZp07w3rrwXnnQb9+5e1GMgsGbeU0Q0yO4zilp6kH0fP1bVVev9cN\nN0Dv3pUx/iC58XAcp9mT7ynXRtJOhG6u1nFb8dO6FOIKok+fciuon3HjQheb4zhOMyBfF9YL+TKa\nWaondgODKebNmyij8oMpLl4cJiFedVUIj+I4jlNGKt4LqyHBFNPkTZRR+QYE4NVX4eCD4Z13YMMN\ny63GcZwWTKlWJGwIy4MpmtkSIBNMcTl5ginWm7fq6NkTjjkGTjghDLQ7juNUMU1tQBoSTDFV3qrj\nggtCpN877ii3EsdxnAZREa5CeYIppqagYIrlZLXV4K674LjjYOBAn2zoOE5JKEswRQBJmwIdSBgc\nM3spRb6igykCndPmrZoxkCRLl4YJiI7jOGWgJMEU4yz0w4D3gKXxsAH1GhAaFkyx3rxVjRsPx3Gq\nnDRdWAcB25jZokILN7Olkn4NPEeNK+6kZDBF4OeSksEUD82Xt1ANjuM4TtOQJhrvM8AhZlbC4FKF\nUZVdWI7jOGWkVOuBLCSsCfIvYHkrxMxObkjFToJvv4WXXgqh4B3HcaqENC2Qo3MdN7OK8UOt+hbI\nnDkhxMlTT0H37uVW4zhOC6BkM9ElrUrwigJ4P07sqxiq3oBACEU/dGhYbrd15YUacxyneVGSmeiS\negNTgb8Cw4ApknqlrUBSP0mTJU2RdGaO80dIeid+XpHUJXFuRjw+VtJbaeusSg47DHbaCYYMKbcS\nx3GcVKTpwhoNHGFm78f9zsB9ZrZzvYWni4W1GzDJzObF4IkXmNlu8dw0YGcz+7Keeqq/BQLwxRfQ\npQvceSfstVe51TiO04wpVSysVTLGA8DMpgCrpCw/TSysN8xsXtx9g9rhSpRSY/Ng3XXh73+Hxx4r\ntxLHcZx6SeOF9bakvwN3x/0jgbdTlp8rnlWPPOmPA55J7BswQtJS4CYzuzllvdVLv37h4ziOU+Gk\nMSAnAicBGbfdlwljIY2KpD7AIGCPxOGeZjZH0gYEQzLJzF7Jlb9qYmE5juOUgbLFwiq68JSxsOLA\n+SNAv0xYkxxlDQXmm9nVOc41jzEQx3GcEtGkYyCSHox/J0gan/1JWf7yeFbRFfhw4ImserYgGI+j\nksZDUhtJa8btNYC+wLuFfDnHcRyn6cjXhXVK/PuTYgtPGQvrPGBdYJgkAUvMrAewETBckkWd95jZ\nc8VqqVqmToUpU6B//3IrcRzHqUUaN94/mtmZ9R0rJ826C2v8eNh7bxg1Cjp0KLcax3GaCaVy4821\nuNO+DanUKYAuXeD3v4dBg2DZsvrTO47jlIh8YyAnSpoAbJs1/jEdmFA6iQ6nnQaLFsF115VbieM4\nznLq7MKS1A5YB7gMOCtxar6ZfVECbalp1l1YGT78EHbbLUTt3W67cqtxHKfKKUkwxeiKO9HM5sf9\ntYDtzOzNhlTcmLQIAwJw662w1lpw8MHlVuI4TpVTKgMyFuiWeULH+FZvm1m3lCL7AddQ44WVPQfk\nCCAzID8f+JWZjU+TN1FGyzAgjuM4jUSpBtFrPZ3NbBnpZrBnjM31wI+BHYABkrbNSjYN6GVmPwAu\nAW4qIK/jOI5TJtIYkGmSTpa0SvycQnjop6EhwRTrzes4juOUjzQG5JfA7sAsQjDEXYETUpafK5ji\npnWkhdrBFAvN6ziO45SQeruizOw/hBAkTUodwRRT0yKDKT7ySAgB36dPuZU4jlPhlDSYoqQzzOwK\nSdcRwqrXwsxOzpEtu4yigymmzRvPtcxB9GefhcGDw2z1du3KrcZxnCqiSb2wJO1vZk9KOjrXeTO7\nI4XAlYD3CSsSzgHeAgaY2aREmi2AfxGCKb5RSN5E2pZpQABOPBG++QZuv73cShzHqSJK4sbbUKIr\n7rXUuOJengymKOlm4GfATMIKhJlgijnz1lFHyzUgCxZA165w5ZXw05+WW43jOFVCU7dAniRH11UG\nMzugIRU3Ji3agAC8+mqYXPjOO7DhhuVW4zhOFdAYBiTfIPpV8e/PgPbULGk7APi0IZU6jUzPnnDO\nOfDFF25AHMcpGWlmor9tZrvUd6yctPgWiOM4ToGUaib6GpK2SlS6JbBGQyp1HMdxqp80IUlOBUZK\nmkYY5O4ADG5SVY7jlI6JE2HhQujUCdZeu9xqnCoilReWpNWATByqyWa2KHUF9QdT3Aa4DegGnG1m\nVyfOzQDmActIeGflqMO7sBynUKZMgfPOC0sEtG8flk/+4IOwnc0330Dr1qXX6DQZJenCktQGOB34\ntZm9A2whKdU66SkDIn4O/Aa4MkcRy4DeZrZTXcbDqYPTToNJK0yZcZzAjTcG54uuXYPRGDsW5s+H\njTZaMe2yZbDxxrDJJrDnnnD88XDFFTB8uK+S2cJJMwZyG7AY+GHcn0WImpuGNMEU55rZaOC7HPmV\nUqOTTadOcNRRsGRJuZU4lch++8HkyTBkCKwRhzSl8MmmVavg4ffmm3D++bDzzvDpp/DAA7nTL1kC\nTz0F778Pixc37fdwykqaMZCOZnaYpAEAZrZQynXX5CRXQMRCWhIGjJC0FLjJzG4uIG/L5oQT4LHH\n4NJLIREnzHEA2HzzwtK3ahXybL45/OhH+dPOmwd//WvoEvvoo5CnUyfo3h0uuqh4zU7FkcaALJbU\nmjipUFJHIPUYSAPpaWZzJG1AMCSTzOyVXAlbZDDFfEhwyy2w007Qv3/453VaFt99F0Lc9OkDHTuW\nrt7114dnYlDtxYth+vRgTObPz51++nS47jro3DkYmk6dYLPNgtFyGo2SBlNcnkDaBzgX2B54DugJ\nHGNm9SopMCDiUMJ661dnn6vvvA+i5+GBB2Do0NDH7YOgLQMzePTRMLl0441h2DDYbrtyq6qb2bPh\n7rvDoP7UqeHz1Vdw+OFhGWenSWjqmejErqrJhNnouxHGJE4xs7kpyx8FbC2pAyEg4uGEmex1Vpmo\nuw3QyswWSFoD6AtcmLJeJ8Nhh8HXX8PSpeVW4pSC55+Hs84KrY9rr4W+fXOPU1QSm2wCZ5xR+9j8\n+aErLBcjR8LTT0OvXrDHHu56XEbStEAmmNmORVdQfzDFjYC3gbYEr6sFhNbOBsBwQtfZysA9HkzR\ncfLwySew995w7rlw6KHNtwtoypTQsn7pJXjjjdA916sXDBwIu1RMgIyKpyTReCXdAVxvZqMaUlFT\n4gbEcSJmld/iaEyWLIExY4Ix2Xln2GuvciuqGkplQCYDnYAZwH8J3UxmZl0aUnFj4gbEaXG0NEPR\nEAYPDhMhe/UKn06d/NpROgPSIddxM5vZkIobEzcgTovhyy/DJL6ZM+Hee8utpjqYMiWMm7z0Erz4\nYmi19OoVPL9yTZxsITT1eiCrA78EtgYmEMYvck32KztuQApg0aLg1nvvvR76vZqYOxeuuQZuuAEO\nOijM7dlss3Krqj7MgvF98cXg5bXaaiumGTQI2rYNHmzt24e/G28MO+5Y+nElM/j886CzbdtGLbqp\nDcgDwBLgZWBfYKaZnVKEyIbEwsqbN5HODUghDBkSwpwMH+5N+WrgoouC8Tj0UDjzTNhyy3Irat48\n+GBwLZ4zJ3w++SR8xoyBlbMcV83C77PRRjWGZv31gxdc584rlr1oEVx2WVhJdMGC4G22YEEICfPk\nkyumnz8//N6XXhq64hqRpnbj3T7jfSXpFsKa5AWRiIX1I2A2MErS42Y2OZEsEwvroCLyOsVwwQXQ\nowfccQccc0y51Tj10b07jB/vLY5Sceih6dN+9114+I8ZU2Nw5s4NrYUJE1ZMv9JKwehstFHwHmvb\nFtZcs+7WRdu2obwKJV8LZIyZdatrP1XhYSLhUDPbN+6nnkhYYF5vgRTK+PEhJMXbb0OHnMNcjuM0\nY5o6Gu8PJH0dP/OBLpltSV+nLD9XLKxNS5DXqY8uXULE3mOO8YiqlcB774XQ6v4i5FQRdXZhmdlK\npRTSUDwWVhGcdhrssIOPg5STceNC//aLL8Jvfxu6RFZZpdyqnGZIWWJhNajwBsTCKjCvd2E51cXo\n0WEsavToYMgHD64Jq+44JaDJY2E1AkXHwioir+NUD5MnQ79+8NBDsPrq5VbjOEXRpC0QKD4WVgyi\nuELeOurwFojjOE4BlGQmejXgBqQRWbo0uBo6DccsrIuxzz4+ruFUHE3qhZXxtkp4Xs0vwgvLqSZm\nzAj98K1a1f7UNf9g1qwV0+ZL31JYujR0TXXtGtbkmD273Iocp0nwFohTm7pcenOFcDCr2+00V/rn\nnw8LWw0eHCZPNTe++w7uuw/+8Ado1y645e63n3u5ORVJU88DSVa0h6RBcXt9SR5LobmSq0VRV/wf\nqbD0G28Mb74JW20VXFfrWjCoWvnHP8IywtdfD6+/HmKOufFwmjFpovEOBXYBtjGzzpI2AR4ys56l\nEJgGb4FUGZMmhXhATz8NJ54Y4js1hxaJh1h3qohStUB+ChxAWAsEM5tN8JhKhaR+kiZLmiLpzDrS\n/EXSVEnjJO2UOD5D0juSxkoqOBaXU6Fstx3ceWdojSxYsGKAukpnwQJYuHDF4248nBZGGgOyOL7e\nG0BcnzwViYCIPwZ2AAZI2jYrzb5ARzPrBAwG/pY4vQzobWY7mVmPtPU6VULHjvDnP1fPPIivvoJL\nLgldcCNGlFuN45SdNAbkQUk3AmtLOh74J3BzyvJ7AFPNbKaZLQHuBw7MSnMgcCeAmb0JtItzQyBM\nLGymCzs7eXn1VZg2rdwqAnPnhnXGt94aPvgAXn4ZDsy+jR2n5VHvw9nMrgIeBh4BOgPnm9l1KctP\nExAxO82sRBoDRkgaFY2X01IYNy6EnB84MMzaLhczZ8I22wQjMmoU3H572HccJ3UokwlAa8IDPUeQ\n+yajp5nNkbQBwZBMMrNXciX0YIrNjJNOgiOPDB5NvXpBnz5hTkWXLqXV0aFDiJTbgpc+dZoHZQmm\nKOk44HzgeUKX0p7ARWZ2a72FpwiIKOkG4AUzeyDuTwb2NLNPs8qqFWwx65x7YTVnFiwIS7k++2wY\ne2iqwWr3onJaECUJZSLpfWB3M/s87q8HvGZm9bbjJa0EvE9YVXAOYVXDAWY2KZFmP+AkM+sfDc41\nZrabpDZAqxgTaw3gOeBCM3suRz1uQFoyd94ZJu9lc9RRodWSJr0ZHHJIGCR3nBZAqaLxfg7MT+zP\nj8fqxcyWSvo14eGfCYg4KRlM0cyelrSfpA8IrsKDYvaNgOGSLOq8J5fxcBx+8pMwXpLNOusUln5T\nX6/McQoh35K2v4ubXYEdgccJYyAHAuPN7JhSCEyDt0Acx3EKo6lbIJnJgh/GT4bHG1Kh4ziO0zzw\nYIqO4zgtkJKMgUQX2jMIM8mXTxk2s70aUrHjOI5T3aSZ5X0PMBnYErgQmEFYbtZxHMdpwaQxIOuZ\n2S3AEjN70cyOBVK3PooIpti1kLzVQmNP4GkqXGfj4jobF9dZWaQxIEvi3zmS+sdoueumKbzIYIo3\npM1bTVTLDeU6GxfX2bi4zsoizTyQSyS1A34PXAesBfw2ZfnLgykCSMoEU0wGN6oVTFFSJpjiliny\nOo7jOGUiTTDFp8xsnpm9a2Z9zGxnoGPK8osJpphJkyav4ziOUyaKcuOV9G8z2yJFup8DPzazE+L+\nL4AeZnZyIs2TwGVm9lrc/yfB62vL+vImynAfXsdxnAIpRSiTXKStdBaQNDSbxWPZaTbPkWbVFHmB\nhl8Ex3Ecp3CKXawp7Rv/KGBrSR0krQocDjyRleYJYCAsj977VYzEmyav4ziOUybqbIFImk9uQyHC\n2iD10pBginXlLeTLOY7jOE1Hswhl4jiO45Seql5vvJImGkq6RdKnksYnjq0j6TlJ70v6v+gOnTk3\nJE6enCSpb4k0bibpeUkTJU2QdHKF6lxN0puSxkadQytRZ6LuVpLGSHqiUnVKmiHpnXhN36pgne0k\nPRTrnShp10rTKalzvI5j4t95kk6uQJ2nSnpX0nhJ90hatdE1mllVfgjG7wOgA7AKMA7Ytox69iCE\nvh+fOPZH4Iy4fSZwedzeHhhL6EL8XvweKoHG9kDXuL0mYbGvbStNZ6y7Tfy7EvAGYU5RxemM9Z8K\n3A08UYm/e6x7GrBO1rFK1Hk7MChurwy0q0SdCb2tgNkER6CK0QlsEn/zVeP+A8DRja2xZBe6CS7Q\nbsAzif2zgDPLrKkDtQ3IZGCjuN0emJxLK/AMsGsZ9D4G7F3JOoE2wNtA90rUSfAOHAH0psaAVKLO\n6YSwRMljFaWTMEn5wxzHK0pnlra+wMuVppNgQGYC60Sj8ERT/K9XcxdWNUw03NDi2u5m9gmwYTye\nrX0WJdYu6XuEFtMbhBuqonTGbqGxwCfACDMbVYk6gT8Dp1Pb4aQSdRowQtIoScdVqM4tgbmSbovd\nQzcpLG1daTqTHAbcG7crRqeZzQb+BPw71jfPzP7Z2Bqr2YBUIxXhsSBpTeBh4BQzW8CKusqu08yW\nmdlOhDfWhTaYAAAFQElEQVT8HpJ2oMJ0SuoPfGpm48g/N6rs1xPoaWbdgP2AkyT9DxV2PQlvyt2A\nv0at/yW8GVeaTgAkrQIcADwUD1WMTklrE0I/dSC0RtaQdGQOTQ3SWM0GJM0kxXLzqUJcLyS1B/4T\nj9c1ebLJkbQywXjcZWaZ1SUrTmcGM/saGAn0o/J09gQOkDQNuA/YS9JdwCcVphMzmxP/fkbouuxB\n5V3Pj4GPzOztuP8IwaBUms4M+wKjzWxu3K8knXsD08zsCzNbCgwHdm9sjdVsQCpxoqGo/Sb6BHBM\n3D6amuWAnwAOj14RWwJbA2+VSOOtwHtmdm2l6pS0fsY7RFJrYB9gUqXpNLOzzWwLM9uKcP89b2ZH\nAU9Wkk5JbWKrE0lrEPrtJ1B51/NT4CNJneOhHwETK01nggGEF4cMlaTz38BuklaXJMK1fK/RNZZy\nwKkJBor6ETyJpgJnlVnLvQRvjEXxxxtEGMD6Z9T4HLB2Iv0QgqfDJKBviTT2BJYSPNbGAmPiNVy3\nwnTuGLWNA8YD58TjFaUzS/Oe1AyiV5ROwthC5jefkPlfqTSdsd4fEF4OxwGPErywKlFnG+AzoG3i\nWEXpBIbG+sYDdxC8VRtVo08kdBzHcYqimruwHMdxnDLiBsRxHMcpCjcgjuM4TlG4AXEcx3GKwg2I\n4ziOUxRuQBzHcZyicAPiVBWSlkm6M7G/kqTPVBNKfX9JZ9RTxsaSHozbR0u6rkANQ1KkuU3Szwop\ntzGR9IKkbuWq32kZuAFxqo3/At+XtFrc34dEEDgze9LMrshXgJnNMbNDk4cK1HB2gemrCkkrlVuD\nUx24AXGqkaeB/nG7VjiJZIsitgKulfSqpA8yLYIY/mZCorwt4hv7+5LOT5Q1PEavnZCJYCvpMqB1\njBZ7Vzw2UDWLNd2RKHfP7LqTRB3vxaiz70p6NmMYky0ISetJmp74fsPjokDTJJ2ksHDQGEmvxSB6\nGQZGTeMldY/52ygsfvaGpNGS9k+U+7ikfxFmKjtOvbgBcaoNA+4HBsSHbRfgzRxpMrQ3s57A/oTF\ndHKl6Q78lBBG45BE188gM+sez58iaR0zGwIsNLNuZnaUpO0JLZLeFqIHn5Ki7iRbA9eZ2feBecDP\n83zvDDsABxECIl4KLLAQvfYNYGAiXeuo6SRCDDSAc4B/mdluwF7AVTHeGMBOwM/MrE8dGhynFm5A\nnKrDzN4lrJo2APgH+UOpPxbzTKJm7YNsRpjZV2b2LSH+0h7x+G8ljSM8mDcDOsXjyfr2Ah4ysy9j\nPV8VWPd0M8u0hkbH71UfL5jZQgtRYL8CnorHJ2Tlvy/W/zLQVtJahECKZymstTISWJWaqNYjzGxe\nivodBwjx9x2nGnkCuJKwEuD6edItSmzXZWhWWCNB0p4E47CrmS2S9AKweoEa09SdTLM0Ucd31Lzg\nZdebzGOJ/WXU/p/OtfaDgJ+b2dTkCUm7EcaXHCc13gJxqo3Mg/hW4EIzm1hE3mz2kbR27Mo5CHiV\nEAX2y2g8tiUsoZxhcWKg+XlCt9e6AJLWKbDuuo7PAHaJ24fUkaY+Doua9iCsSDcf+D/g5OWVS12L\nLNtx3IA4VYcBmNksM7s+Tdo8+xneInRdjSN0R40BngVWkTQR+APweiL9TcAESXeZ2Xvx/IuxW+hP\nBdZd1/GrgBMljSaE4K6LfOV+K2kMMAw4Nh6/mPC9xkt6F7goT9mOkxcP5+44juMUhbdAHMdxnKJw\nA+I4juMUhRsQx3EcpyjcgDiO4zhF4QbEcRzHKQo3II7jOE5RuAFxHMdxiuL/ASnM58AN96HCAAAA\nAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1d828c23e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Compute the moving average loss to smooth out the noise in SGD\n",
    "plotdata[\"avgloss\"] = moving_average(plotdata[\"loss\"])\n",
    "plotdata[\"avgerror\"] = moving_average(plotdata[\"error\"])\n",
    "\n",
    "# Plot the training loss and the training error\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(1)\n",
    "plt.subplot(211)\n",
    "plt.plot(plotdata[\"batchsize\"], plotdata[\"avgloss\"], 'b--')\n",
    "plt.xlabel('Minibatch number')\n",
    "plt.ylabel('Loss')\n",
    "plt.title('Minibatch run vs. Training loss')\n",
    "\n",
    "plt.show()\n",
    "\n",
    "plt.subplot(212)\n",
    "plt.plot(plotdata[\"batchsize\"], plotdata[\"avgerror\"], 'r--')\n",
    "plt.xlabel('Minibatch number')\n",
    "plt.ylabel('Label Prediction Error')\n",
    "plt.title('Minibatch run vs. Label Prediction Error')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation / Testing \n",
    "\n",
    "Now that we have trained the network. Let us evaluate the trained network on data that hasn't been used for training. This is called **testing**. Let us create some new data and evaluate the average error and loss on this set. This is done using `trainer.test_minibatch`. Note the error on this previously unseen data is comparable to training error. This is a **key** check. Should the error be larger than the training error by a large margin, it indicates that the trained model will not perform well on data that it has not seen during training. This is known as [overfitting][]. There are several ways to address overfitting that is beyond the scope of this tutorial but the Cognitive Toolkit provides the necessary components to address overfitting.\n",
    "\n",
    "Note: We are testing on a single minibatch for illustrative purposes. In practice one runs several minibatches of test data and reports the average. \n",
    "\n",
    "**Question** Why is this suggested? Try plotting the test error over several set of generated data sample and plot using plotting functions used for training. Do you see a pattern?\n",
    "\n",
    "[overfitting]: https://en.wikipedia.org/wiki/Overfitting\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Run the trained model on newly generated dataset\n",
    "test_minibatch_size = 25\n",
    "features, labels = generate_random_data_sample(test_minibatch_size, input_dim, num_output_classes)\n",
    "\n",
    "trainer.test_minibatch({feature : features, label : labels}) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking prediction / evaluation \n",
    "For evaluation, we map the output of the network between 0-1 and convert them into probabilities for the two classes. This suggests the chances of each observation being malignant and benign. We use a softmax function to get the probabilities of each of the class. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "out = C.softmax(z)\n",
    "result = out.eval({feature : features})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us compare the ground-truth label with the predictions. They should be in agreement.\n",
    "\n",
    "**Question:** \n",
    "- How many predictions were mislabeled? Can you change the code below to identify which observations were misclassified? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label    : [1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1]\n",
      "Predicted: [1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1]\n"
     ]
    }
   ],
   "source": [
    "print(\"Label    :\", [np.argmax(label) for label in labels])\n",
    "print(\"Predicted:\", [np.argmax(result[i,:]) for i in range(len(result))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualization\n",
    "It is desirable to visualize the results. In this example, the data is conveniently in two dimensions and can be plotted. For data with higher dimensions, visualization can be challenging. There are advanced dimensionality reduction techniques that allow for such visualizations [t-sne][].\n",
    "\n",
    "[t-sne]: https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 8.00762081 -8.00761986]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEPCAYAAACtCNj2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8FPW9//HXJyRAAgQBQbmJRqWAiop4a9WmXlpE1NYq\n3uq1ytFW661WxV8LnB711GoV613xQi2KESu2HhAU410QBRQQxfYAIkIBEYGEXMjn98cOe2JMQnaz\nu7OX9/Px2IeZ2dmZN4r7yWe+M/M1d0dERAQgL+wAIiKSPlQUREQkSkVBRESiVBRERCRKRUFERKJU\nFEREJColRcHMJpjZGjP7oN66W83sIzObb2ZTzKw4FVlERKRpqeoUHgV+1GDdDGAfdz8AWArckKIs\nIiLShJQUBXd/A9jQYN1L7l4XLL4D9ElFFhERaVq6jClcCEwLO4SISK4LvSiY2Y1AjbtPCjuLiEiu\nyw/z4GZ2PjAcOHoH2+kBTSIicXB3i2X7VHYKFrwiC2bDgGuBk9y9akcfdveMfY0ZMyb0DLmaP5Oz\nK3/4r0zPH49UXZI6CXgL6G9mK8zsAuDPQEdgppm9b2b3piKLiIg0LSWnj9z9rEZWP5qKY4uISMuF\nPtCcC0pLS8OO0CqZnD+Ts4Pyhy3T88fD4j3vlEpm5pmQU0QknZgZnsYDzSIikuZUFEREJEpFQURE\nolQUREQkSkVBRESiVBRERCRKRUFERKJUFEREJEpFQUREolQUREQkSkVBRESiVBRERCRKRUFERKJU\nFEREJEpFQUREolQUREQkSkVBRL5l3bp1zJo1i4ULF4YdRVJMRSGJautqGf3yaNZuWRt2FJEWe+ut\nt9inpIRxp5zCsEMP5fKLLkIzH+YOFYUkuvXNW7nljVsYdO8gyhaVhR1HpEXO/elPeWjTJl7duJGP\nKip4+amnmDFjRtixJEVUFJJk1aZVjHt1HADrKtYx8pmRjCwbqa5B0lpdXR3/u2YNw4PlTsBRdXV8\n+umnYcaSFFJRSJJenXox9Yyp9CnuE11XtrhMXYO0iLtTVlbG6OuvZ8KECdTW1qbkuHl5eexXUsKj\nwfIXwHQz9t9//5QcX8JnqThXaGYTgBHAGncfHKzrAkwG+gHLgJHuvrGJz3umntPcuHUjV794NY/M\nf+Qb608bdBr3DL+H7h26h5RM0tk1l13GS489xqlbtvBSURE7H3UUZS+8QF5e8n+PW7x4MSf84AdY\nRQXra2oYfeONXPfb3yb9uJJ4Zoa7W0yfSVFROALYDEysVxT+AKx391vN7Dqgi7tf38TnM7YobDf9\n0+lc/PeLWfn1yui6nYt25t7h93LaPqeFmEzSzdq1a9mrTx+WVVfTBagG9unQgUmvvMLBBx+ckgzV\n1dUsW7aMbt260a1bt5QcUxIvnqKQktNH7v4GsKHB6pOBx4OfHwd+nIosYRm21zAWXrqQCw+4MLpO\nYw3SmE2bNlGcn89OwXJboGd+Pps2bUpZhrZt29K/f38VhBwU5phCD3dfA+Duq4EeIWZJic7tOzPh\n5AlMO3uaxhqkSf369aNrz56MbdOGFcADZvxvmzYMGTIk7GiSA1Jy+gjAzPoBf693+uhLd+9a7/31\n7t7oryVm5mPGjIkul5aWUlpamuTEyaWxBmnO559/zqizzmL+Bx+w5+67c/9f/8qgQYPCjiVprry8\nnPLy8ujyuHHj0nNMARotCh8Bpe6+xsx2BV5x94FNfDbjxxSaorEGEUmWtB1TCFjw2u554Pzg5/OA\nqSnMkjY01iAi6SRVVx9NAkqBbsAaYAzwHFAG9AWWE7kk9asmPp+1nUJ96hpEJJHS9pLU1sqVogAa\naxCRxFFRyCLqGkSktVQUsky2dw01NTWUlZWxZs0ajjjiiJTdmCWSK1QUslQ2dg21tbUM//732bpg\nAfvX1PBMmzb88f77+dm554YdTSRrqChksWzrGp599lluO+88Xt+8mTbAh8D3i4pYv3kzZjH9HRaR\nJqT7JanSCtl2N/S6desYUFdHm2B5ALBp61a2bdsWZiyRnKeikGGy5b6GI488kr+78zqRJyXekJ/P\nUUOHkp+fH3Y0kZym00cZLNPHGv7xj39w2YUXsnrDBkoPPZSJzz5Ljx5Z/wgskZTRmEIOyraxBhFJ\nHBWFHJbpXYOIJJ6KQo5T1yAi9akoCKCuQUQiVBQkSl2DiKgoyLeoaxDJXSoK0ih1DSK5SUVBmqWu\nQSS3qCjIDqlrEMkdKgrSYuoaRLKfioLERF2DSHZTUZC4qGsQyU4qChI3dQ0i2UdFQVpNXYNI9lBR\nkIRQ1yCSHVQUJKHUNYhktowsCmZ2FfBzoI7IVL0XuHt1g21UFEKirkEkc2VcUTCzXsAbwAB3rzaz\nycAL7j6xwXYqCiFT1yCSeeIpCukwR3MboIOZ5QNFwKqQ80gjsmVuaBFpXjqcPvoVcBNQAcxw93Ma\n2UadQhpR1yCSGTLx9NFOwBTgNGAj8AxQ5u6TGmznY8aMiS6XlpZSWlqawqTSkMYaRNJPeXk55eXl\n0eVx48ZlXFE4FfiRu18cLJ8DHOrulzXYTp1CmlLXIJK+MnFMYQVwmJm1NzMDjgE+CjmTxEBjDSLZ\npcWdgpkVAz2BSndfkbAAZmOAM4AaYB5wkbvXNNhGnUIGUNcgkl4SPqZgZp2AS4GzgI7AOqA90I3I\npaT3uvvrcSduaUgVhYyhsQaR9JGMovAS8FfgeXdfX299HnAwcA7wvrs/0sQuEkJFIfOoa0isrVu3\nsnHjRnr06EHkTKvIjmXc1UctpaKQmdQ1JMa9d93Fb669lnZm9OzZk+dffpmSkpKwY0kGSGpRMLNB\nwO5A/vZ17v58LAeLl4pCZlPXEL/Zs2fz06OP5vWKCvYAbs/L4+kBA5i9aFHY0SQDJK0omNlDwFBg\nMZFnFAG4u58bc8o4qChkPnUN8bn77rtZdO213Ld1KxC5GqMwL4/qmhry8sK+eFDSXTIvST0CGOLu\nZ7v7OcErJQVBskPn9p2ZcPIEpp09jT7FfaLryxaXMejeQZQtKgsxXfrq168f7+TnszVYfhXo3bWr\nCoIkTUv/Zs0G+icziOQG3dcQmxEjRrDvD3/I/h06cFJxMWd26MAjTz0VdizJYi09fXQk8Hfgc6AK\nMCKnj4YkN170+Dp9lIU01tAy7s6bb77J2rVrGTp0KH379g07kmSIZI4pLAWuIzLfwfYxBdz9n7GG\njIeKQvbSWINI8iSzKLzj7ofFnayVVBSyn7oGkcRLZlG4Gygmcgqpavt6XZIqiaSuQSSxklkU/tLI\nal2SKkmhrkEkMXRHs2QNdQ0irZe0+xTMbEIwIc725S7BDW0iSaH7GkTC0dLTR/Pc/cAG697XJamS\nCuoaROKTzDua88ysc70DdQEKYjmQSLzUNYikTks7hQuAa4HJwarTgVvd/bHkRfvG8dUpCKCuQSQW\nyX5K6mDg6GBxlrt/EGO+uKkoSEO6Qklkx5I6R7O7f+DudwavlBUEkcboGUrJ8cknn3DIoEG0zc9n\n4G678c4774QdSVJMl6RKxlPXkBg1NTUM2n13rvjiC37uzjTgF8XFLPznP9l5553DjidxSGqnIJKu\n1DUkxvLly6nZuJHL3CkETgEGmLFgwYKwo0kKqShIVtAVShFVVVU89thj3HbbbcydOzemz3bt2pUv\na2pYHSxvAf5VU6MuIce09Oa1k83sIzPbaGZfm9kmM/s62eFEYpXLXUNVVRXHHn44f73sMlaOHs2I\no45icgxzL3Tt2pXrb7iB7xYVcXm7dhzeoQPDTjmFwYMHJzG1pJuWXpL6KfATd/8w+ZEaPb7GFCRm\nuTbW8MQTT/DoJZfw0pYtGDAXOGmnnVi1YUNM+ykvL2f+/PnsueeejBgxArOYTklLGknmA/HedPfv\nxZ2s+X13Bh4G9iUyV8OF7j67wTYqChKXXLqv4a677uLj3/yGe6oiDzLeAnTLz6eyulpf7DkqmUXh\nTqA78BwJfnS2mT0GvOruj5pZPlDk7l832EZFQVolF7qGDz74gGMPO4xnKyvZFxhdUMDKI47g+Vmz\nwo4mIcm4R2ebWTEwz9333MF2KgrSarnQNUydOpUrL76YdV9/zTFHHskjkyfTtWvXsGNJSDLu0dlm\ntj/wILAY2J/IadAr3L2ywXYqCpIwO+oa3J2bx43j3vHjcXdG/eIXjLnpprQ/BVNXV8fatWvp1q0b\n+fn5YceRNJDwomBm17j77Wb2p8bed/erY8zYcP8HAe8Ah7v73OA01UZ3H9NgOx8z5v9WlZaWUlpa\n2ppDS45rrms49N9D+ct1/8nkLVvIA84oKuKCm27isiuvDCdsC8yZM4dTjj+erRUV1OXlMXHyZEaM\nGBF2LEmx8vJyysvLo8vjxo1LeFH4sbs/Z2Y/b+x9d58Qy8Ea2f8uwNvuXhIsHwFc5+4nNthOnYIk\nRWNdQ9vqAi57robbF0eWnwfu/+53+Z833wwn5A5UVVVR0rMn92zYwI+B2cCIoiIWLF1Kr169wo4n\nIYqnU2i2x3T354J/turLv5n9rzGzz8ysv7t/AhxD5FSSSEpsv6+hftdQ3baGP42EzxbBPS/A0kqj\nS/f0HW9YsWIF7Wpq+HGwfCiwX0EBCxcuVFGQmO2oU7gPuMvdP2rkvULgNKDG3Z+MO0BkXOFhIvMz\n/Au4wN03NthGnYIkXWNdQ/st0HZWe96e+B6DBg0KMV3Tvv76a/r26MG7VVX0B9YB+xUWMuu99xg4\ncGBCjrF27VoqKiro27cveXl6EEKmSMazjx4GbjKzhWb2pJndZWYPmtkrwBwil6n+Lc68ALj7Anc/\n2N0PcPdTGhYEkVRp7G7orR3g6xO3MnbR2LS9G7q4uJg7/vxnjiws5CfFxRxYVMQlV16ZkIJQV1fH\npeefz959+nD4wIF8d//9Wbs2Pf89SGK09JLUYuAQoCdQCXzk7ouSnK3+8dUpSEpl4n0NS5YsYeHC\nhZSUlDBkSGJmyn3kkUd48PLLmVlRQUfg6oIC1hx/PJOmTk3I/iW5Mu6S1JZSUZAw5MJ9DTty+ahR\n7PnQQ2y/7mohcFrv3ny0cmVzH5M0oUdniySQnrwKJQMGMLOwkNpg+cW8PEr2bPZeU8lw6hREWiBX\nu4aqqipOOuYYVixYQLe8PFa2a8fLb7/NnioMGSHpp4/MrJ27V+14y8RSUZB0kYljDa21bds25syZ\nQ2VlJUOHDqW4uDjsSNJCyXz20SHABKCzu+8WXEZ6kbtfHl/U2KgoSDrJ1a5BMk8yi8I7wOnAc+5+\nYLBuobvvG1fSGKkoSDrKxa5BMksyB5rz3H15g3XbYjmQSLbJ5VneJHu1tCh8FpxCcjNrY2ZXAp8k\nMZdIRgjjCqWZM2dy4w03MH78eCoqKhK+f8ltLT191AO4Czg2WPUScJm7r0titvrH1+kjSXupGGu4\n989/5tbrr+eCigrmt2/P5yUlvDp3LoWFha3et2SfZI4p9HT3Lxqs21lFQeTbkjXW4O507dCB2ZWV\n9AccOLZjRy568EHOPPPM1geXVnF33nzzTVatWsWQIUPYa6+9wo6U1DGFlWb2FzNrV2/djFgOJJIr\nkjXW4O5sqa6mb7BsQL+6OjZt2tT60NIq7s6l55/PBcOGMfmiizh88GD+9uyzYceKS0uLwgfAu8Cb\nZrZHsC69p6ESCVEyxhry8vI44eijubRdO5YRmTD978DRRx+dqNgSp9dee41Xpkxh/pYtTNm0iemV\nlVx4zjnU1dWFHS1mLS0K7u53AVcDL5jZ8US6VxFpRqK7hsefeYZtw4dzRJcu/H7vvZkybVpanKbI\ndStWrGCIGR2C5SFAVXU1W7ZsCTNWXFo6pjCv3v0JvYGngQPcvUPzn0wMjSlINtB9Ddlr8eLF/ODg\ng5lVUcE+wP1m3NW3L4uXN7ySP7WSOdDcx91X1lsuAI5091mxx4ydioJkC90Nnb0mPfEEl1x8MVZX\nR89dduG5GTMYMGBAqJkSXhTM7Ex3f9LMftXY+8EppaRTUZBso64hO9XW1rJx40a6du2KWfjDrsm4\n+qhL8M/uTbxEQrdkyRL+8Y9/sHTp0rCjtJjuhs5O+fn5dOvWLS0KQrz06GzJaLfffhe//e1NtG07\nhOrq97jjjpv5j/+4KOxYMVHXIMmSzDGFW4BbgArgBeAA4Cp3nxRP0FipKEhjVqxYwXe+cyBbt84H\n+gL/pH37oaxY8Qndu2dWI6uxBkmGZN68dry7fw2MAL4ABgLXxZhPJKE+++wz2rXbG6K3c+1J27a9\n+fzzz8OMFRfN8ibpoqVFIT/453DgaXf/Et2nICHr378/NTWfAm8Ha16hrm4NJSUlYcZqFY01SNha\nWhSmmdlC4FBgppntDKR8BjaR+rp3705Z2UQ6dBhBUVFvOnU6nalTJ2f8zGDqGiRMLR5oDp6U+qW7\n15pZRyKzsCWkTzezPGAusNLdT2rkfY0pJNEf/nA7N910K7W1VZx11s+47747KCgoCDvWN2zatInp\n06ezbds2jjvuOLp16xZ9r6qqitWrV7PrrrvSrl27ZvaSeTTWIK2R9Dmak8XMrgIOAopVFFJr8uSn\nufDC31FRMRXoTGHhOfzyl4fyxz/+V9jRotauXctBBx3Jhg39gPa0a/c+c+a8mtGniWKlK5QkHskc\naE4aM+tDZKzi4bCz5KK//e1FKiquAr4D7Epl5Tiefz69HoA7duwtrF59HJs3v8jmzVPZsOEXXHnl\njWHHSimNNUiq7LAoWETPJGa4A7gWDVyHYtddu5Gf/1G9NUvo3r1bk9uHYdmyVdTUHBZdrqs7lBUr\nVoWYKBwaa5BUyN/RBu7uZjYT2DfRBzezE4A17j7fzEpp5nHcY8eOjf5cWlpKaWlpouPkpOuvv5on\nnzycTZvWsG1bZwoKpjB+/PSwY33Dccd9j/Lye6ioGA60o7DwDo455nthxwrN9q6h/ljD9q7htMUa\na8hl5eXllJeXt2ofLb157Qngdnef16qjfXu/NwM/A2qBQqAT8Ky7n9tgO40pJNH69et5+umnqaqq\n4qSTTkq7c/V1dXVceumVTJjwAAAnn3wakyZNyLpB5XhorEGak8w7mhcROen8T2ALkd/o3d2HxBO0\niWN8H7hGA83SlOrqaty92WLw4YcfMnv2bHr27Mnxxx9PXl7ow2ZJpyuUpCnJLAp7Nrbe3f8Zy8F2\ncAwVhRTYsmULS5cupXv37vTu3TvsOAk1adJTXHTRFZgNJy9vPqWl/Zk69cmcKAygrkG+LamXpJrZ\nPsCRweLr7r4oxnxxU1FIjHnz5nHssSdSW9uFqqrPueaaK7jppjFhx0oId6dDhy5UVr4GDAaq6djx\nYMrK/sCwYcPCjpcy6hqkvqRdkmpmlwFlwG7B62kz+0XsESVMJ598Fl9+eStff/0hVVUfM37847z+\n+uthx0qIrVu3Ul1dCewXrGmL+36sXr06zFgppyuUpLVa2lePAg5x99HuPprI4y4uSV4sSQR3Z8qU\nKVx11bXceeedfPbZx8Dpwbvdqas7hkWLUtbwJVVhYSH9+w8mL+8WYBvwLu4zOOyww3b00ayk+xok\nXi0tCgZU11uuoZnLRyU9jB49lvPO+x133tmV0aNfoqCgC/BM8O6X5OWVJ2W6wK1bt1JXV5fw/e7I\ntGnPMHDg85i1o1On4UyceH/o0yGGSV2DxKOlA82/Ac4EpgSrfgI86e63JTFb/eNrTCFGW7dupVOn\nLtTWLgd6AHUUFR0ELKegYE+qq5dz6aUXcfvtNyfsmOvXr+fEE89gzpzXaNMmn1tuuYWrr250Jtdm\nzZgxg99fey1bNm/mlJ/9jBt+9zvatGnT4s/X1NSQn5+f0bNfJZrGGnJTsgeaDwGOCBZfd/d3Y8wX\nNxWF2H311Vf06NGXmpqvgMgXaqdOJ3H33afSv39/evTokfD7EYYN+ymzZvWkpuZO4DOKio5m6tSH\nOPbYY1u8j3fffZcRpaXcX1FBL+CqoiJ+eMUVjL05ccUrl+kKpdyS7KJQDPSh3l3Q7v5BTAnjpKIQ\nn4MPLmXBgn2pqbkSeJ3i4hv45JMF7LLLLkk5XnHxLmzaNA/oBYDZ/+O3v81n3LixLd7H9ddeS9Ft\nt/G7YHkBcHqvXizJwIlz0pW6htyRzKuPxgAfAQ8C9wSvu2NOKCk1ffoUfvSjdey883Hsv/8jvPrq\n9KQVBIDu3XcFtjeQdRQWzqV3714x7aN9YSHr650qWg+0153LCaWxBmlOS8cUPgYGu3soE+uoU8gM\nr732GsOH/xQ4DrNlDBjQljfeeDGmx1GsXLmSQ/bbj7M3baLXtm3cXlTEbQ8/zBlnnpm84DlMXUN2\nS+Ydzc8Co9x9XbzhWkNFIXMsW7aMV199lZ122onhw4fHNVnPihUruPeuu9j81Vf8+IwzYhqTkPho\nrCE7JbMoHAQ8B3xAvWk43f2UWEPGQ0VBwuDu/PHmm5n4wAMUFBRwzZgx/Ozcc3f8wQylriH7JLMo\nLAQeAT4Eohegu/vLsYaMh4qChOHO225j4tixPLhlC5uBc4uKuO/ppznhhBPCjpZU6hqyRzKLwrvu\nfnDcyVpJRUHCcMR++/GfCxdydLD8ADB75EgemTw5zFgpoa4hOyRzOs7XzOz3ZnawmQ3e/oojo0jG\nKOrQgTX1lr8wo6hTp9DypJKuUMpdLe0UGntqmrv7UYmP1Ojx1SlIypWXl3Pa8OFcXlnJ5rw8Hu/Q\ngdfnzqV///5hR0spdQ2ZK6k3r4VJRSF31dbWMnr0OJ566lkKCvIYM+Zazk3hYO97773HU3/5C23b\nt+fno0al3ax0qaSxhsyTzDGF0Y2td/eUPHtARSF3XXHFb7j//v+hunoVkWnCF3DJJedz333jw46W\nk9Q1ZJZkFoXr6i22B04AFrn7BbFFjI+KQu7q0qUPX321GXgZOAhYQ0HBvsybV84+++wTcrrcpa4h\nM6Ts9JGZtQemu3tpzB+Og4pC7urRY3fWrt0K/N9kOe3aHUNZ2ZWceOKJ4QUTdQ0ZIJlXHzXUjsjD\n8USS6ve/vxHYBLwQrPkIswUMGjQoxFQCukIpWzXbKZhZvrvXmtk8YPuGbYCewM3ufmcKMqpTyHG3\n3XYbN974X7h3IC9vMw88cDfnnXdO2LGkHnUN6Snhp4/M7H13H2Jme9ZbXQusTuXD8VQUZOvWraxc\nuZJddtmFTjlyr0Am0lhDeklGUZjn7ge2OlkrqShIY9auXcvy5cvZY4896NatW9hxJKCuIX0koyis\nBP7U1Pvu3uR7iaSiIA098cQkRo26jIKCftTWruDxxx/i1FNT8nxGaSF1DeFLRlH4ArgPaHSn7j4u\npoTf3n8fYCKwC5EH7T3k7nc1sp2KgkStXr2akpJ9qKx8lci9C/MoLDyGlSs/pWvXrmHHk3rUNYQr\naWMKrU7W9P53BXZ19/lm1hF4DzjZ3Zc02E5FQaLeeusthg+/io0bZ0fXFRcPprz8cQ48MPSzndII\ndQ3hSMYlqTHtLFbuvtrd5wc/byYy5WfvZB5TMt8ee+xBdfWnwOJgzQJqalbSr1+/MGMlVEVFBZec\ndx57dO/OkL33ZubMmWFHapVhew1j4aULufCAC6Pr1lWsY+QzIxlZNpK1W9aGmE7q21Gn0NXdv0xJ\nELPdgXJg36BA1H9PnYJ8w8SJT3DJJb+ioGAPamqW8dhjDzBy5Klhx0qYC04/nU3PP88tW7fyMXBB\nURGvzJ7NvvvuG3a0VlPXkDoZ+0C84NRROfB7d5/ayPs+ZsyY6HJpaSmlpaUpyyfpac2aNSxbtoyS\nkhK6d8+uc9Ndior4uLKSHsHylQUF9Ln5Zn7961+HmitRNNaQHOXl5ZSXl0eXx40bl3lFwczygX8A\n09y90aecqVOQXNO3Wzee+/JLDgqWf1pUxHG3384ll1wSaq5EU9eQXBnZKZjZRGCdu1/dzDYqCpJT\nHn/0UW785S/5j8pKPm7Xjnk9e/L2ggUUFxeHHS3h1DUkT8YVBTP7HvAakbmfPXiNdvfpDbZTUZCc\nM2vWLGZOn07X7t0ZNWoUnTt3DjtSUqlrSLyMKwotpaIgkhvUNSSWioKIZAV1DYmhoiAiWUNdQ+up\nKIhI1lHXED8VBRHJSuoa4qOiICJZTV1DbFQURCTrqWtoORUFEckZ6hp2TEVBstIXX3zB559/zl57\n7cVOO+0UdhxJI+oampeMR2eLhGr8+HsoKdmHY465mL59+/Pyyy+HHUnSSOf2nZlw8gSmnT2NPsV9\nouvLFpcx6N5BlC0qCzFdZlKnIGlryZIlDBlSSmXlbKAfUE6nTiNZv/5zCgoKwo4naUZdw7epU5Cs\n8vHHH1NQMJRIQQAopbY2j3//+99hxpI0pa4hMVQUJG3179+fmpq5wIpgzau0abONHj16NPcxyXGa\n5a11VBQkbQ0cOJCbbrqR9u0PoLj4IDp0OJVnn52kU0eyQ+oa4qcxBUl7q1atYuXKley999506dIl\n7DiSYXJ5rEGXpIqINCEX72tQURARaUaudQ0qCiIiLZArXYOKgohIC+VC16CiICISo2zuGlQURETi\nkK1dg4qCiEgrZFvXoKIgItJK2dQ1qCiIiCRINnQNGflAPDMbZmZLzOwTM7su7DwiIpC7z1AKtVMw\nszzgE+AYYBXwLnCGuy9psJ06BREJTaZ2DZnYKRwCLHX35e5eAzwFnBxyJhGRb8ilriHsotAb+Kze\n8spgnYhIWmnuyav73rcvGyo3hJgucfLDDtBSY8eOjf5cWlpKaWlpaFlEJHdt7xrqX6E0ctBIuhSG\n/wTf8vJyysvLW7WPsMcUDgPGuvuwYPl6wN39Dw2205iCiKSd6Z9O55Y3buGFs16gY9uOYcf5loy7\nJNXM2gAfExlo/gKYA5zp7h812E5FQUQkRvEUhVBPH7n7NjO7DJhBZHxjQsOCICIiqaOb10REslQm\nXpIqIiJpREVBRESiVBRERCRKRUFERKJUFEREJEpFQUREolQUREQkSkVBRESiVBRERCRKRUFERKJU\nFEREJEpFQUREolQUREQkSkVBRESiVBRERCRKRUFERKJUFEREJEpFQUREolQUREQkSkVBRESiVBRE\nRCRKRUEWAAaAAAAInUlEQVRERKJCKwpmdquZfWRm881sipkVh5VFREQiwuwUZgD7uPsBwFLghhCz\nJFV5eXnYEVolk/NncnZQ/rBlev54hFYU3P0ld68LFt8B+oSVJdky/S9WJufP5Oyg/GHL9PzxSJcx\nhQuBaWGHEBHJdfnJ3LmZzQR2qb8KcOBGd/97sM2NQI27T0pmFhER2TFz9/AObnY+cDFwtLtXNbNd\neCFFRDKYu1ss2ye1U2iOmQ0DrgWOaq4gQOx/KBERiU9onYKZLQXaAuuDVe+4+y9CCSMiIkDIp49E\nRCS9pMvVRzuUiTe7mdkwM1tiZp+Y2XVh54mFmfUxs1lmtsjMPjSzX4WdKR5mlmdm75vZ82FniZWZ\ndTazsuDv/SIzOzTsTLEws6vMbKGZfWBmfzWztmFnao6ZTTCzNWb2Qb11Xcxshpl9bGYvmlnnMDM2\npYnscX1nZkxRIMNudjOzPOBu4EfAPsCZZjYg3FQxqQWudvd9gMOBX2ZY/u2uABaHHSJO44H/cfeB\nwP7ARyHnaTEz6wVcDgxx98FExi/PCDfVDj1K5P/X+q4HXnL37wCzSN/vncayx/WdmTFFIQNvdjsE\nWOruy929BngKODnkTC3m7qvdfX7w82YiX0i9w00VGzPrAwwHHg47S6yC3+qOdPdHAdy91t2/DjlW\nrNoAHcwsHygCVoWcp1nu/gawocHqk4HHg58fB36c0lAt1Fj2eL8zM6YoNJAJN7v1Bj6rt7ySDPtS\n3c7MdgcOAGaHmyRmdxC5wi0TB872ANaZ2aPB6a8Hzaww7FAt5e6rgNuBFcDnwFfu/lK4qeLSw93X\nQOQXJaBHyHni1eLvzLQqCmY2Mzj/uP31YfDPE+tto5vdUsjMOgLPAFcEHUNGMLMTgDVBt2PBK5Pk\nA0OAe9x9CFBB5FRGRjCznYj8lt0P6AV0NLOzwk2VEBn3C0as35mh3afQGHc/rrn3g5vdhgNHpyRQ\n63wO7FZvuU+wLmMEbf8zwF/cfWrYeWL0PeAkMxsOFAKdzGyiu58bcq6WWgl85u5zg+VngEy6WOFY\n4F/u/iWAmT0LfBfItF/m1pjZLu6+xsx2Bf4ddqBYxPOdmVadQnPq3ex20o5udksT7wJ7mVm/4KqL\nM4BMuwLmEWCxu48PO0is3H20u+/m7iVE/t3PyqCCQHDK4jMz6x+sOobMGjBfARxmZu3NzIjkz4SB\n8oZd5fPA+cHP5wHp/MvRN7LH+52ZMfcpZOLNbsF/lPFEiu8Ed//vkCO1mJl9D3gN+JBIy+zAaHef\nHmqwOJjZ94Fr3P2ksLPEwsz2JzJIXgD8C7jA3TeGm6rlzGwMkYJcA8wDLgouukhLZjYJKAW6AWuA\nMcBzQBnQF1gOjHT3r8LK2JQmso8mju/MjCkKIiKSfBlz+khERJJPRUFERKJUFEREJEpFQUREolQU\nREQkSkVBRESiVBQkbZjZjcGjlhcEz/s5OI599DOzD2P8zKNmdkqsx2oNM3s5eIQIZvZGCo97gpmN\nS9XxJPOoKEhaMLPDiNyOf4C770/kMQmfNf+pJqX1zTfBozfmb3+WlLsfkYRjNPr/tru/AIwws/aJ\nPqZkBxUFSRc9gXXuXgvg7l8GT6XEzA42szeDyULeMbMOQUfwmpnNDV6HNdxhMMHOrWY2O/jsxfXe\nuzuYgGQGTTz50swuMrM5ZjYvmOymfbC+xMzeDjqa35vZpnqf+XXwmfnBHb2NOZt6j0vY/nkz+76Z\nvVJvYp2/NJKpxMzeq7e81/ZlM/tfM/tvM5sLnGpml1tkcp75wR2v25UDI5rIJjlORUHSxQxgN4vM\nVHePmR0FYGYFROaiuDyYLORYoJLIrfzHuvtQIo9S+HMj+/w5kUc2H0pkfotRQTH5CbB3MHnNeUQe\n1NaYKe5+iLsfCCwJ9geRR5fcEXQ0Kwk6EzM7LtjvIcCBwFAza6wL+B7wXr3l+p3NAcCvgEHAnmb2\njWzu/i/gKzMbHKy6AJhQb5N17j7U3Z8m8lTVA4J/b5fU2+Y94Mgm/syS41QUJC24+xYij4oeBawF\nnjKzc4HvAKvc/f1gu83BxCFtgYctMv1gGTCwkd3+EDjXzOYRmQuiK7A3cBTwZLC/L4jMqNWY/YJu\n5APgLCIz6EFkJrpngp/r/wb+Q+A4M3sfeD/Ivncj++0S/HkbM8fdv/DI82fmA7s3ss0E4ILgFNHp\nDTJMrvfzAmCSmZ0NbKu3/t9EHmct8i1p9ehsyW3BF+FrwGvBYPG5RL5cG5sL4SpgtbsPNrM2RLqH\nhoxIhzHzGysjcy20xGNEnjC50MzOA76/PWqDY9T/+RZ3f2gH+61t5r36T7PcRuP/j04h8sCzV4C5\nDR7QVr/YnECkAJ4E3Ghm+wYFtT2N//sSUacg6cHM+pvZXvVWHUDkqZQfA7ua2UHBdh2DItAZ+CLY\n9lwiUz829CLwi2BeCMxsbzMrIlJ4Tg/GHHoCP2giVkdgdXAK6+x6698BTg1+rj/v8IvAhWbWIThe\nLzPr3sh+Pzazkvp//CaO36jgMcgvAvcRmZv3W4LHVe/m7q8SOY1UHPx5APoDC2M5puQOdQqSLjoC\nfzazzkR+k/4UGOXuNWZ2OnC3RaajrCAyrnAvMCU4xTSdb/6GvN3DRE6/vB98Sf4b+LG7/83MjgYW\nEXnu/1tNZPotMCf43GygU7D+KuAJMxtN5Mt5I4C7zzSzAcDbkcOxCfgZkdNh9b1ApBD9K1hu6mqp\n5q6i+iuR+YJnNLF9myBjMZGiM77eHM8/IINmcZPU0qOzRWJkZoXuXhn8fDpwhrv/JIbP7wo87u4/\nakWGa4Bid2/qCqemPtcD+OuOZjmU3KVOQSR2B5nZ3UR+A99AZFL0FnP31Wb2kJl1jGfea4tMbVlC\nfNPS7gZcE8fnJEeoUxARkSgNNIuISJSKgoiIRKkoiIhIlIqCiIhEqSiIiEiUioKIiET9fxRgGILG\nUlSlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1d828c92908>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Model parameters\n",
    "print(mydict['b'].value)\n",
    "\n",
    "bias_vector   = mydict['b'].value\n",
    "weight_matrix = mydict['w'].value\n",
    "\n",
    "# Plot the data \n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# given this is a 2 class \n",
    "colors = ['r' if l == 0 else 'b' for l in labels[:,0]]\n",
    "plt.scatter(features[:,0], features[:,1], c=colors)\n",
    "plt.plot([0, bias_vector[0]/weight_matrix[0][1]], \n",
    "         [ bias_vector[1]/weight_matrix[0][0], 0], c = 'g', lw = 3)\n",
    "plt.xlabel(\"Scaled age (in yrs)\")\n",
    "plt.ylabel(\"Tumor size (in cm)\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Exploration Suggestions** \n",
    "- Try exploring how the classifier behaves with different data distributions - suggest changing the `minibatch_size` parameter from 25 to say 64. Why is the error increasing?\n",
    "- Try exploring different activation functions\n",
    "- Try exploring different learners \n",
    "- You can explore training a [multiclass logistic regression][] classifier.\n",
    "\n",
    "[multiclass logistic regression]: https://en.wikipedia.org/wiki/Multinomial_logistic_regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}