code/ch02_perceptron/ch02_perceptron.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Accompanying code examples of the book \"Introduction to Artificial Neural Networks and Deep Learning: A Practical Guide with Applications in Python\" by [Sebastian Raschka](https://sebastianraschka.com). All code examples are released under the [MIT license](https://github.com/rasbt/deep-learning-book/blob/master/LICENSE). If you find this content useful, please consider supporting the work by buying a [copy of the book](https://leanpub.com/ann-and-deeplearning).*\n",
    "\n",
    "Other code examples and content are available on [GitHub](https://github.com/rasbt/deep-learning-book). The PDF and ebook versions of the book are available through [Leanpub](https://leanpub.com/ann-and-deeplearning)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ch02 - The Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hands-on Section"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Table of Contents\n",
    "* [Preparing the Dataset](#Preparing-the-Dataset)\n",
    "* [Implementing a Perceptron in NumPy](#Implementing-a-Perceptron-in-NumPy)\n",
    "* [Implementing a Perceptron in TensorFlow](#Implementing-a-Perceptron-in-TensorFlow)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka 2017-03-31 \n",
      "\n",
      "tensorflow 1.0.1\n",
      "numpy 1.12.1\n",
      "matplotlib 2.0.0\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -d -p tensorflow,numpy,matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import os\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Load dataset from tab-seperated text file\n",
    "- Dataset contains three columns: feature 1, feature 2, and class labels\n",
    "- Dataset contains 100 entries sorted by class labels, 50 examples from each class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class label counts: [50 50]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEKCAYAAAASByJ7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHddJREFUeJzt3X9wXfV55/H3gxCViMFObLVgyamdhnUTwMaDQFBPoGta\nDAQwgTDdkJA62YRtdvkRfpgBnOkIQqmpaRPSJEvskLJ2TLP1wrppKHGTuISWDDQSBoPCOiaUHxJk\n45gVOFiOZfnZP+6VkeR7r47uPb/P5zXjGevo6t7nCHOe8/1+n+d7zN0RERE5LOkAREQkHZQQREQE\nUEIQEZEyJQQREQGUEEREpEwJQUREACUEEREpU0IQERFACUFERMoOTzqAqZg1a5bPnTs36TBERDKl\nt7f3l+7eNtnrMpUQ5s6dS09PT9JhiIhkipm9FOR1mjISERFACUFERMqUEEREBMjYGkIlw8PD9Pf3\ns3fv3qRDSY2WlhY6Ojpobm5OOhQRyZDMJ4T+/n6OOuoo5s6di5klHU7i3J1du3bR39/PvHnzkg5H\nRDIk8SkjM2sys61m9p16fn7v3r3MnDlTyaDMzJg5c6ZGTCIyZYknBOAa4LlG3kDJYDz9PkSkHolO\nGZlZB/BB4M+A65KMRSRX7miHfb869PgR0+CWgfjjkUxIeg3hi8CNwFEJxxG67u5upk2bxg033BD6\ne/f29rJ8+XKGhoY477zzuPvuuzUqkPEqJYNKx6eSOJRkci+xKSMzOx/4hbv3TvK6K8ysx8x6du7c\nGVN06faZz3yGtWvXsmPHDnbs2MF3v/vdpEOSrAqaOCZ7bff0UsKQTEtyDWExcKGZvQh8C1hiZt+c\n+CJ3X+Pune7e2dY26VYck9q0dYDFq7Yw76aHWLxqC5u2Nn5ns27dOhYsWMDChQu5/PLLD/n+2rVr\nOeWUU1i4cCGXXHIJe/bsAWDjxo2ccMIJLFy4kDPOOAOAvr4+Tj31VE466SQWLFjAjh07xr3Xa6+9\nxptvvslpp52GmfHxj3+cTZs2NXwOIg2rljAkMxKbMnL3m4GbAczs94Eb3P1jUX7mpq0D3PzgMwwN\njwAwMDjEzQ8+A8BFi+q7u+nr6+P222/nRz/6EbNmzeL1118/5DUXX3wxn/70pwH43Oc+x7333stV\nV13FbbfdxubNm2lvb2dwcBCAe+65h2uuuYaPfvSj7Nu3j5GRkXHvNTAwQEdHx8GvOzo6GBjQcF1E\nGpeGKqPYrN68/WAyGDU0PMLqzdvrfs8tW7Zw6aWXMmvWLADe9a53HfKaZ599lg984AOceOKJbNiw\ngb6+PgAWL17M8uXLWbt27cEL/+mnn84dd9zBnXfeyUsvvURra2vdsYmITEXSi8oAuPsjwCNRf86r\ng0NTOh6W5cuXs2nTJhYuXMh9993HI488ApRGA0888QQPPfQQJ598Mr29vVx22WV0dXXx0EMPcd55\n5/G1r32NJUuWHHyv9vZ2+vv7D37d399Pe7vmbmWCI6ZVXwCOmxajMyMVCSEus2e0MlDh4j97Rv13\n4UuWLOFDH/oQ1113HTNnzuT1118/ZJSwe/dujj32WIaHh9mwYcPBC/jPfvYzurq66Orq4uGHH+aV\nV17hjTfe4D3veQ9XX301L7/8Mtu2bRuXEI499liOPvpoHn/8cbq6uli3bh1XXXVV3fFLTgW90E4l\ncVR77WSmsnAtiSpUQlixdP64NQSA1uYmViydX/d7Hn/88axcuZIzzzyTpqYmFi1axH333TfuNZ//\n/Ofp6uqira2Nrq4udu/eXYpnxQp27NiBu3PWWWexcOFC7rzzTtavX09zczPHHHMMt9xyyyGf+dWv\nfvVg2em5557LueeeW3f8UnBTuUMffW2tO37JNHP3pGMIrLOz0yc+IOe5557jfe97X+D32LR1gNWb\nt/Pq4BCzZ7SyYun8uheU02yqvxeRyHRPr/G9N+KLo8DMrNfdOyd7XaFGCFCqJspjAhAJheb7C61w\nCUFEatB8f3gymFwLVXYqIgmotraQ9zWHDCZXjRBEJFopvRuWQykhiEh6ZHCapW4pPFdNGYlIemRw\nmqVuKTxXJYSIdHd3c9ddd0Xy3itXrmTOnDlMm5bzOViJX1Hn+wXQlFEmXXDBBVx55ZUcd9xxSYci\neZO3aZkkpWn7kICKlRAimrNbt24dd911F2bGggULWL9+/bjvr127ljVr1rBv3z7e+973sn79eo48\n8kg2btzIrbfeSlNTE9OnT+fRRx+lr6+PT3ziE+zbt48DBw7wwAMPHHLhP+200+qOVURiksHkWqyE\nEMGcXdzbX4uEIoULmpI8rSE0SNtfSyalcEETKNYaRgrPtVgjhISEuf21SK4VaXSSwnPVCKFBS5Ys\nYePGjezatQug4pTRxO2vR41uf33bbbfR1tbGK6+8wgsvvHBw++tly5axbdu22M5FRIpNI4QGJbH9\n9Y033sj999/Pnj176Ojo4FOf+hTd3d0xnK2kmtYFpEHF2v66QP/DaPvrAprKNtNZ35I6zf8vNxpb\nBOem7a8rSfofikhaZLBGfpy0LorXiiFobAmeW7ESgkhR3dE+/oYozzdHE0c/aRg1ZEQuEoK7Y2ZJ\nh5EaWZoGlJjEceec1mmcNIwaMiLzVUYtLS3s2rVLF8Eyd2fXrl20tLQkHYoUTZqncSSQzI8QOjo6\n6O/vZ+fOnUmHkhotLS10dHQkHYbErdq6gEhAmU8Izc3NzJs3L+kwRJJ3y0Dt6qGpTumkdQpoNIa0\nJr9GF+wTXPDPfEIQkYCmOqUTxxRQvUmn0vdqJcM4NZosE0y2mV9DEJExUrg/Tk1hJp2snXsKaYQg\nkie17i4nu4OudrceVNK9DUlPY+WAEoKIlDQ6FaQLcuYpIYhkSZoXeiXzlBBEsqSROfewpnTSNCev\nBBkqJQSRomjkAhnVhneNJik1w4VKCUFEkhP1XfxkIwiNMMZRQhDJi0YvbklXCUVhshGERhjjKCGI\n5EWjF7cC3hHLeGpME8kSNV9JhDRCEMmSRhrP8iiP01wJUkIQkexqZJrrjvbw4sgJJQSRIqg0eihC\nJU2tXVH3/UojjAkSSwhmNgdYB/wW4MAad787qXhEMm+qW0LXem2tiqVqP5vGBDPZluBpizdhSY4Q\n9gPXu/uTZnYU0Gtm33P3nyQYk0h2Vbu41bO2UE/FUkFLNfMksSojd3/N3Z8s/3038BygST0RkYSk\nouzUzOYCi4AnKnzvCjPrMbMePSZTRCQ6iS8qm9k04AHgs+7+5sTvu/saYA1AZ2enxxyeSPGktfpG\n20xELtERgpk1U0oGG9z9wSRjEcmtqTazpXUtoNa6Rvf09CayDEmyysiAe4Hn3P2vkopDJPfCvHue\nrMooSWlNZBmS5JTRYuBy4Bkze6p87BZ3/8cEYxKRWjQ1k2uJJQR3/1fAkvp8EREZLxVVRiIikjwl\nBBEZL607qtb7+Wk9nxRKvOxUREJQrSSzksnKNNO6TjAa12TbalT7OZmUEoJIHoS1h1EW6AIfGSUE\nkZBs2jrA6s3beXVwiNkzWlmxdD4XLVJtvGSHEoJICDZtHeDmB59haHgEgIHBIW5+8BkAJQXJDCUE\nkRCs3rz9YDIYNTQ8wurN25UQkqBtLuqihCASglcHh6Z0PHWiuoAmdWGuZ/tuUdmpSBhmz2id0vHQ\nTaWEstJro7qA6sKcKRohiIRgxdL549YQAFqbm1ixdH48AcQ9DaIpmVxSQhAJweg6QWGqjHTnn0tK\nCCIhuWhRez4TQPd03fkXhNYQRGRyWbvz13YVddEIQURKF8ooLvrV3jfqC7NGM3VRQkiAOloldUYv\noN3To3lfyQQlhJipo1VyIak7f4mUEkLM1NEqU5XKEWXW7/xVNluREkLMMt/RKrGKfUSZpjv/WtNX\n3W809t4qm61ICSFms2e0MlDh4h9bR6tkSuwjygLfHYsSQuwS72jNmFROl8QoNyNKTdFkghJCzArX\n0doALcDnaESpKZpMUEJIQG47WkOmBXiNKCVeSgiSWrmZLmmARpQRSdPieYooIUhq5Wa6pEGpHlFm\ndW0g6tgy+ntRQpDU0nRJBkS5NtBoaWmSMrpmUnVzOzObY2bfMrN/MbNbzKx5zPc2xROeFNlFi9r5\n84tPpH1GKwa0z2jlzy8+Mb13y1KdNpvLhFojhG8ADwCPA/8Z+KGZXeDuu4DfjiM4kVRPl0hwKZ4m\nkbfVSght7n5P+e9XmdnHgEfN7ELAow9NREKX0bltiUethNBsZi3uvhfA3b9pZj8HNgPviCU6EQlX\nRue2JR61EsLXgS7gh6MH3P37ZnYp8BdRByYi46WyazvJ8s00j3YyWtZaNSG4+xeqHN8K/GFkEYkE\nkMqLY4RS27Wd5IU3zaOdpBNSnfQITcmc0YvjwOAQztsXx01bs/k/YRC1urZFwqKEIJlTxIujurYl\nDkoIkjlFvDhW686ecte2+gGkhkk7lc3st4A7gNnufq6ZvR843d3vjTw6kQqKuKVFaF3bGZ3blngE\nGSHcR6nUdHb5658Cn40qIJHJrFg6n9bmpnHH8r6lhbq2K9BoJ3RB9jKa5e5/Z2Y3A7j7fjMbmeyH\nRKJS1B1A1bU9gUY7oQuSEN4ys5mUu5PN7DQglF2nzOwc4G6gCfi6u68K430l/3RxFAlfkIRwHfBt\n4HfM7DGgDfhwox9sZk3AVyj1NPQDPzazb7v7Txp9b0mHovUKTIV+N5JGNROCmR0GtABnAvMBA7a7\n+3AIn30q8Ly7v1D+rG8BywAlhBxIbSNVCuh3I2lVc1HZ3Q8AX3H3/e7e5+7PhpQMANqBV8Z83V8+\nJjlQxF6BoPS7kbQKUmX0AzO7xMws8mgqMLMrzKzHzHp27tyZRAhShyL2CgSl342kVZCE8F+AjcCv\nzexNM9ttZm+G8NkDwJwxX3eUj43j7mvcvdPdO9va2kL4WIlDaI1UOaTfjaTVpIvK7n5URJ/9Y+A4\nM5tHKRH8J+CyiD6r0JJYwNTjL6vLzO8mzbuJSiSCdCqfUem4uz/ayAeX+xmupNT01gR8w937GnlP\nOVRSC5hF7RUIIjO/mzTvJiqRMPfaDz8zs38Y82ULpeqgXndfEmVglXR2dnpPT0/cH5tpi1dtqbjN\nQ/uMVh67Kfb/hJFQCWdEuqfX+F4orUgSEzPrdffOyV4XZMrogglvPAf4YgOxSYzyvoCpEk6R8NSz\n22k/8L6wA5Fo5H0BUyWcIuEJsobw15S3raCUQE4CnowyKGnM2CmU6a3NNDcZwyNvTw2mcgGzTnkf\nAYnEKcjWFWMn7fcDf+vuj0UUjzRo4hTK4NAwzYcZ7zyymcE9w7mbYy/iVtixyehzgaV+QRLCDHe/\ne+wBM7tm4jFJh0pTKMMHnCOPOJytf3p2QlFFJzMlnFmk0tLCCbKG8McVji0POQ4JSdGmUPScAJHw\nVB0hmNlHKDWKzTOzb4/51lHA61EHJsGNXTM4zIyRCqXEeZ5C0VbYIuGoNWX0I+A1YBbwl2OO7wa2\nRRmUBDdxzaBSMtAUiogEUTUhuPtLwEvA6fGFI1NVac0AoMmMA+65W0SeKjWtiQQXpOz0NOCvKfUe\nHEFpm4m33P3oiGOTAKqtDRxw599XfTDmaNJFTWsiUxNkUfnLwEeAHUAr8ClKTzrLhE1bB1i8agvz\nbnqIxau2sGlrvion8t541gg1rYlMTaBOZXd/Hmhy9xF3/xvgnGjDCsfoHeLA4BDO23eIeUoKK5bO\np7W5adwxrRmUFK3iSqRRQRLCHjM7AnjKzP7CzK4N+HOJK8Idosouq9PoSWRqgjSmXU4pAVwJXEvp\noTaXRBlUWIpyh6iyy8rUtCYyNUF2O33JzFqBY9391hhiCo22NSi2zDx3QCQlglQZXQDcRanCaJ6Z\nnQTc5u4XRh1co3SHKBo9iQQXZMqom9JDcR4BcPenyo+9TL2s3CGqVl5E0iBIQhh29zfMbOyx2o9Z\nS5G03yGqVr46JUqReAVJCH1mdhnQZGbHAVdT2tZCQlCrEqqIF7/RJDAwOITx9p3HxESpZFGdfjdS\nryAJ4SpgJfBr4H5gM3B7lEEVSRoqodJyAZk4Wpo4DB1bMjxxVHXt/3yKnpde5/aLTowz5NTRiFMa\nUbWfwMzWl//6aXdf6e6nlP98zt33xhRf7iVdK5+m5r1q+zKN9ergUMXXObDh8ZcnjTvvnetF6L2R\n6NRqMDvZzGYDnzSzd5rZu8b+iSvAvEu60zhNF5Ago6LZM1qrvs6hZtxpSn5RScOIU7KrVkK4B/gB\n8LtA74Q/PTV+TqYg6U7jNF1AJhsVNR9m7Nm3v2ZFQ62405T8opL0iFOyrdb2118CvmRm/93dPxNj\nTIWTZCVUmpr3KvWNjC4sz2ht5q19+/l/e4ZrvketuNOU/KKi3htpxKR7EikZ5FvSU1ZjVRotfeGP\nTuLFVR/kHb9xOMMjtaudJ4u7CHfPSY84JduCVBlJjqWtea/aaKnWXbxBoLiLcvec9t4bSS8lBIn8\nAhJGWWu1qa32Ga08dtOSQO+RtuQnkjZKCBKpsOriw7q7n2ryG9so12TGiDvtSiSSU5l4roFkV1iV\nPUnMjY8tUwUY8dIaRh7LVUVAIwSJWLW5/0rTP5OJe268VqNckbcXkfzSCKHgou7crVbBY+XPTrPJ\nylHzVK4qAkoIhRZH5+6KpfOxCscn6ypOg8nKUfNUrioCSgiFFkfn7kWL2qt2Fqf9DrtSj8aoPJar\niighFFhcnbvtGW0IG7uQDdBUfiaImr0kr7SoXGBxbVuR5YYwNXlJkWiEUGBxbVuh7RREskEjhAKL\ns3M373faaXnIkEgjlBAKLisX6jRfcKfSjZ3m8xBJJCGY2WrgAmAf8DPgE+4+mEQsaaCLRG1pfyxk\n0OdiN3Ie+jcicUhqDeF7wAnuvgD4KXBzQnEkrghP8WpU2h9sE7Raq97z0L8RiUsiCcHd/8nd95e/\nfBzoSCKONEj7xS4N0v5gm6DPWaj3PPRvROKShiqjTwIPV/ummV1hZj1m1rNz584Yw4pH2i92jQhr\nW4y0P9gmaLVWveeR538jki6RJQQz+76ZPVvhz7Ixr1kJ7Ac2VHsfd1/j7p3u3tnW1hZVuIlJ+8Wu\nXmFOc6TpqW6VBC2rrfc88vpvRNInskVld/+DWt83s+XA+cBZ7l772Yg5lqWmrU1bB7j1H/oOPtd4\nRmsz3RceX3FxM+hCaxBZeLBNkGqtes8jS/9GJNuSqjI6B7gRONPd9yQRQ1pk4WIHpWSw4n89Pe65\nxoNDw6zY+DRwaJVM2NMc1S64Wau+qafMNyv/RiT7LImbczN7HvgNYFf50OPu/ieT/VxnZ6f39PRE\nGptUtnjVlqrPMBh9gtjYC9Zbv97P4NBwxdcGfeTlZCaWcULpzlld0CLjmVmvu3dO9rpERgju/t4k\nPlfqV+vOfnR9YGx9fXOT0XyYMXzg7RuOsKc5wpyWEpF0VBlJBtRawGwyO+TCPDziTGs5PNL9i1R9\nIxIubV0hgaxYOv+QNQTgkFHAWIN7htn6p2dHFlNcu7WKFIVGCBLIRYvaWf3hhbzzyOaDx2a0NrP6\n0oWJPe8g7eWoIlmjEYIAwap1alXIJFEWqeobkXApIUjDm8cleWHOym6tIlmghCChVOvowiySfVpD\nEFXriAigEYJQjGqdrHU0iyRBIwTJfbVOFM8TCGsnV5E0UUKQwLt1ZlXYzxPQA2skrzRlJEC+F4XD\nXiPRlhmSVxohSO6F/TwBLcJLXmmEILk0dhF5emszzU02btuNRtZIirAIL8WkEYLkzsQ5/sGhYXB4\n55HNoayR5H0RXopLIwTJnUpz/MMHnCOPODyUzfa0ZYbklRKC5E4cc/x5XoSX4lJCkNzRHP+h1Jgn\nQWgNQXJHc/zjqW9CglJCkNzJe6PdVIXdmCf5pSkjySXN8b9NfRMSlEYIIjkXdmOe5JcSgkjOaU1F\ngtKUkUjOqW9CglJCkFCorDHdtKYiQSghSMMafSZzEShhShZoDUEaprLG2tQHIFmhhCANU1ljbUqY\nkhVKCNIwlTXWpoQpWaGEIA1TWWNtSpiSFUoI0jBtFVGbEqZkhaqMCiLqKheVNVanPgDJCiWEAlBZ\naPKUMCULlBAKoFaViy5SMhXqp8g3JYQCUJWLhEEjzfzTonIBqMplvE1bB1i8agvzbnqIxau2qEEs\nIPVT5J8SQgGoyuVt6hqun0aa+aeEUAAqC32b7nLrp5Fm/iW6hmBm1wN3AW3u/sskY8k7VbmU6C63\nfiuWzh+3hgDFHWnmVWIjBDObA5wNvJxUDFI8usutn0aa+ZfkCOELwI3A3ycYgxSM7nIbo5FmviWS\nEMxsGTDg7k+bWRIhSBV5rzNX17BIdZElBDP7PnBMhW+tBG6hNF0U5H2uAK4AePe73x1afHKootSZ\n6y5XpDJz93g/0OxE4AfAnvKhDuBV4FR3/3mtn+3s7PSenp6IIyyuxau2MFBhcbV9RiuP3bQkgYhE\nJAxm1uvunZO9LvYpI3d/BvjN0a/N7EWgU1VGyVMFjkixqQ9BDlIFjkixJZ4Q3H2uRgfpoI5mkWLT\n5nZykCpwRIpNCUHGUQWOSHElPmUkIiLpoBGC5L4ZTUSCUUIouKI0o4nI5DRlVHDaDlpERikhFJya\n0URklBJCwakZTURGKSEUnJrRRGSUFpULTs1oIjJKCUHUjCYigKaMRESkTAlBREQAJQQRESlTQhAR\nEUAJQUREymJ/pnIjzGwn8FLScURkFlDUBwUV9dyLet6gc4/73H/b3dsme1GmEkKemVlPkIdg51FR\nz72o5w0697Seu6aMREQEUEIQEZEyJYT0WJN0AAkq6rkX9bxB555KWkMQERFAIwQRESlTQkghM7ve\nzNzMZiUdS1zMbLWZ/R8z22Zm/9vMZiQdU5TM7Bwz225mz5vZTUnHExczm2Nm/2xmPzGzPjO7JumY\n4mRmTWa21cy+k3QslSghpIyZzQHOBl5OOpaYfQ84wd0XAD8Fbk44nsiYWRPwFeBc4P3AR8zs/clG\nFZv9wPXu/n7gNOC/FejcAa4Bnks6iGqUENLnC8CNQKEWd9z9n9x9f/nLx4GOJOOJ2KnA8+7+grvv\nA74FLEs4pli4+2vu/mT577spXRwLsfe6mXUAHwS+nnQs1SghpIiZLQMG3P3ppGNJ2CeBh5MOIkLt\nwCtjvu6nIBfFscxsLrAIeCLZSGLzRUo3eweSDqQaPSAnZmb2feCYCt9aCdxCabool2qdu7v/ffk1\nKylNK2yIMzaJl5lNAx4APuvubyYdT9TM7HzgF+7ea2a/n3Q81SghxMzd/6DScTM7EZgHPG1mUJoy\nedLMTnX3n8cYYmSqnfsoM1sOnA+c5fmuhx4A5oz5uqN8rBDMrJlSMtjg7g8mHU9MFgMXmtl5QAtw\ntJl9090/lnBc46gPIaXM7EWg090LsQGYmZ0D/BVwprvvTDqeKJnZ4ZQWzs+ilAh+DFzm7n2JBhYD\nK93t/A/gdXf/bNLxJKE8QrjB3c9POpaJtIYgafFl4Cjge2b2lJndk3RAUSkvnl8JbKa0qPp3RUgG\nZYuBy4El5f/OT5XvmiUFNEIQERFAIwQRESlTQhAREUAJQUREypQQREQEUEIQEZEyJQQpFDO72sye\nM7Mpd0Kb2VwzuyyKuMrvf4aZPWlm+83sw1F9jkg1SghSNP8V+EN3/2gdPzsXmHJCKO9uGsTLwHLg\n/ql+hkgYlBCkMMrNbu8BHjaza83sHWb2DTP7t/Ie9cvKr5trZv9Svlt/0sx+r/wWq4APlJuprjWz\n5Wb25THv/53RfWrM7Fdm9pdm9jRwupmdbGY/NLNeM9tsZsdOjM/dX3T3baR48zPJN+1lJIXh7n9S\n3iLjP7r7L83sDmCLu3+y/ECefytvwPcLSqOIvWZ2HPC3QCdwE2O2HCjvvVTNO4An3P368t49PwSW\nuftOM/sj4M8o7eoqkhpKCFJkZ1PacOyG8tctwLuBV4Evm9lJwAjwH+p47xFKG7gBzAdOoLQtB0AT\n8FoDcYtEQglBisyAS9x9+7iDZt3A/wUWUppW3Vvl5/czftq1Zczf97r7yJjP6XP308MIWiQqWkOQ\nItsMXFXegRMzW1Q+Ph14zd0PUNqIbXRReDelDfhGvQicZGaHlR99emqVz9kOtJnZ6eXPaTaz40M9\nE5EQKCFIkX0eaAa2mVlf+WuArwJ/XF4Q/l3grfLxbcCImT1tZtcCjwH/DvwE+BLwZKUPKT8m88PA\nneX3fAr4vYmvM7NTzKwfuBT4Wjkmkdhot1MREQE0QhARkTIlBBERAZQQRESkTAlBREQAJQQRESlT\nQhAREUAJQUREypQQREQEgP8PzP8v2tm5NVsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10486d2e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = np.genfromtxt('perceptron_toydata.txt', delimiter='\\t')\n",
    "X, y = data[:, :2], data[:, 2]\n",
    "y = y.astype(np.int)\n",
    "\n",
    "print('Class label counts:', np.bincount(y))\n",
    "\n",
    "plt.scatter(X[y==0, 0], X[y==0, 1], label='class 0', marker='o')\n",
    "plt.scatter(X[y==1, 0], X[y==1, 1], label='class 1', marker='s')\n",
    "plt.xlabel('feature 1')\n",
    "plt.ylabel('feature 2')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Shuffle dataset\n",
    "- Split dataset into 70% training and 30% test data\n",
    "- Seed random number generator for reproducibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "shuffle_idx = np.arange(y.shape[0])\n",
    "shuffle_rng = np.random.RandomState(123)\n",
    "shuffle_rng.shuffle(shuffle_idx)\n",
    "X, y = X[shuffle_idx], y[shuffle_idx]\n",
    "\n",
    "X_train, X_test = X[shuffle_idx[:70]], X[shuffle_idx[70:]]\n",
    "y_train, y_test = y[shuffle_idx[:70]], y[shuffle_idx[70:]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Standardize training and test datasets (mean zero, unit variance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mu, sigma = X_train.mean(axis=0), X_train.std(axis=0)\n",
    "X_train = (X_train - mu) / sigma\n",
    "X_test = (X_test - mu) / sigma"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Check dataset (here: training dataset) after preprocessing steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEKCAYAAAAFJbKyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYVPWd5/H3R9I+9EQjE+mN0A0D7jKMlxGJHdAwmxhy\nQZgYDGpGzeqQGxs3Jk42kkXxmTDGZMiSmdkkJkEcHYesiRM3StjFyGiYxIxZLyAGRUIgTrJ0y0RC\nFsWAK5fv/lGnoemu6q7qrjrnVNXn9Tz1UOfU6XO+Vd2cb/3uigjMzMzKdVzWAZiZWX1x4jAzs4o4\ncZiZWUWcOMzMrCJOHGZmVhEnDjMzq0hmiUPSOEn/JOlZSZslXVvkGEn6sqTtkjZJemMWsZqZ2VGv\nyfDaB4FPRcSTkk4ENkh6MCKe7XXMbGBS8pgOfD3518zMMpJZiSMidkbEk8nzvcAWoL3PYXOBlVHw\nKDBK0piUQzUzs16yLHEcIWkCMBV4rM9L7cCOXttdyb6dA51v9OjRMWHChOoFaGbW4DZs2PDriGgr\n59jME4ekE4DvAH8WES8N4zwLgAUA48ePZ/369VWK0Mys8Un6ZbnHZtqrSlILhaRxV0TcW+SQbmBc\nr+2OZF8/EbEiIjojorOtraykaWZmQ5BlryoBtwNbIuKvSxy2Grgq6V11LvBiRAxYTWVmZrWVZVXV\nDOBK4GlJTyX7bgDGA0TEcuB+YA6wHdgHfCCDOM3MrJfMEkdE/DOgQY4J4GPVuN6BAwfo6urilVde\nqcbpGsLIkSPp6OigpaUl61DMrI5k3jielq6uLk488UQmTJhAoZasuUUEu3fvpquri4kTJ2YdjpnV\nkaaZcuSVV17h5JNPdtJISOLkk092CczMKtY0iQNw0ujDn4eZDUXTVFWZWUo+3w6vvtx///EnwA1F\ne9NbnXHiyNiSJUs44YQTuO6666p+7g0bNjB//nz279/PnDlz+NKXvuRShg1NJcmg2HED7be601RV\nVc3m6quv5rbbbmPbtm1s27aNBx54IOuQrF5VKxksOamQhKyuOXGUsGpjNzOWrmPiojXMWLqOVRuH\nX8ReuXIlZ511FlOmTOHKK6/s9/ptt93Gm970JqZMmcLFF1/Mvn37ALjnnns488wzmTJlCm95y1sA\n2Lx5M9OmTePss8/mrLPOYtu2bceca+fOnbz00kuce+65SOKqq65i1apVw34PZsPmkkfdc1VVEas2\ndnP9vU+z/8AhALr37Of6e58G4KKpQ/u2tHnzZm6++WZ+/OMfM3r0aH7zm9/0O2bevHl85CMfAeDG\nG2/k9ttv5+Mf/zg33XQTa9eupb29nT179gCwfPlyrr32Wt7//vfz6quvcujQoWPO1d3dTUdHx5Ht\njo4Ourtdv2xmw+fEUcSytVuPJI0e+w8cYtnarUNOHOvWrePSSy9l9OjRALz+9a/vd8wzzzzDjTfe\nyJ49e3j55ZeZNWsWADNmzGD+/Pm8733vY968eQCcd955fO5zn6Orq4t58+YxadKkIcVllhtuVK8b\nrqoq4vk9+yvaXy3z58/nlltu4emnn+Yzn/nMkTEWy5cv5+abb2bHjh2cc8457N69myuuuILVq1fT\n2trKnDlzWLdu3THnam9vp6ur68h2V1cX7e2uW7YUHH/C0H7Ojep1w4mjiLGjWivaX46ZM2dyzz33\nsHv3boCiVVV79+5lzJgxHDhwgLvuuuvI/p///OdMnz6dm266iba2Nnbs2MFzzz3Hqaeeyic+8Qnm\nzp3Lpk2bjjnXmDFjeN3rXsejjz5KRLBy5Urmzp075PityZVKBsX239ANS16sbTyWKVdVFbFw1uRj\n2jgAWltGsHDW5CGf84wzzmDx4sW89a1vZcSIEUydOpU777zzmGM++9nPMn36dNra2pg+fTp79+4t\nxLNwIdu2bSMiePvb386UKVP4whe+wDe+8Q1aWlo45ZRTuOGGG/pd82tf+9qR7rizZ89m9uzZQ47f\nmtxQqoqOP6F01ZPVNRXmEWwsnZ2d0Xchpy1btnDaaaeVfY5VG7tZtnYrz+/Zz9hRrSycNXnI7Rt5\nVunnYlYzS04a4DWXYGpN0oaI6CznWJc4SrhoantDJgozs+FyG4eZ5UMl7SiWKZc4zCwf3OW2bmS9\n5vgdkl6Q9EyJ18+X9KKkp5LHn6cdo5mZHSvrEsedwC3AygGO+VFEvDudcMysLniwYKYyLXFExMNA\n/wENZmYD8WDBTNVD4/ibJW2S9D1JZ5Q6SNICSeslrd+1a1ea8Q3LkiVL+OIXv1iTcy9evJhx48Zx\nwgluXDSz6sl74ngSGB8RZwFfAUpO7xoRKyKiMyI629raUgswzy688EIef/zxrMMwswaT68QRES9F\nxMvJ8/uBFkmja37hz7cXBiP1fQxzHYE0p1UHOPfccxkzZsywYjYz6yvrxvEBSToF+FVEhKRpFBLd\n7ppfuAb1p2lPq25mViuZJg5J3wLOB0ZL6gI+A7QARMRy4BLgakkHgf3AZVGnc6R4WnWzKvI8WJnK\nNHFExOWDvH4Lhe66TWH+/PmsWrWKKVOmcOedd/KDH/wAKJQuHnvsMdasWcM555zDhg0buOKKK5g+\nfTpr1qxhzpw53HrrrcycOTPbN2CWFne5zVSu2zgaSdrTqpuZ1Uqu2zgaSRbTqn/605/mm9/8Jvv2\n7aOjo4MPf/jDLFmyJIV3a2Y1l+EgSE+rXkwTjUr1tOrW0PL8f3m4sVV5GnpPqz5cWf9BmVl15HmE\neZ5jG4TbOMzMrCJNVeKICCRlHUZuNGI1peVcnquOrGxNU+IYOXIku3fv9s0yERHs3r2bkSNHZh2K\nNZM6rp6xo5qmxNHR0UFXVxf1NAFirY0cOZKOjo6sw7C8qbRU4FJENjIcBNk0iaOlpYWJEydmHYZZ\n/lVaKshzKSLPI8yHG1uGSblpEoeZNaE8l3jyHNsgmqaNw8zMqsMlDjOrTKk2jXLkuerIyubEYWaV\nGU7bRR1Xz9hRThxmjWg4PZ2qVSpwKaJhOXGYNaLh9HQaTqlgCHMkWf3JtHFc0h2SXpD0TInXJenL\nkrZL2iTpjWnHaGZmx8q6xHEnhYWaVpZ4fTYwKXlMB76e/GtmVj0exFiRTEscEfEw0H9Fo6PmAiuj\n4FFglKQx6URnZkWVaruo5zaNPA9izKGsSxyDaQd29NruSvbtzCYcM/M3cGuYAYCSFkhaL2m956Oy\npteIpQLLjbyXOLqBcb22O5J9/UTECmAFFFYArH1oZjnmUoHVUN5LHKuBq5LeVecCL0aEq6nMLD2f\nb886gtzJtMQh6VvA+cBoSV3AZ4AWgIhYDtwPzAG2A/uAD2QTqVmDa/ZeRaUGPYIbyIvINHFExOWD\nvB7Ax1IKx6x5VfumWW+J6IZuWHJS1lHUjbxXVZlZPfK394bmxGFmZhVx4jAzs4rkvTuumeXVQO0Y\nWRpq+4rXCimbE4eZDe2mmdd2jIHiWnJS6QSSx0b7nHLiMLPq3zTz/O0968TWAJw4zKz6/O29oblx\n3MzMKuLEYWZmFXHiMLOhyesMvFlfvwm4jcOsmVRzKpC8tmP0xJXX7sINwInDrJnktQttLeQ1sTUA\nJw6zlK3a2M2ytVt5fs9+xo5qZeGsyVw01VN3W/1w4jBL0aqN3Vx/79PsP3AIgO49+7n+3qcBnDys\nbjhxmKVo2dqtR5JGj/0HDrFs7dbmThxZTcNeb9O/54QTh1mKnt+zv6L9uVOrG21WbS/N1OZTRZl2\nx5V0gaStkrZLWlTk9fMlvSjpqeTx51nEaVYtY0e1VrS/6obbhdY3WiPDEoekEcBXgXcCXcATklZH\nxLN9Dv1RRLw79QDNamDhrMnHtHEAtLaMYOGsyekEkHb1i6uCGlKWJY5pwPaIeC4iXgXuBuZmGI9Z\nzV00tZ2/nPeHtI9qRUD7qFb+ct4fNkb7xpKTComiN5dQGlKWbRztwI5e213A9CLHvVnSJqAbuC4i\nNhc7maQFwAKA8ePHVzlUs+q5aGp7YySKYpwQmkLepxx5EhgfEWcBXwFWlTowIlZERGdEdLa1taUW\noJlVQVbTl+R12pScy7LE0Q2M67Xdkew7IiJe6vX8fklfkzQ6In6dUoxm1lupdTaGK6v2DrezDEmW\nieMJYJKkiRQSxmXAFb0PkHQK8KuICEnTKJSQdqceaUY8wthyp+dGu+SkbOOwTGWWOCLioKRrgLXA\nCOCOiNgs6aPJ68uBS4CrJR0E9gOXRURkFXOaPMLYKpXLLxp5XgnQhkyNeB/u7OyM9evXZx3GsMxY\nuo7uIoPC2ke18siimRlEZHnW94sGFLr51qzHVp662dYyljy9zxqTtCEiOss51iPHc6ruRxhbqlKf\nyiRPN81advl1d+KinDhyauyo1qIljtRGGFtdaZgvGk30Db+e5b07btNaOGsyrS0jjtmX6gjjOrNq\nYzczlq5j4qI1zFi6jlUbm+smk/lUJtXib/h1wYkjpxp6hHGV9dTvd+/ZT3C0I0EzJQ9/0bA0uaoq\nxxp6hHEVearyoz3tcteryhqSE4fVvYap3x8mf9GoAXcnLsqJw+qeOxLUgVo2etfy5u4G+aKcOKzu\nZT5VuQ2u3EbvoSQB39xTVzJxSBoHLKMwi+33gGURcSB5bVVEXJROiGYDc/1+A3ESqAsDlTjuAL4D\nPAp8CPihpAsjYjfwe2kEZ1Yu1++bpWegxNGWzBcF8HFJ/wF4WNJ7gMabp8TMzMoyUOJokTQyIl4B\niIj/LulfKUxK+NpUojMzGy6PRq+6gQYA/i19VuSLiIeAS4FnahmUmTWYLBdM8mj0qitZ4oiIvymx\nfyPwzppFZFZFuZxqvBn5m31DcXdca1he08SsNjKdq0rSBZK2StouaVGR1yXpy8nrmyS9MYs4rT4N\nNBWJmQ1dZolD0gjgq8Bs4HTgckmn9zlsNjApeSwAvp5qkFbXPBWJWW0MmjgkvUHS7ZK+l2yfLulD\nVbj2NGB7RDwXEa8CdwNz+xwzF1gZBY8CoySNqcK1rQk0zFTjNjxZNsw3qHLaOO4E/g5YnGz/DPgH\n4PZhXrsd2NFru4s+vbhKHNMO7Bzmta0JeCoSA9wwXwPlVFWNjohvA4cBIuIgcGjgH0mfpAWS1kta\nv2vXrqzDsRzwmiZmtVFOieO3kk4mGS0u6VzgxSpcuxsY12u7I9lX6TEARMQKYAVAZ2enR7bXkVp2\nmfVUJGbVV07i+M/AauDfSnoEaAMuqcK1nwAmSZpIIRlcBlzR55jVwDWS7qZQjfViRLiaqoG4y+zA\nPA7F8mjAxCHpOGAk8FZgMiBga88sucMREQclXUNhCpMRwB0RsVnSR5PXlwP3A3OA7cA+4APDva7l\ni1fvK81J1fJqwMQREYclfTUipgKbq33xiLifQnLovW95r+cBfKza17X8cJfZ0pxULa/KaRz/vqSL\nJanm0VjTcZfZ0pxULa/KSRz/EbgH+H+SXpK0V9JLNY7LmsTCWZNpbRlxzD53mS1wUrW8GjRxRMSJ\nEXFcRBwfEa9Ltl+XRnDW+NxltjQnVcurQXtVSXpLsf0R8XD1w7Fm5C6zxXlJXMurcrrjLuz1fCSF\nqUI2ADNrEpFZjdRj11YnVcujQRNHRFzYe1vSOOC/1Swisxpw11az6hnK7LhdwGnVDsSsljzFuln1\nlNPG8RWS6UYoJJqzgSdrGZSlox6rbobKXVvNqqecNo71vZ4fBL4VEY/UKB5LSbNV3Ywd1Up3kSTh\nrq1mlSunqmpURPx98rgrIh6RdG3NI7OaaraqG3dtNauechLHnxbZN7/KcVjKmq3qxuNFzKqnZFWV\npMspzFY7UdLqXi+dCPym1oFZ9fVu0zhO4lD0n32+katu3LXVrDoGauP4MYWV9kYDf9Vr/15gUy2D\nsurr26ZRLGm46sbMylEycUTEL4FfAuelF47VSrE2DYAREocjGr5XlZlVTzndcc8FvkJh7MbxFNbO\n+K3nq6ovpdouDkfwL0v/OOVozKyeldMd9xYKq/PdA3QCVwG/X8ugstDoYxrcHdXMqqWskeMRsR0Y\nERGHIuLvgAuGc1FJr5f0oKRtyb+/W+K4X0h6WtJTktYXO6Yaeur/u/fsJzg6pmHVxqLLm9cld0c1\ns2opJ3Hsk3Q88JSk/yrpk2X+3EAWAd+PiEnA95PtUt4WEWdHROcwr1lSM4xpcHdUM6uWcqqqrqSQ\nKK4BPgmMAy4e5nXnAucnz/8e+AHwX4Z5ziFrljEN7o5qZtVQzuy4v5TUCoyJiL+o0nXfEBE7k+f/\nCryh1OWBhyQdAm6NiBWlTihpAbAAYPz48RUF4/p/M7PyDVrlJOlC4CnggWT77D4DAkv93EOSniny\nmNv7uIgIjk6i2NcfRcTZwGzgY6UWlUrOsyIiOiOis62tbbDwjuH6fzOz8pVTVbWEwuJNPwCIiKck\nTRzshyLiHaVek/QrSWMiYqekMcALJc7Rnfz7gqT7kjiqvvKgV1ozMytfOYnjQES8KKn3vlIlhHKt\npjAH1tLk3+/2PUDSa4HjImJv8vxdwE3DvG5Jrv+vX43eldosb8pJHJslXQGMkDQJ+ASF6UiGYynw\nbUkfojA6/X0AksYCfxsRcyi0e9yXJKzXAN+MiAeGeV0bhjzdoHti6d6zH3H0m0yjTw9fTXn6fVp9\nURSZs+iYA6TfARZT+MYPsBa4OSJeqXFsQ9bZ2Rnr19ds2EdT6jvXFRTagbLo0lsslr7aR7XyyKKZ\nR473DfJYefp9Wj5I2lDusIeSjeOSvpE8/UhELI6INyWPG/OcNKw28jTWpdS8W731dKUeyuDOVRu7\nmbF0HRMXrWHG0nUNNRC0R55+n1Z/BqqqOiepOvqgpJXAsY0cEZ5avYnkaaxLOdc8TmLiojVFp4/v\nuUEW+2bdLCsj5un3afVnoO64yymM6v4DYEOfh+uBmkypMS1ZjHUp55qHIgiKTx8PpW+QzfJNPE+/\nT6s/JRNHRHw5Ik4D7oiIUyNiYq/HqSnGaDmQp7EuxWLpKQ6POLb3X0mlbpDN8k08T79Pqz+DDgCM\niKvTCMTyLU9zXRWL5W/+5Gx+sfSPOTxIZw8Y+AbZLN/E8/T7tPozaK+qeuReVc1rxtJ1RaePKXfB\nKvc2smZVSa+qcsZxmKWiGt1mF86aPKwbv2cRMBucE4flQrV6M1Xjxj+UWQR6D0gckfTkanfSsQbl\nxGG5MFBvpkpvvGlPH9M36fX05GrUrrxmThxWllqPvi7Va6lYe0XeDDQgcajJzyzPhruSnzWBNJbW\nLdVrScn182ywrrqN1pXXzInDBpXGoLiFsyZTbARGJNfPs8G66jZaV14zJw4bVBqD4i6a2l5yrv68\nf2MvNpiuhwfVWSNy4rBBpTUorr1OB9/1HkwHR0eve1CdNSo3jtugSo2NqPY36bSuUwteCMyaSSYl\nDkmXStos6bCkkiMVJV0gaauk7ZIWpRmjHZXW9BSeBsOsPmQy5Yik04DDwK3AdRHRb34QSSOAnwHv\nBLqAJ4DLI+LZwc7vKUcsr7yolOVV7qcciYgtABp4JtNpwPaIeC459m5gLjBo4rDmlecbcyWj4/P8\nPszy3MbRDuzotd0FTM8ollzwzWRgeV+EqdzR8UN9H/77sLTUrI1D0kOSninymFuj6y2QtF7S+l27\ndtXiEplKYxBevcv7Ikzldmseyvvw34elqWaJIyLeERFnFnl8t8xTdAPjem13JPtKXW9FRHRGRGdb\nW9twQs+lvN8U8yDvizCV2615KO/Dfx+WpjyP43gCmCRpoqTjgcuA1RnHlJm83xSHY9XGbmYsXcfE\nRWuYsXTdkL8l530RpnJX3RvK+2jkvw/Ln6y6475XUhdwHrBG0tpk/1hJ9wNExEHgGmAtsAX4dkRs\nziLePMj7TXGoqlnFkvflUMvtbjyU99Gofx+WT14BsE7U08p0qzZ28xf/czP/d98BAEa1trDkPWcU\njbPUin3to1p5ZNHMIV27ERqIK30f9fT3YfmU++64Vrl6WZlu1cZuFv6Pn3Dg0NEvJHv2H2DhPT8B\n+vcKqnYVS6kR3PWWUCodiV4vfx/WGJw46kg9TGuxbO3WY5JGjwOH40hDbe+b20mtLezZf6Df8dWs\nYsl7N91qqYe/D2sMeW4ctzo0UEmh54bduz3jt68epOW4YweCVrtdwj2OzKrLicOqaqCSwgip3w38\nwKHghJGvqen8VO5xZFZdrqqyqlo4a3K/Ng6AluPEgcPFO2Ls2XeAjX/+rprFNHZUa9EGePc4Mhsa\nlzisqi6a2s6yS6bwu7/TcmTfqNYWll06JbP1NvLeTdes3rjEYVU3UCNtFuttuMeRWXU5cVhqsryB\nu8eRWfU4cViqfAM3q39OHGa91NtAQbMsOHGYJZploKDZcLlXlVnCAwXNyuPEYZbwQEGz8jhxmCU8\nNblZeZw4zBIeKGhWHjeOmyU8UNCsPJkkDkmXAkuA04BpEVF01SVJvwD2AoeAg+UuMmI2VB5nYja4\nrEoczwDzgFvLOPZtEfHrGsdjZmZlyiRxRMQWAEmDHWpmZjmT98bxAB6StEHSgoEOlLRA0npJ63ft\n2pVSeGZmzadmJQ5JDwGnFHlpcUR8t8zT/FFEdEv6N8CDkn4aEQ8XOzAiVgArADo7O4sv/GBmZsNW\ns8QREe+owjm6k39fkHQfMA0omjjMzCwdua2qkvRaSSf2PAfeRaFR3czMMpRJ4pD0XkldwHnAGklr\nk/1jJd2fHPYG4J8l/QR4HFgTEQ9kEa+ZmR2VVa+q+4D7iux/HpiTPH8OmJJyaGZmNgiPHLdUeb0L\ns/rnxGGp8XoXZo0ht43j1ni83oVZY3DisNR4vQuzxuDEYanxehdmjcGJw1Lj9S7MGoMbxy01Xu/C\nrDE4cViqvN6FWf1zVZWZmVXEicPMzCrixGFmZhVx4jAzs4q4cdysCM+pZVaaE4dZH55Ty2xgrqoy\n68NzapkNLKuFnJZJ+qmkTZLukzSqxHEXSNoqabukRWnHac3Jc2qZDSyrEseDwJkRcRbwM+D6vgdI\nGgF8FZgNnA5cLun0VKO0puQ5tcwGlkniiIh/jIiDyeajQEeRw6YB2yPiuYh4FbgbmJtWjNa8PKeW\n2cDy0Dj+QeAfiuxvB3b02u4CpqcSkQ2qkXsdeU4ts4HVLHFIegg4pchLiyPiu8kxi4GDwF1VuN4C\nYAHA+PHjh3s6G0Az9DrynFpmpdUscUTEOwZ6XdJ84N3A2yMiihzSDYzrtd2R7Ct1vRXACoDOzs5i\n57MqGajXkW+2Zo0vq15VFwCfBt4TEftKHPYEMEnSREnHA5cBq9OK0UpzryOz5pZVr6pbgBOBByU9\nJWk5gKSxku4HSBrPrwHWAluAb0fE5ozitV7c68isuWXSOB4R/67E/ueBOb227wfuTysuK8/CWZOP\naeMA9zoyayZ56FVldca9jsyamxOHDYl7HZk1LycOK1sjj90ws/I5cVhZmmHshpmVx7PjWlk8Y6yZ\n9XDisLJ47IaZ9XDisLJ47IaZ9XDisLJ4xlgz6+HGcSuLx26YWQ8nDiubx26YGbiqyszMKuTEYWZm\nFXHiMDOzijhxmJlZRZw4zMysIk4cZmZWERVf7ru+SdoF/DLrOAYwGvh11kHkjD+T/vyZ9OfPpL9q\nfSa/FxFt5RzYkIkj7yStj4jOrOPIE38m/fkz6c+fSX9ZfCauqjIzs4o4cZiZWUWcOLKxIusAcsif\nSX/+TPrzZ9Jf6p+J2zjMzKwiLnGYmVlFnDgyIGmZpJ9K2iTpPkmjso4pa5IulbRZ0mFJTd1rRtIF\nkrZK2i5pUdbx5IGkOyS9IOmZrGPJC0njJP2TpGeT/zvXpnVtJ45sPAicGRFnAT8Drs84njx4BpgH\nPJx1IFmSNAL4KjAbOB24XNLp2UaVC3cCF2QdRM4cBD4VEacD5wIfS+tvxYkjAxHxjxFxMNl8FOjI\nMp48iIgtEbE16zhyYBqwPSKei4hXgbuBuRnHlLmIeBj4TdZx5ElE7IyIJ5Pne4EtQCoL5jhxZO+D\nwPeyDsJyox3Y0Wu7i5RuBla/JE0ApgKPpXE9rwBYI5IeAk4p8tLiiPhucsxiCsXNu9KMLSvlfCZm\nVhlJJwDfAf4sIl5K45pOHDUSEe8Y6HVJ84F3A2+PJukTPdhnYgB0A+N6bXck+8z6kdRCIWncFRH3\npnVdV1VlQNIFwKeB90TEvqzjsVx5ApgkaaKk44HLgNUZx2Q5JEnA7cCWiPjrNK/txJGNW4ATgQcl\nPSVpedYBZU3SeyV1AecBayStzTqmLCSdJq4B1lJo7Px2RGzONqrsSfoW8L+ByZK6JH0o65hyYAZw\nJTAzuY88JWlOGhf2yHEzM6uISxxmZlYRJw4zM6uIE4eZmVXEicPMzCrixGFmZhVx4jArQtInJG2R\nVPGofkkTJF1Ri7iS879F0pOSDkq6pFbXMSvFicOsuP8EvDMi3j+En50AVJw4kplxy/F/gPnANyu9\nhlk1OHGY9ZEMyDwV+J6kT0p6bbIexOOSNkqamxw3QdKPkm//T0p6c3KKpcC/TwZkfVLSfEm39Dr/\n/5J0fvL8ZUl/JeknwHmSzpH0Q0kbJK2VNKZvfBHxi4jYBByu8UdhVpTnqjLrIyI+mkwL87aI+LWk\nzwPrIuKDyaJbjycTNr5AoVTyiqRJwLeATmARcF1EvBuOzEtWymuBxyLiU8m8Qz8E5kbELkl/AnyO\nwgzKZrnhxGE2uHcB75F0XbI9EhgPPA/cIuls4BDw+0M49yEKk9QBTAbOpDAVDcAIYOcw4jarCScO\ns8EJuLjvQlOSlgC/AqZQqPZ9pcTPH+TYauGRvZ6/EhGHel1nc0ScV42gzWrFbRxmg1sLfDyZjRRJ\nU5P9JwE7I+Iwhcnmehq391KYxLLHL4CzJR0naRyFVf6K2Qq0STovuU6LpDOq+k7MqsCJw2xwnwVa\ngE2SNifbAF8D/jRp2P4D4LfJ/k3AIUk/kfRJ4BHgX4BngS8DTxa7SLJU7CXAF5JzPgW8ue9xkt6U\nzCR8KXBrEpNZajw7rpmZVcQlDjMzq4gTh5mZVcSJw8zMKuLEYWZmFXHiMDOzijhxmJlZRZw4zMys\nIk4cZmY7ublFAAAACElEQVRWkf8P6LxbxT/kOlAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10db68940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], label='class 0', marker='o')\n",
    "plt.scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], label='class 1', marker='s')\n",
    "plt.xlabel('feature 1')\n",
    "plt.ylabel('feature 2')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing a Perceptron in NumPy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Implement function for perceptron training in NumPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def perceptron_train(features, targets, mparams=None,\n",
    "                     zero_weights=True, learning_rate=1., seed=None):\n",
    "    \"\"\"Perceptron training function for binary class labels\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    features : numpy.ndarray, shape=(n_samples, m_features)\n",
    "        A 2D NumPy array containing the training examples\n",
    "\n",
    "    targets : numpy.ndarray, shape=(n_samples,)\n",
    "        A 1D NumPy array containing the true class labels\n",
    "\n",
    "    mparams : dict or None (default: None)\n",
    "        A dictionary containing the model parameters, for instance\n",
    "        as returned by this function. If None, a new model parameter\n",
    "        dictionary is initialized. Note that the values in mparams\n",
    "        are updated inplace if a mparams dict is provided.\n",
    "\n",
    "    zero_weights : bool (default: True)\n",
    "        Initializes weights to all zeros, otherwise model weights are\n",
    "        initialized to small random number from a normal distribution\n",
    "        with mean zero and standard deviation 0.1.\n",
    "\n",
    "    learning_rate : float (default: 1.0)\n",
    "        A learning rate for the parameter updates. Note that a learning\n",
    "        rate has no effect on the direction of the decision boundary\n",
    "        if if the model weights are initialized to all zeros.\n",
    "\n",
    "    seed : int or None (default: None)\n",
    "        Seed for the pseudo-random number generator that initializes the\n",
    "        weights if zero_weights=False\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    mparams : dict\n",
    "        The model parameters after training the perceptron for one epoch.\n",
    "        The mparams dictionary has the form:\n",
    "        {'weights': np.array([weight_1, weight_2, ... , weight_m]),\n",
    "         'bias': np.array([bias])}\n",
    "\n",
    "    \"\"\"\n",
    "    # initialize model parameters\n",
    "    if mparams is None:\n",
    "        mparams = {'bias': np.zeros(1)}\n",
    "        if zero_weights:\n",
    "            mparams['weights'] = np.zeros(features.shape[1])\n",
    "        else:\n",
    "            rng = np.random.RandomState(seed)\n",
    "            mparams['weights'] = rng.normal(loc=0.0, scale=0.1,\n",
    "                                           size=(features.shape[1]))\n",
    "\n",
    "    # train one epoch\n",
    "    for training_example, true_label in zip(features, targets):\n",
    "        linear = np.dot(training_example, mparams['weights']) + mparams['bias']\n",
    "\n",
    "        # if class 1 was predicted but true label is 0\n",
    "        if linear > 0. and not true_label:\n",
    "            mparams['weights'] -= learning_rate * training_example\n",
    "            mparams['bias'] -= learning_rate * 1.\n",
    "\n",
    "        # if class 0 was predicted but true label is 1\n",
    "        elif linear <= 0. and true_label:\n",
    "            mparams['weights'] += learning_rate * training_example\n",
    "            mparams['bias'] += learning_rate * 1.\n",
    "\n",
    "    return mparams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Train the perceptron for 2 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_params = perceptron_train(X_train, y_train, \n",
    "                                mparams=None, zero_weights=True)\n",
    "\n",
    "for _ in range(2):\n",
    "    _ = perceptron_train(X_train, y_train, mparams=model_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Implement a function for perceptron predictions in NumPy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def perceptron_predict(features, mparams):\n",
    "    \"\"\"Perceptron prediction function for binary class labels\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    features : numpy.ndarray, shape=(n_samples, m_features)\n",
    "        A 2D NumPy array containing the training examples\n",
    "\n",
    "    mparams : dict\n",
    "        The model parameters aof the perceptron in the form:\n",
    "        {'weights': np.array([weight_1, weight_2, ... , weight_m]),\n",
    "         'bias': np.array([bias])}\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    predicted_labels : np.ndarray, shape=(n_samples)\n",
    "        NumPy array containing the predicted class labels.\n",
    "\n",
    "    \"\"\"\n",
    "    linear = np.dot(features, mparams['weights']) + mparams['bias']\n",
    "    predicted_labels = np.where(linear.reshape(-1) > 0., 1, 0)\n",
    "    return predicted_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Compute training and test error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training errors 0\n",
      "Number of test errors 2\n"
     ]
    }
   ],
   "source": [
    "train_errors = np.sum(perceptron_predict(X_train, model_params) != y_train)\n",
    "test_errors = np.sum(perceptron_predict(X_test, model_params) != y_test)\n",
    "\n",
    "print('Number of training errors', train_errors)\n",
    "print('Number of test errors', test_errors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Visualize the decision boundary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perceptron is a linear function with threshold\n",
    "\n",
    "$$w_{1}x_{1} + w_{2}x_{2} + b \\geq 0.$$\n",
    "\n",
    "We can rearrange this equation as follows:\n",
    "\n",
    "$$w_{1}x_{1} + b \\geq 0 - w_{2}x_{2}$$\n",
    "\n",
    "$$- \\frac{w_{1}x_{1}}{{w_2}} - \\frac{b}{w_2} \\leq x_{2}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa4AAADFCAYAAAAMsRa3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4VNX5wPHvyWRCErJBErYECPuWEELCGtzQVqoCAVll\nETeUzR1/WFpF3EttVXYoiCAqoBBEqlhLrWWVBMIOsgkkUMFgAEmQ7f7+SEIDZJJZ7sy9M/N+nofn\nkUly5yXOmXfOed9zrtI0DSGEEMJbBBgdgBBCCOEISVxCCCG8iiQuIYQQXkUSlxBCCK8iiUsIIYRX\nkcQlhBDCq0jiEkII4VUkcQkhhPAqkriEEEJ4lUAjnjQmJkZLSEgw4qmF0EV2dvZPmqbFGh1HKRlT\nwhfYO64MSVwJCQlkZWUZ8dRC6EIpddjoGMqSMSV8gb3jSpYKhRBCeBVJXEIIIbyKJC4hhBBexZAa\nlxDC/S5evEhubi7nz583OhRTCQ4OJj4+HqvVanQowkk+lbj+tecE//7+JOPvboHVIpNJ4d9yc3MJ\nDw8nISEBpZRT1zj36yV+PneBOtVCCHDyGmaiaRr5+fnk5ubSoEEDo8MRTvKpd/ecowXMW/cDg2Zv\n5OTZX40ORwhDnT9/nujoaKeTFkDhhcucKrzAoZ/OcenyFR2jM4ZSiujoaJmFejmfSlxP/aYp7wxo\nw7a8ArpPXkPO0QKjQxLCUK4kLYDY8CrUrR5K4YXLHDh5jl8vXdYpMuO4+jsRxvOpxAXQs00cn47o\nTKBF0W/GehZvOmp0SEJ4tWqhQTSMqcqlK1c4cOIc5369ZHRIws/5XOICaFUnkhWju9C+QXWe+3Qb\nf8zcwYVL3r/MIYRRqlYJpHFsGJYAOPjTOQoKLzh9rQkTJvDnP/9Zx+j+Jzs7m6SkJBo3bszjjz+O\npmlueR5hLJ9MXADVqgYx74F2PHpzQxZsOMx9szdw4qysawthS+aWPNLfWE2DcStJf2M1mVvyrvl6\nFauFRrFhhFotHDlVyIkz502XGEaMGMHs2bPZt28f+/bt48svvzQ6JOEGPpu4AAItATx/VwsmD0xh\n57EzdJ+8hi1HfjY6LCFMJ3NLHs8v3U5eQREakFdQxPNLt9+QvAItATSIrUpUaBD/PXOe3J+LuFJB\n8po/fz6tW7cmOTmZIUOG3PD12bNn065dO5KTk7n33nspLCwEYMmSJSQmJpKcnMzNN98MwM6dO2nf\nvj1t2rShdevW7Nu375prHT9+nDNnztCxY0eUUgwdOpTMzEwXfzPCjHw6cZXqnlyHpSM7ExQYQP+Z\nG1i06YjRIQlhKpNW7aXo4rWNF0UXLzNp1d4bvjdAKepWC6FmRDA/F17gBxsdhzt37uSVV15h9erV\nbN26lXfeeeeG7+nduzebNm1i69attGjRgjlz5gAwceJEVq1axdatW/nss88AmDFjBk888QQ5OTlk\nZWURHx9/zbXy8vKueSw+Pp68vGsTr/ANfpG4AFrUjmDF6C50aFid//t0O+OXbZe6lxAljhUUOfS4\nUoqaEcHUrRbKORsdh6tXr6Zv377ExMQAUL169Ruus2PHDm666SaSkpJYuHAhO3fuBCA9PZ1hw4Yx\ne/ZsLl8uvm6nTp147bXXePPNNzl8+DAhISFO/3uFd/ObxAUQFRrEvAfaM+LWRizceISBszdw4ozU\nvYSoE1V+ErD1eKlqVYNo4ELH4bBhw5gyZQrbt2/nxRdfvLq/asaMGbzyyiscPXqU1NRU8vPzue++\n+/jss88ICQnhrrvuYvXq1ddcKy4ujtzc3Kt/z83NJS4uzqF4hHfwq8QFYAlQ/F+35ky9ry27jp3h\nnslryD4sdS/h38be2YwQq+Wax0KsFsbe2azSnw2rEkij2DACrus47Nq1K0uWLCE/Px+AU6dO3fCz\nZ8+epXbt2ly8eJGFCxdeffzAgQN06NCBiRMnEhsby9GjRzl48CANGzbk8ccfp2fPnmzbtu2aa9Wu\nXZuIiAg2bNiApmnMnz+fnj17Ovy7EObnd4mr1N2ta7NsVGdCgiwMmLWeDzdK3Uv4r4yUOF7vnURc\nVAgKiIsK4fXeSWSk2DdjCbZaaHxdx2HLli0ZP348t9xyC8nJyTz99NM3/NzLL79Mhw4dSE9Pp3nz\n5lcfHzt2LElJSSQmJtK5c2eSk5NZvHgxiYmJtGnThh07djB06NAbrjdt2jQefvhhGjduTKNGjfjd\n737n9O9EmJcyop01LS1NM8tN704XXuTxj7fw7+9PMrB9XSb0aEWVQEvlPyj8mlIqW9O0NKPjKFXe\nmNq9ezctWrTwaBxXrmjk/lxEQdEFqocGmfaMQyN+N6Jy9o4rv51xlYoMtTJ3WDtG3tqIj747yoBZ\nG/hR6l5COCUgQFG3egg1woM5VdpxeEWaoIS+/D5xQXHd67luzZk2qC17/3uWeyavIeuHG9fjhRCV\nU0pRKzKY+NKOwxPnuOADZxwK85DEVcZdSbXJHJVO1SALA2dv4IMNh013MoAQ3qJ61SAaRIdy6coV\n9p84R6GccSh0IonrOk1rhrN8dBe6NI7hD5k7GPfpdp84EVsII4QFW6/pODztwhmHQpRyOXEppeoq\npf6llNqllNqplHpCj8CMFBliZc797RjTtTGLso7Sf+YG/nta6l5COKO04zDYauHwqUJOnDXfGYfC\nu+gx47oEPKNpWkugIzBKKdVSh+saKiBA8cxvmzFjcCr7fiyue313SOpeQjgj0BJAw5iqRIVY+e/p\n8+QVVHzGoRAVcTlxaZp2XNO0zSX/fRbYDfjMdvVuibXIHJVORHAg983ewPz1P8inReFWvriKAaUd\nh6HMnzKJv771Fj/8dI7LOnccjh8/nrp16xIWFqbrdYW5BOp5MaVUApACbCzna8OB4QD16tXT82nd\nrknNcDJHp/PUxzm8sHwn23NP83JGIsFW2e8l3KJ0FWOzUiocyFZK/UPTtF1ue8bX4uDCLzc+HhQG\nv9fvoFqlFGHBVgKCrJz7tfiMw4ToUIJ02jvZvXt3Ro8eTZMmTXS5njAn3ZozlFJhwKfAk5qmnbn+\n65qmzdI0LU3TtLTY2Fi9ntZjIoKtzB6axuO3N2FJdi79Z663eQCpEK4wZBWjvKRV0eN2snVbk9Cg\nQBrEhPLR/Pdom9aOpNatXb6tCUDHjh2pXbu2SzEL89NlxqWUslKctBZqmrZUj2uaUUCA4unfNCWx\nTgRPL95KjylrmHpfWzo0jDY6NOGjvHkVo/S2JuvWrSMmJuaGswrDgq08PGQA/QcP49IVjffffYM5\nc+YwZsyYq7c1iYuLo6CgAPjfbU0GDRrEhQsXrp4aL/yPHl2FCpgD7NY07S+uh2R+v21VUvcKsTLo\nbxuZt/aQ1L2E7rx9FcOe25rs37ubh/vezb13dObDjz4kK2cbmqbJbU1EhfRYKkwHhgBdlVI5JX/u\n0uG6pta4RhiZo9K5tVkNJqzYxbNLtnH+oh9/AnwtDiZE3vjnNZ/p0/Eof1nFGDZsGFOnTmH3zh08\n9dzvKThzjryCIqZPn+7wbU2E/3B5qVDTtDWA+U7R9ICIYCuzhqQyefV+/vr193z/41lmDEklrpJ7\nGHk1W0V8Wy78UvwzOhb4fZ2vrGJ07dqVXr168fTTTxMdHc2pU6dumHWV3tbk8uVLrFr+CdVja3Hq\n3AX27z/ATe3a0aFDB7744guOHj3K6dOnr97W5MiRI2zbto2uXbsa9K8TRpKTM1wUEKB44o4m/G1o\nGj/8dI7uk9ew/kC+0WG5jzPFehcL/H7I86sYQTbax209bodWrVo5fFuTkCAL8dVCeOWF39OyVRKt\nHLytyXPPPUd8fDyFhYXEx8czYcIEp+N3C1mZ0IXf39ZETwdO/sLw+Vn8kF/I+Lta8EB6AsqEt3Rw\nyYRI/a6lc6u1J8ltTdzr7PmLHMkvRAUoEqJDCQ3SdeeOcb+bisbPhNOei8Ok5LYmBmgUW1z3ur15\nDSZ+votnFm/177pXZWQmJmwID7bSqEYYAcDBk+c4XXTR6JCEiUji0ll4sJUZg1N55jdNWZaTx73T\n15H7c6HRYQnhdYKtFhrVKDnjMP8cJ8/+Kt27ApDE5RYBAYoxtzdhzv1pHMkvpPvkNazb/5PRYQk/\n5O1v9NaSMw4jQ6wcP13EsQLXD+j19t+JkMTlVl2b12T56HRiwqowZO53/O0/B71/0FRUxHehkC/0\nFxwcTH5+vte/5gICFPWqhxIbXoX8c7/yQ34hl68492/SNI38/HyCg4N1jlJ4kr4VT3GDhrFhLBuV\nzrOLt/LKyt1szzvNG71bExLkpeccOtNMoWdDh7BbfHw8ubm5nDx50uhQdHPh10scKLzIYYsiJqwK\nlgDHm5+Cg4OJj493Q3R2CAqzfSakh86L9AWSuDwgrEog0we3Zdo3B/jzV3vZ9+MvzBySSt3qoUaH\n5hm2BiuUn9RkoOrCarXSoEEDo8PQ3b+/P8mohZsJDbIwd1g7EuO86INRRa9rWx/wzNjEZHCSlaVC\nD1FKMeq2xsy9vx1Hfy6kx5Q1rNnnJ3Wv3+cVt/pe/8cWMw5UYRq3NI3l0xGdsVoC6DtjPV/v+tHo\nkPyPmw5ltpckLg+7rXkNVozuQmx4FYbO3cjsb32g7iWEhzWrFc6ykZ1pUjOMRxZk8d7aQ0aHJEp5\nYGO1JC4DJMRUZdnIdLol1uLVv+/m8Y9zKLxwyeiwhPAqNSKC+Xh4R37ToiYvrdjFi8t3cOmyvjem\nFC5y0wxMEpdBqlYJZOp9bXmuWzM+33aM3tPWcfSU7PcSwhGhQYFMH5zKIzc14P31hxm+IJtzv8qH\nQF8nzRkGUkox8tbGtKwdweMfbaH7lDVMHpjCTU3Md4sKnyWdXF7PEqAYf3dL6kVX5cXlO+g7Yz1z\nhqVRO9LLDruuqONQXEMSlwnc2qwGK8Z04dEF2dw/9zue69acR29u6HvnHF7PkW5DZxNJZYnJ4CKz\n0M+QjvWJrxbC6IWbyZi6ljn3e0HHobd+cKpo7HqAJC6TqB9dlaUjOzP2k2288cUetuedZlKf1rof\nLuox9gzI8gam3i3Bkpj8ym3NavDJiM48OG8T/WauZ/LAFG5vUdPosGzz1tfn9WPXw3s1vfRd0TeF\nBgUyZWAKSXGR/OnLPRw4Ubzfq350VaNDc5y3Dkjh9VrUjiBzVDoPvb+JR+Zn8WL3VtzfOcHosBy/\nl5038fAypyQuk1FK8dgtjWhZO4IxH22hx5S1vDswhVuaGlj3qmj2BL47GIXXqhkRzOJHO/H4Rzm8\n+NlOfsg/xx/ubunUSRu68eVx4uFlTUlcJnVz01hWjO7C8AVZDHvvO8be2YwRtzQypu4lsyfhhUKD\nApk5JJVXV+5m7tpDHD1VyDsDUqhaxY/f9ry1pnYdaYc3sXrRoSwd2Zl7WtfhT1/uZdSHm6XVV29u\nuPOvMA9LgOKF7i2Z2LMVq/ecoN/M9fx45rzRYRnHRz6E+vFHD+8QGhTIuwPa0Doukte/2M3+E78w\na0gaCTFeWPeyh95r5ZVdz4s+ZQrnDe2UQN1qoYz+8H8dhy3rRBgdlm3ywalCkri8gFKKR25uSIva\nEYz+aDM9pqzhnYEp3Nashn5PUlkdSy+VXU/vRCKJSZS4rXkNFj/WiYfmZdF3xjqmDGqr7xhyRUVn\nd9rDR5YA7aVL4lJKzQXuAU5ompaoxzXFjbo0iWHF6OL9Xg/O28Szv23GyFt1qnu5YwnB1cFYys8G\npXCfVnUiyRyVzoPzNvHQvE281DORIR3rGx2W63xkCdBeetW45gHddLqWqEDd6qF8OqIzPZLrMGnV\nXkZ8sJlf3F33cmbWpedMzc8GpXCvWpHBLHmsE7c2q8EfM3fwyue7nL4xpTCGLjMuTdO+VUol6HEt\nUbmQIAtv929DUlwkr3+xh15T1zJraBoN3FX3cmSjMOg30xLCTapWCWT20DRe/nwXf1tziMOnCnln\nQBv3bvg3w5FOZohBBx6rcSmlhgPDAerVq+epp/VZSikevqkhLWtHMOrDkrrXgDZ0be6hUwJ8ZAAI\n/2UJUEzo0Yr60aG8/Pku+s/cwJz706gREeyeJzRyWdvHlts9lrg0TZsFzAJIS0uTeblOOjeOuXrO\n4UPvZ/HUHU0ZfVtjAvTcaGnrLsUys3IbqRt7zgPpDahXPZQxH20hY+pa5j7Qjua1TNxx6IzKltu9\nLLFJV6EPiK9WXPd6ful2/vKP79mRd5q3+iUTHmy1/yKOHppZ3vd62Yvf5OYBU4D5BsfhF25vUZPF\nj3biofc30Wf6eqYOaqvvaTXuHhuuroB4WR1ZEpePCLZa+Eu/ZJLiInn177vJKKl7NYq184Vra/A4\ncniml734zUzqxp6XGFfacZjFg/M2MbFnKwZ10Knj0N1jw88+GOrVDv8RcCsQo5TKBV7UNG2OHtcW\n9lNK8WCXBrQoqXtlTFnLX/u34Y6Wbqp76XloqKf2kfkQqRvrr3ZkCEse68SYDzczftkODucXMq5b\nc32X3oXL9OoqHKjHdYQ+OjWKZsWYLjy2IJuH52fx5B1NeLxrE/0Hn54zKZmtOUzqxu4RVtJxOPHz\nXcz69iCH88/xdv8UQoIs7n9yWW63i5xV6KPiooo/OfZuG8fbX+9j+IJszpy/aHRYQniFQEsAL/Vo\nxQv3tOSrXT8yYNZ6Tpz1wBmH7voA52NnckqNy4cFWy281TeZ1nGRvLyypO41JI3GNRx4sVZU9DV6\nNuSlg054h9Kl9/hqITzxcQ69pq5j7rB2NKsVbnRojqtstuZl21skcfk4pRTD0hvQvHYEo0puaf6X\nfsn8tlUt2z9k73KFvY0berz4/az1XurG5vHbVrVY/GgnHnx/E32mr2Pa4Lbc1MTBjkOzJQYvX5KU\nxOUnOjYsrnuN+CCb4Quyefz2Jjx5u426lx7LFX6WaPQmdWNzSYov7jh8aN4mhr23iVcyEhnY3oGG\nGLMlAy+vKUuNywdlbskj/Y3VNBi3kvQ3VpO5pXjQ1IkKYdGjneiTGs+7/9zHI/OzOF3kQt1Lz3Vz\nky5JCFGqtG7cpXEMzy/dzutf7OaKnHFoCJlx+ZjMLXk8v3Q7RRcvA5BXUMTzS7cDkJESR7DVwqQ+\nrUmOj+SlFbtK6l6pNKnpxLq9PZ8i7V2S+H1exe311y9LesmShvAt4cFW5tyfxouf7WTmvw9yJL+Q\nv/ZvQ7BVp45Dsy0pmpQkLh8zadXeq0mrVNHFy0xatZeMlDiguO41pFMCzWpFMHJhNhlT1/JWvzZ0\nS6yg7lWWI+vjjixJOHKYr5csaQjfE2gJ4JWMRBrEVOXVv+/m+KwNzB6aRmx4FdcvLh/G7CKJy8cc\nKyiy+/H2DaoX7/f6YDOPfZDNmK6NefKOplT62bGiZOTISRtCeKnSQ67jq4Xy5KIt9Jq2lveGtbNv\n5cLLGyPMQGpcPqZOVIhDj9eODGHR8I70S4tn8ur9PPz+Jk5bbXRMyXKFENfolliLRcM7cf7iFXpP\nX8fa/T9V/kNmaIzw8n1dMuPyMWPvbHZNjQsgxGph7J3NbP5MsNXCm/e2pnV8FC+t2EnPqNnMejiN\nps7UvYTwM8l1o8gc1ZkH523i/rnf8VqvJPq1q2t0WBXz8pmdX8y4bHXZ+aKMlDhe751EXFQIiuJO\nqNd7J12tb9milGJwx/p89EhHfvn1MhlT1/LF9uOeCVoILxdfLZRPRnSmU6Nonvt0G3/6co90HLqR\nz8+4Kuuy80UZKXEO/dsyt+QxadVejhUUUScqhNG3NWL51mOMWLiZkbc24pnfNsOi9zmHpUsSFXUS\nBoVVfEJH2Xqa1AeEwSKCrcwd1o4Xlu9k2jcHOJxfyFv9kvXrOBRX+XzisqfLzp+Vl9jf/HIvL/ds\nRfNa4Uz75gA7j53h3QEpRIaW3N9Lj+OeLvxS+enyF365cSOzdBkKE7NaAnitVyINYkJ57e97OHa6\niNlD04gJ06HjUFzl80uFjnTZeZoZljBtJfa/fr2P13u35rVeSaw78BM9pq5h73/PFn+DXjMbSTbC\nBymlGH5zI6YPasuuY2foNW0t+0+c/d83eHljhBn4/IyrTlQIeeUkKVtddp5iliXMyhL7fR3q0axW\nOCM+yKbXtLVM6pPM3a1rV7xR8vrEJi3ywg/9Lqk2tSKDeWR+Fr2nrWPGkFQ6N4qRJW0d+PyMa+yd\nzQi5bo25si47T6hoCdOT7GmfT61fjRVjutC8VjijPtzMG1/s4fK43OJlvOv/QHGiKvtHCD+VUq8a\ny0amUzMimKFzvmNJ1lGjQ/IJPp+4nO2yczezLGHam9hrRgTz8fBODOpQjxn/PsCw976joPDCjReU\n5T8hrlG3enHHYceG0Yz9ZBt/XrVXOg5d5PNLheB4l50nmGUJs/T3UrarcOydzcr9fQUFBvBqrySS\n4iJ5YflOuk9Zw6whabSoHeF8ABU1epS35i9nuQkvFBli5b0H2vGHZTuY8q/9HD5VyKQ+raXj0El+\nkbjMyJmNwu7iaGIf0L4eTUvqXr2nrePNPq3pkVzH8Sd2poVd6gPCS1ktAbxxbxIJMVV588s9HCso\nYtaQVKKl49BhkrgMUt5M57bmsUxatZenFuVUOPMxg7b1iuteIz/YzOMfbWFn3mnG3tms4heUPffo\nknPchA9TSjHi1kbUqx7KU4tz6D29+K7KjWJlxcARkrgMVHamY5YuQ0fUCA/mw0c6MvHzncz89iA7\nj51hshZGNeVCncsM57gJ4WZ3ty7uOBxe0nE4c0gqHRtGGx2W19ClOUMp1U0ptVcptV8pNU6Pa/ob\ns3QZOiooMIBXMpJ4894kvjt0iu4XXmPnlfrlfKN8ohSirNT6xR2HMWFBDJmzkU+zc40OyT6vxd3Y\nOTwhsvhxD3F5xqWUsgBTgd8AucAmpdRnmqbtcvXa/sQsXYbO6t+uHs1qRfDYgmzuLXqTNzNa07ON\nOWeKQphFvehQlo5IZ8TCbJ5ZspXDpwp56o4mKKXzEWt6MsGqiB4zrvbAfk3TDmqadgH4GOipw3X9\niqO3IzGjNnWjWDGmC63jonji4xxeXbmLS5evGB2WEOZUMnOJ/FMM83Lvoq/lG9795z6e/OOL/Hrp\ncuU/78f0SFxxQNlddbklj11DKTVcKZWllMo6efKkDk9rbo4e52TWjdKOig2vwsJHOnB/p/rM/s8h\nhs79jlPnytnvJYS/KzNDCVKX+VPgLMYGLmL5pQ4M/ttGGTcV8NgGZE3TZmmalqZpWlpsrI0bFfqI\n0kaLvIIiNP7XaFFR8vLERmlPnY1otQTwUs9EJvVpTdbhn+k+eQ078uzoKAQ5x034DEfHm1IwKnA5\nk63vsjX3NL2nreXQT+c8FK130aOrMA8oe9e0+JLH/JazJ9K7c6O0EV2LfdPq0qxWOI8uyObe6et4\n897WlT+XtLwLH+DKeOtu2UCdBzrwyPzi80FnDk6lg3QcXkOPGdcmoIlSqoFSKggYAHymw3W9lhkb\nLYzqWmwdX1z3Sq4bxZOLcpi4Qupe9pBOXe/m6nhLrV+dZSM7U71qEIPnbGTZFhN1HJpgVcTlGZem\naZeUUqOBVYAFmKtp2k6XI/NiZjnOqSwjk2lMWBUWPtyBV1fuZu7aQ+w6fpqp97WVEwNskE5d76fH\neKsfXZVlI9J59IMsnlq0lcP5hTxxuwk6Dk2wKqJLjUvTtL9rmtZU07RGmqa9qsc1vZkrjRbuqkMZ\n3bVotQQwoUcr3uqbzJYjBXSfvIbtuXbWvfyPdOp6ObvGmx0zl8hQK/Mf7MC9beN5++t9PL14q3Qc\nIidnuIUjB9eW5c46lFnORrw3NZ6mNcN5dEEWfWas4/XeSfRuG+/RGLxAeZ26Ha7/JqXUcGA4QL16\n9TwTmbCLXePNzplLUGAAf+7bmoToUN76x/fklZxxGBUapHfYXkNpmueP109LS9OysrI8/rz2ytyS\n53DS0UP6G6vLXWKMiwph7biuLl/fqH9XefJ/+ZVRH25mw8FTDOucwPi7W2C1eM9ddpRS2Zqmpbnp\n2n2AbpqmPVzy9yFAB03TRtv6GbOPKX/kjvG2PCePsUu2EVcthPeGtSMhpqpO0ZqDveNKZlzXMfLM\nQHfXocx0e5fosCp88FAHXv9iD3PWHGL38TNMHdSWGKl7gXTq+gR3jLeebeKoExXC8PlZ9Jq2lllD\n02iXUF3X5/AG3vMR1wMyt+TxzOKthp0ZaHQdytMCLQH88Z6WvN2/DTlHi+te23ILjA7LDKRTV9jU\nLqE6y0amExUaxKDZG1me43+faWTGVaJ0pnXZxtKpJ7rvzFKH8rSMlDga1wjj0QXZ9JmxnlczEumb\nVrfyH/RRvtypa6blatOy49Y+CTFVWTqiM49+kM0TH+dwJL+Q0V0bG99x6CGmnXF56pSHUuXtuyjL\nE7MeT5yeYVaJcZGsGNOFtPrVGPvJNl5cvoOLfrzfyxc7dZ05UcYv2XmIbbWqQSx4qD29UuJ46x/f\n8+ySbVy45B9jxpQzLiPqTBXNqDw56zFTHcrTqlcNYv6D7Xnzyz3M/s8hdh8/y9RBbYkNl7qXL3D2\nRBlhW5VAC3/pl0z96FDe/nofeQWFzBycRmSo1ejQ3MqUMy4jTnmwNaOyKOU3sx4zCLQEMP7ulrwz\noA3b8orrXjlHpe7lC8x4oowvUErx5B1N+Wv/ZDYfLqDX9LUczvftMw5NmbiMeIHb2jT8Vr9kSVoG\n6Nkmjk9HdCbQoug3Yz2LNx2t/IeEqflb85Gn9UqJZ8FD7Tl17gK9pq0j+/Apo0NyG1MmLiNe4P5c\nXzKrVnUiWTG6C+0bVOe5T7fxh8ztfrOG74t85dY9ZtahYTRLR3QmIjiQgbM3smLrMaNDcgtT1riM\n6q7zVH2ptLMqr6AIi1Jc1jTipMOqXNWqBjHvgXZMWrWXmd8eZM/xs0wb3JYa4cFGhyYc5OyJMn4n\nKMx2V6EdGsaGsXRkOo8uyGLMR1s4cqqQkbc28qmOQ9OenGHGtlk9Yrq+8aQsq0VRNSiQ00UXTfFv\nNtv/g8+2HuO5T7YSGWJl+uBU2tarZlgs7jw5wxlycoZ9zPaadqdfL13muU+2sTznGH1T43m1VxJB\ngaZcZLuY0GpFAAAOa0lEQVTK60/OMFt3nV6djhW13V+8rFFQdPGG65f+nCcHm5EniNjSI7kOjWPD\nePSDLAbM3MDEnq0Y0F7O6BP2MeNr2p2qBFp4u38b6kdX5d1/7iOvoIjpg1J9ouPQ3OnXRPTqdHSk\nwaTo4mUmfLbTkL0vRt2/qzIt60SwYnQXOjSszril2xm/TOpewj5mfU27k1KKp3/TlLf6JrPph1P0\nnr6WI/mFRoflMklcdtKr09HRBpOCoouGDDYzty5HhQYx74H2PHZLIxZuPMLA2Rs4cea80WEJkzPz\na9rd7k2NZ8FDHfjplwv0mraWzUd+Njokl0jispMznY7lnf5RXmeVM9w92MzeumwJUIz7XXOm3JfC\nrmNnuGfyGp9u/xWuM/tr2t06Noxm6cjOVK0SyMBZG1i57bjRITlNEpedHG3ltXW8DXC17b4yIVYL\n1WysR5cOtswtebR56SsSxq0kYdxKUiZ+pcsyore0Lt/Tug7LRnUmJMjCgFkb+HDjEaNDEiblLa9p\nd2oUG8aykZ1JjItk1Iebmf7NAYxo0HOVabsKzciRjiRb99aKCrGS8+JvK/0+i1K81S8ZoNytAa/3\nTgJg7JKtXLxy7f9Dq0UxqY/rG6e9qQPrdOFFxny8hW+/P8nA9nWZ0KMVVQJdn9naIl2FxbzpNQLe\nF6+7nL94mWeXbOXzbccZ0K4uL2ckmuJ+eF7fVejtbC3lFRRdJHNL3tXBYmvP2vWbn8sbbOlvrL4h\naUFxd6Ie57+ZrbOzIpGhVt4b1o63vtrLtG8OsOe/Z5kxOJWaEbLfy13M3qVnK0mZITajBVstvDsg\nhYToqkz5135yfy5i6qC2RIZ4R8ehJC47OTpI60SFlDuTAq5JKvZsyrQ12Cqqc/lDwfl6lgDFc92a\nkxgXybNLtnLP5DVMH9SWND+80Z4nmPnQXLMnVTMICFA8e2cz6keH8vzS7fSZvo65w9pRt3qo0aFV\nyvi5oZdwtJW2onXz65NKRkoca8d15dAbd7N2XFe7B1ZFRWV/KTiX566k2mSOSqdqkIWBszfwwYbD\nXrmOb3Zm7tLzx9Z3Z/VNq8v8B9vz45nz9Jq21isOtZbEZSdHB2lGSlyljRWuGntnM6wBNx7jYrUo\nvyo4l6dpzXCWj+pCeuMY/pC5g3Gfbud8BfdbE44zc5eemZOqGXVuHMPSkcVNTv1nrueL7ebuOHQp\ncSml+iqldiqlriilTFOodgdnBumL3VtV2sXkyg0zM1LimNQ3magy69LVQq30b1eXSav2XnNNT9+Y\n0wwiQ63Mub8dY7o2ZlHWUfrP2sDx0/LGpRczd+mZOamaVeMa4SwbmU7LOhGM/HAzM/9t3o5Dl7oK\nlVItgCvATOBZTdPsamvyxg6o8s4YLK+JwpHndPaajsZpDVCgips2ynsef+i0+nLHcZ5ZvJWQIAvT\nBqXSvoFrdS/pKixm1teOO8aWvzh/8TLPLNnKym3HGdi+HhN7tvJYx6FHugo1Tdtd8mSuXMYjXC3W\nOnuydUVdTO4obpd3zfI6D8uu9/tDEbtbYm0axYYxfEE2983ewAvdWzKkY32veO2amVm79OQkeucF\nWy1MHpBC/eqhTPvmALk/FzJ1UFsigs3TcajLPi6l1DdUMuNSSg0HhgPUq1cv9fDhwy4/ryNs7ZeK\niwph7biubnnOyj6NNhi3kvJ++wo49MbdTj2nrWuWR2G7+9GdvxcjnS66yNOLcvjnnhP0TY3n5YxE\ngp04yURmXMIfLN50lN8v207D2KrMHdaO+Gru7TjUbcallPoaqFXOl8Zrmrbc3oA0TZsFzILiQWbv\nz+nFXcXazC15vLRiJz8XFp/qHhViZUKPVkDlMxlbScOVdfiK2vDL+15/K2JHhliZPTSNt/+5j3f/\nuY+9Pxbv95LahxA36teuLnHVQnjsg2wypq5jzv1pJNeNMjqsypszNE27Q9O0xHL+2J20zMAdxdrM\nLXmM/WTr1aQFxRuMn1yUw5OLciptxy2vuK0oTnLONlCUd01rgMJquXZJrLSI7o9F7ICA4hOzZw5J\n5eDJc/SYsoaNB/ONDksIU0pvHMPSEZ0JtgbQf9Z6vtzxX6ND8p92eHd0QE1atfeahgd7lJ3JZKTE\nXXNuoYKry3zO3r6k7DUVxUt+k/omM6lP8jWPlRapzdwZ5m53tqpF5qjORIRYGfS3jby/7gejQxLC\nlJrULO44bF4rghELs/nbfw4a2nHoUnOGUqoXMBmIBVYqpXI0TbtTl8h05o5irTPLadfPZEqL2+XV\n4Jxt1LBVMK/oMX8tYjeuEU7mqHSeXrSVwguyz0sIW2LDq/Dx8I48vTiHV1bu5tBP53ipRysCDTjj\n0NWuwmXAMp1icTu9O6AcqSdBxTMZI2tNZu0M85SIYCuzhqRihgZDpVRfYALQAmhv7xYTITwh2Gph\nysC2/Kn6Xmb8+wC5Pxcx5b4Uwj3cceg3S4XuMPbOZjfUjmwpuzxXHn+sNZlJQIAyS2v8DqA38K3R\ngQhRnoCSe+G93juJNft/ou+M9R5v5pLE5YKMlDgm9Um2ebQTFM+y3u7fptIzCG3Vmm5rHut3J174\nM03TdmuaJgfqCdMb2L4e8x5oR97PRWRMXcv23NMee245Hd5F1y+zOXuSQHm1ptuax/Jpdp7Pbw4W\nzrlub6TB0Qh/dFOTWD4Z0ZkH522i38z1vDswhd+0rOn255UbSZqYEZumhX1c2YBsz95Iezb1lyVj\nShjpxNnzPPJ+FtvyTvOHu1vyYHqCU0vvciNJH+Bvm4P9haZpdxgdgxB6qhEezMfDO/Hkoi28/Pku\nDuef44V7Wrqt41BqXCYmDRtCCG8REmRh+qBUht/ckPnrD/PI/Cx++fWSW55LEpeJ+fPmYH+llOql\nlMoFOlG8N3KV0TEJYa+AAMXv72rBKxmJfLvvJ0Z8kO2W55GlQhPz983B/sjb9kYKUZ7BHetTt3oo\nEcHuSTGSuEzO3zcHC99n1nt6Cdfc0jTWbdeWxCWEMIyr98kT/klqXEIIw1R0M1UhbJEZl85k2UMI\n+8mWD+EMSVw6kmUPIRzjjpup6kk+iJqTLBXqSJY9hHCMmbd8lH4QzSsoQsP5e+QJ/cmMS0ey7CGE\nYzy55cPR2VNFH0Rl1mUsSVw6MvuyhxBm5IktH84s48sHUfOSpUIdmXnZQwh/5swyvhy5Zl6SuHSU\nkRLH672TiIsKQVH5zSOFEJ7hzOzJ3R9EM7fkyb32nCRLhTpz17KHdDcJ4TxnlvHdWX+TDmTXSOLy\nAvIiF8I1Y+9sds0YAvtmT+76ICqNH66RpUIvIG32QrjGbMv40vjhGpdmXEqpSUB34AJwAHhA07QC\nPQIT/yMvciFcZ6YDq6UD2TWuzrj+ASRqmtYa+B543vWQzM/TRVXpbhLCXFx9D5AOZNe4lLg0TftK\n07TSW1xuAOJdD8ncjNhNLy9yIcxDj/cAsy1dehs9mzMeBBbZ+qJSajgwHKBevXo6Pq1nGVFUlRtK\nCmEeer0HmGnp0ttUmriUUl8Dtcr50nhN05aXfM944BKw0NZ1NE2bBcwCSEtL05yK1gSMqjfJi1wI\nc5Cas/EqTVyapt1R0deVUsOAe4DbNU3z2oRkLymqCuEab9+TKO8BxnOpxqWU6gY8B/TQNK1Qn5DM\nK3NLHud+vXTD41JvEsI+vnDiuidrznK6Rvlc7SqcAoQD/1BK5SilZugQkymVDriCoovXPF4t1CpF\nVSHs5At7Ej3VWOELSd5dXGrO0DStsV6BmF15Aw4gNChQkpYQdvKV+pAnas5yuoZtcnKGnXxlwAlh\nJNmTaD95z7FNEpedZMAJT1BKTVJK7VFKbVNKLVNKRRkdk55kT6L95D3HNklcdpIBJzzEp0+jkY23\n9pP3HNvkdHg7ySZg4Qmapn1V5q8bgD5GxeIusifRPvKeY5skLgfIgBMe5hen0Qjb5D2nfJK4hPAw\nOY1GCNdI4hLCw+Q0GiFcI4lLCBMpcxrNLf5wGo0QzpCuQiHMxW9OoxHCWTLjEsJE/Ok0GiGcpYxY\nQldKnQQOu+nyMcBPbrq2MyQe28wUCzgWT31N02LdGYwjZEwZykzxmCkWcDweu8aVIYnLnZRSWZqm\npRkdRymJxzYzxQLmi8cszPZ7kXhsM1Ms4L54pMYlhBDCq0jiEkII4VV8MXHNMjqA60g8tpkpFjBf\nPGZhtt+LxGObmWIBN8XjczUuIYQQvs0XZ1xCCCF8mCQuIYQQXsUnE5fZbsanlOqrlNqplLqilDKk\nVVUp1U0ptVcptV8pNc6IGMrEMlcpdUIptcPIOEpiqauU+pdSalfJ/6MnjI7JjGRMlRuDjKnyY3H7\nmPLJxIX5bsa3A+gNfGvEkyulLMBU4HdAS2CgUqqlEbGUmAd0M/D5y7oEPKNpWkugIzDK4N+NWcmY\nKkPGVIXcPqZ8MnFpmvaVpmmXSv66AYg3OJ7dmqbtNTCE9sB+TdMOapp2AfgY6GlUMJqmfQucMur5\ny9I07bimaZtL/vsssBuQGyBdR8bUDWRM2eCJMeWTies6DwJfGB2EweKAo2X+nou8Od9AKZUApAAb\njY3E9GRMyZiyi7vGlNcesqvXzfg8GY8wL6VUGPAp8KSmaWeMjscIMqaEntw5prw2cZntZnyVxWOw\nPKBumb/HlzwmAKWUleIBtlDTtKVGx2MUGVMOkTFVAXePKZ9cKixzM74ecjM+ADYBTZRSDZRSQcAA\n4DODYzIFpZQC5gC7NU37i9HxmJWMqRvImLLBE2PKJxMXJrsZn1Kql1IqF+gErFRKrfLk85cU1UcD\nqygulC7WNG2nJ2MoSyn1EbAeaKaUylVKPWRULEA6MAToWvJayVFK3WVgPGYlY6oMGVMVcvuYkiOf\nhBBCeBVfnXEJIYTwUZK4hBBCeBVJXEIIIbyKJC4hhBBeRRKXEEIIryKJSwghhFeRxCWEEMKr/D/V\nKrdRWCwXbQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10dd9de80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_min = -2\n",
    "y_min = ( -(model_params['weights'][0] * x_min) / model_params['weights'][1]\n",
    "          -(model_params['bias'] / model_params['weights'][1]) )\n",
    "\n",
    "x_max = 2\n",
    "y_max = ( -(model_params['weights'][0] * x_max) / model_params['weights'][1]\n",
    "          -(model_params['bias'] / model_params['weights'][1]) )\n",
    "\n",
    "\n",
    "fig, ax = plt.subplots(1, 2, sharex=True, figsize=(7, 3))\n",
    "\n",
    "ax[0].plot([x_min, x_max], [y_min, y_max])\n",
    "ax[1].plot([x_min, x_max], [y_min, y_max])\n",
    "\n",
    "ax[0].scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], label='class 0', marker='o')\n",
    "ax[0].scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], label='class 1', marker='s')\n",
    "\n",
    "ax[1].scatter(X_test[y_test==0, 0], X_test[y_test==0, 1], label='class 0', marker='o')\n",
    "ax[1].scatter(X_test[y_test==1, 0], X_test[y_test==1, 1], label='class 1', marker='s')\n",
    "\n",
    "ax[1].legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Suggested exercises\n",
    "\n",
    "\n",
    "1. Train a zero-weight perceptron with different learning rates and compare the model parameters and decision boundaries to each other. What do you observe?\n",
    "\n",
    "2. Repeat the previous exercise with randomly initialized weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# %load solutions/01_weight_zero_learning_rate.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# %load solutions/02_random_weights_learning_rate.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing a Perceptron in TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Setting up the perceptron graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "g = tf.Graph()\n",
    "\n",
    "\n",
    "n_features = X_train.shape[1]\n",
    "\n",
    "with g.as_default() as g:\n",
    "    \n",
    "    # initialize model parameters\n",
    "    features = tf.placeholder(dtype=tf.float32, \n",
    "                              shape=[None, n_features], name='features')\n",
    "    targets = tf.placeholder(dtype=tf.float32, \n",
    "                             shape=[None, 1], name='targets')\n",
    "    params = {\n",
    "        'weights': tf.Variable(tf.zeros(shape=[n_features, 1], \n",
    "                                        dtype=tf.float32), name='weights'),\n",
    "        'bias': tf.Variable([[0.]], dtype=tf.float32, name='bias')}\n",
    "    \n",
    "    # forward pass\n",
    "    linear = tf.matmul(features, params['weights']) + params['bias']\n",
    "    ones = tf.ones(shape=tf.shape(linear)) \n",
    "    zeros = tf.zeros(shape=tf.shape(linear))\n",
    "    prediction = tf.where(tf.less(linear, 0.), zeros, ones, name='prediction')\n",
    "    \n",
    "    # weight update\n",
    "    diff = targets - prediction\n",
    "    weight_update = tf.assign_add(params['weights'], \n",
    "                                  tf.reshape(diff * features, (n_features, 1)))\n",
    "    bias_update = tf.assign_add(params['bias'], diff)\n",
    "    \n",
    "    saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Training the perceptron for 5 training samples for illustration purposes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model parameters:\n",
      " {'weights': array([[ 1.47257054],\n",
      "       [ 0.30436274]], dtype=float32), 'bias': array([[-1.]], dtype=float32)}\n",
      "Number of training errors: 3\n"
     ]
    }
   ],
   "source": [
    "with tf.Session(graph=g) as sess:\n",
    "    \n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    i = 0\n",
    "    for example, target in zip(X_train, y_train):\n",
    "        feed_dict = {features: example.reshape(-1, n_features),\n",
    "                     targets: target.reshape(-1, 1)}\n",
    "        _, _ = sess.run([weight_update, bias_update], feed_dict=feed_dict)\n",
    "        \n",
    "        i += 1\n",
    "        if i >= 4:\n",
    "            break\n",
    "        \n",
    "\n",
    "    modelparams = sess.run(params)    \n",
    "    print('Model parameters:\\n', modelparams)\n",
    "\n",
    "    saver.save(sess, save_path='perceptron')\n",
    "    \n",
    "    pred = sess.run(prediction, feed_dict={features: X_train})\n",
    "    errors = np.sum(pred.reshape(-1) != y_train)\n",
    "    print('Number of training errors:', errors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Continue training of the graph after restoring the session from a local checkpoint (this can be useful if we have to interrupt out computational session)\n",
    "- Now train a complete epoch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training errors 0\n",
      "Number of test errors 0\n"
     ]
    }
   ],
   "source": [
    "with tf.Session(graph=g) as sess:\n",
    "    saver.restore(sess, os.path.abspath('perceptron'))\n",
    "\n",
    "    for epoch in range(1):\n",
    "        for example, target in zip(X_train, y_train):\n",
    "            feed_dict = {features: example.reshape(-1, n_features),\n",
    "                         targets: target.reshape(-1, 1)}\n",
    "            _, _ = sess.run([weight_update, bias_update], feed_dict=feed_dict)\n",
    "            modelparams = sess.run(params)\n",
    "\n",
    "    saver.save(sess, save_path='perceptron')\n",
    "    \n",
    "    pred = sess.run(prediction, feed_dict={features: X_train})\n",
    "    train_errors = np.sum(pred.reshape(-1) != y_train)\n",
    "    pred = sess.run(prediction, feed_dict={features: X_train})\n",
    "    test_errors = np.sum(pred.reshape(-1) != y_train)\n",
    "\n",
    "    print('Number of training errors', train_errors)\n",
    "    print('Number of test errors', test_errors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Suggested Exercises\n",
    "\n",
    "\n",
    "3) Plot the decision boundary for this TensorFlow perceptron. Why do you think the TensorFlow implementation performs better than our NumPy implementation on the test set?\n",
    " - Hint 1: you can re-use the code that we used in the NumPy section\n",
    " - Hint 2: since the bias is a 2D array, you need to access the float value via `modelparams['bias'][0]`\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# %load solutions/03_tensorflow-boundary.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Theoretically, we could restart the Jupyter notebook now (we would just have to prepare the dataset again then, though)\n",
    "- We are going to restore the session from a meta graph (notice \"`tf.Session()`\")\n",
    "- First, we have to load the datasets again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training errors 0\n",
      "Number of test errors 0\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    \n",
    "    saver = tf.train.import_meta_graph(os.path.abspath('perceptron.meta'))\n",
    "    saver.restore(sess, os.path.abspath('perceptron'))\n",
    "    \n",
    "    pred = sess.run('prediction:0', feed_dict={'features:0': X_train})\n",
    "    train_errors = np.sum(pred.reshape(-1) != y_train)\n",
    "    pred = sess.run('prediction:0', feed_dict={'features:0': X_test})\n",
    "    test_errors = np.sum(pred.reshape(-1) != y_test)\n",
    "    \n",
    "    print('Number of training errors', train_errors)\n",
    "    print('Number of test errors', test_errors)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}