5.1-feed-forward.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "import pydot\n",
    "from IPython.display import Image, display, SVG\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A neural network, or \"multilayer perceptron\", is made up of units computing a nonlinear transformation of a weighted sum of their inputs. So the output of a unit looks something like:\n",
    "\n",
    "\\begin{equation}\n",
    "o(x_1,x_2,...x_n) = \\sigma(w_0 + w_1 x_1 + w_2 x_2 + ... w_n x_n)\n",
    "\\end{equation}\n",
    "\n",
    "$w_0$ represents a bias that does not depend on the input.\n",
    "\n",
    "Here the nonlinear function $\\sigma$ is the logistic sigmoid function, which as we shall see, is useful because it is a continuous function of its input and is differentiable. It \"squashes\" its input domain into a range between 0 and 1. (The tanh function may also be used, which has a range between -1 and 1). \n",
    "\n",
    "The definition and plot of the sigmoid function is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1160378d0>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXgV5dn48e99TnYSsrJDTECQRRYxgBsKKNS6gFrXasXd9nVptfYVtbZWa1/0tfX111orVRSXglVbQYsiiII7BEFZBcIawhIS1uwn5/79MQcMMfs5ySQ59+e65pqZZ56ZuQ/hnHvmmeURVcUYY0z48rgdgDHGGHdZIjDGmDBnicAYY8KcJQJjjAlzlgiMMSbMRbgdQFOkpaVpRkaG22EYY0ybsmzZsr2q2ql6eZtMBBkZGWRnZ7sdhjHGtCkisrWmcmsaMsaYMGeJwBhjwpwlAmOMCXNt8hpBTSoqKsjNzaW0tNTtUFq1mJgYevbsSWRkpNuhGGNaiXaTCHJzc0lISCAjIwMRcTucVklVKSgoIDc3l8zMTLfDMca0EiFpGhKR6SKyR0RW1bJcROT/ichGEflGRIZXWTZZRDYEhslNjaG0tJTU1FRLAnUQEVJTU+2syRhzjFBdI3gROLeO5T8E+gaGW4BnAEQkBfgtMAoYCfxWRJKbGoQlgfrZv5ExprqQNA2p6mIRyaijyiTgJXXeef2FiCSJSDdgDDBfVQsBRGQ+TkKZGYq4jDGtmCpUlkNFCfhKA0M5VJY55ZUV3439leD3fTeoPzBdCVrpzB8d9PvTqDNd15jvRt+bOeZ1/bWV17yuqqIKflX8R8ffTav/uzIlUBZYroEy1e+2k3b2HcQld23qv3qNWuoaQQ9ge5X53EBZbeXfIyK34JxNkJ6e3jxRhthNN93E3XffzcCBA5ttH+eddx7/+Mc/SEpKOqb8oYceIj4+nnvuuafZ9m3MUSX74eAOOLwbDu9xhuICKCmEkn1QehDKDkLZISgvCgyHAz/S7ZsEhsY2v/i15rP37YMv47g2mghq+kRaR/n3C1WnAdMAsrKy2kRvOs8991yz72Pu3LnNvg9jAKgohT1rnGHvBijYCIWb4cB250e+Ok8kxKVAbDLEJEJcGiRnQnQ8RMVDZBxExjpDRExgiAZvVGAc6WzjyNjjCYy9IF7weFHxcKBM2VdcQUGxn8KSCvYXV1BYUsn+Yh/7SnzsL63kQHEFB0p9HCr1cbDMR3klaODnR5HAwNH5o0SIifASEyHEREYQE+klKsJDdGQE0RGewLRTFuX1EOn1EBnhIdIjREV4iPA605FeD16vEOnxEOEVIrweIjyCVwSvR4jwCp7AtLdKuccjeAS8cmRaGNwjMeR/2pZKBLlAryrzPYG8QPmYauUftVBMIVVUVMTll19Obm4ulZWVPPjggzzzzDM88cQTZGVl8fzzz/PYY4/RvXt3+vbtS3R0NH/5y1+47rrriI2NZd26dWzdupUXXniBGTNm8PnnnzNq1ChefPFFAGbOnMkf/vAHVJXzzz+fxx57DPjudRtpaWk8+uijvPTSS/Tq1YtOnTpx8sknu/gvYtq8A7mw5VPY+gns+Ary1znNMeD8IKf0doaMMyCpF3TsAQldIb4LdEiD6I4QxDUpVecHPndfMTv2lZC7r4SdB0rZdbCYnQdK2XOwjPxDZZRX1nxWkRgbSVJcJElxUSTFd+D4TpEkxkaQEBNJQkwECdERxMdE0CEqgvjoCDpER9Ah2ktsVARxkV5io7xER3jC4rpaSyWCOcDtIjIL58LwAVXdKSLzgD9UuUA8Abgv2J397u3VrMmr4QglCAO7d+S3Fw6qdfl7771H9+7d+c9//gPAgQMHeOaZZwDIy8vjkUce4auvviIhIYFx48YxdOjQo+vu27ePhQsXMmfOHC688EI+/fRTnnvuOUaMGMGKFSvo3Lkz9957L8uWLSM5OZkJEybw1ltvcdFFFx3dxrJly5g1axbLly/H5/MxfPhwSwSmcfyVkLsU1r0D377rHPEDRCdCzyzoOwG6DYUuJ0JyBnhD8/NRXO4jZ08ROfmH2ZR/mJy9RWwtKGJrQTGHSn3H1I2L8tItMYauiTGM6p1Cp4RoOifE0CkhmrT4KNLio0npEEVSbCQRXntetqFC8pcUkZk4R/ZpIpKLcydQJICq/g2YC5wHbASKgesDywpF5BFgaWBTDx+5cNzWDB48mHvuuYd7772XCy64gNGjRx9dtmTJEs466yxSUlIAuOyyy1i/fv3R5RdeeCEiwuDBg+nSpQuDBw8GYNCgQWzZsoWtW7cyZswYOnVyXhp49dVXs3jx4mMSwccff8zFF19MXFwcABMnTmz2z2zaicJNsPwVWPEPOLTTOdrPHA1ZNzpH+10GOc0xQVJV8g6UsnrHAVbnHWTNzoOs332IbYXFR6+5egR6pcSRkdqB4enJpKfE0Ssljh5JsfRMjiUxNjIsjtBbWqjuGrqqnuUK3FbLsunA9FDEcURdR+7NpV+/fixbtoy5c+dy3333MWHChKPLtNY7CxzR0dEAeDyeo9NH5n0+HxERDfsz2RfENMqWT2DxE7DpQxAPHD8eJvwe+o532vSDVFJeyYrt+1m2tZAV2/ezYvt+9h4uB5wWo95pHTixeyKXnNSTfl3iOb5zPOmpcURHBJ90TOO0myeL3ZaXl0dKSgrXXHMN8fHxR9v2AUaOHMldd93Fvn37SEhI4M033zx61N8Qo0aN4uc//zl79+4lOTmZmTNncscddxxT58wzz+S6665jypQp+Hw+3n77bW699dZQfTzTnmxeDB9Nha2fQofOMPYBGHY1JNZ4w16DlVZUkr1lH5/m7OXznAJW7TiAz+8cBPXu1IEz+3ViWK8kBnVPZEC3BOKi7OentbC/RIisXLmSX/3qV3g8HiIjI3nmmWeO3rrZo0cP7r//fkaNGkX37t0ZOHAgiYkNP+Lq1q0b//M//8PYsWNRVc477zwmTZp0TJ3hw4dzxRVXMGzYMI477rhjmqaMAeBgHrw3BdbMhoTu8MPHYfi1zl07TbS1oIiF6/awcN0evtxcSLnPT4RHGNIzkZtG92ZkZjInp6eQGGfvtmrNpL5mi9YoKytLq3dMs3btWgYMGOBSRPU7fPgw8fHx+Hw+Lr74Ym644QYuvvhiV2Jp7f9WJsT8fljyLCz8vXPXz5n3wGl3OrdoNpKqsjrvIPNW7+K9VbvYsOcw4Bzxj+nXmTP6pjIyM5X4aDvGbI1EZJmqZlUvt79WC3nooYdYsGABpaWlTJgw4ZgLvcY0m+JC+NctsHG+cw3gvP+FlMa/cHDz3iLeWr6D2St2sKWgGI/AqMxUfjwqnXH9O3NcaodmCN60FEsELeSJJ55wOwQTbnKXweuTnad9z/+jcxdQI24oOFzm452v83gtezvLt+1HBE7tncpPz+rD+IFdSI1v/BmFaZ0sERjTHq2ZDW/eBPFd4Yb3oEfDnylZt+sgMz7bwuwVeRSXV3J853juP68/E4f2oGtiTDMGbdxiicCY9mbFP2D2bdBzBFw1y3nNQz38fmX+2t288OlmvthUSHSEh4lDu3PlyHSGpyfZrcntnCUCY9qTJX+HufdA5llw1UyIqrvtvtzn560VO3h2UQ45+UX0SIplyg/7c0VWL5I7RLVQ0MZtlgiMaS++etlJAiecB5e+AJG1N+OU+/y8vmw7Ty/cSN6BUgZ068j/u+okzjuxq72aIQzZXzxE9u/fz1//+tcmrz9mzBiq3xJrTINt+gje+QX0GQeXv1RrEqj0K69nb2fcHz/igX+voktiDC9cP4K5d57BxKHdLQmEKTsjCJEjieC//uu/3A7FhJv8b+G1ayG1L1z2ovPa5hosXp/PH+auZd2uQwzpmcgjF53ImH6drP3f2BlBqEyZMoWcnByGDRvGXXfdxdlnn83w4cMZPHgws2fPBmDLli0MGDCAm2++mUGDBjFhwgRKSkqObuP1119n5MiR9OvXj48//titj2LakqK98OqlzsNhV/+zxncEbdxziGunL+Ha6UsoLq/k6R8PZ/ZtpzP2hM6WBAzQXs8I3p0Cu1aGdptdB8MPp9a6eOrUqaxatYoVK1bg8/koLi6mY8eO7N27l1NOOeXo20A3bNjAzJkz+fvf/87ll1/Om2++yTXXXAOAz+djyZIlzJ07l9/97ncsWLAgtJ/BtC+qMPt2OLQbbngXko7tue9wmY8/f7CB5z/ZTFyUlwcvGMg1p6TbS93M97TPROAyVeX+++9n8eLFeDweduzYwe7duwHIzMxk2LBhAJx88sls2bLl6HqXXHJJjeXG1GjZi7D+XfjB/3zvOYF5q3fxm9mr2H2wjMuzenLvuf3tATBTq/aZCOo4cm8Jr776Kvn5+SxbtozIyEgyMjIoLS0FOOY1016v95imoSPLvF4vPt+xHXIYc4y9G2He/dB7DIz66dHiPYdK+e3s1by7ahf9uybw16tP5uTjkmvdjDHQXhOBCxISEjh06BDg9E7WuXNnIiMj+fDDD9m6davL0Zl2pbIC/nWzc13gor+Bx4OqMntFHr+ZvYpSn59f/eAEbjmzN5F2F5BpgFD1UHYu8BTgBZ5T1anVlj8JjA3MxgGdVTUpsKwSONKgv01V22TXWqmpqZx++umceOKJjBgxgnXr1pGVlcWwYcPo37+/2+GZ9uTzv0DeV3DZDOjYjX1F5Tzw1krmrtzF8PQk/veyofTpFO92lKYNCfo11CLiBdYD43E6o18KXKWqa2qpfwdwkqreEJg/rKqN+l/bFl9D3ZrYv1UbdnAn/Plk6DMWrnyVxevz+eXrX7O/uJy7xvfj1jP74PXYnUCmZs35GuqRwEZV3RTY0SxgElBjIgCuwunT2BjTWAt+C34fFec8wh/fXcffFuXQt3M8L14/gkHdg+9e0oSnUCSCHsD2KvO5wKiaKorIcUAmsLBKcYyIZAM+YKqqvlXLurcAtwCkp6fXVMWY9m3bl/DNaxwccSfXvraTFdv3c9XIdH5zwUBio+yWUNN0oUgENZ2H1tbedCXwhqpWVilLV9U8EekNLBSRlaqa870Nqk4DpoHTNFTTxlXVHpCpR1vskc7g9DL27n9TFtuFH2Rncdh/mKd/PJzzh3RzOzLTDoTiloJcoFeV+Z5AXi11rwRmVi1Q1bzAeBPwEXBSU4KIiYmhoKDAfujqoKoUFBQQE2PvlG9r/Cv+ATtXcO/BH5GYmMzbd5xhScCETCjOCJYCfUUkE9iB82P/4+qVROQEIBn4vEpZMlCsqmUikgacDjzelCB69uxJbm4u+fn5TVk9bMTExNCzZ0+3wzCNcKi4hJK5v2eXPxPPkMv598VDrCnIhFTQiUBVfSJyOzAP5/bR6aq6WkQeBrJVdU6g6lXALD32kH0A8KyI+HHOTqbWdrdRfSIjI8nMbHxfrMa0Zlv2FvHac49zr28nq4b/H3+cNMyaP03IBX37qBtqun3UmPbm0417uf2Vpfybu+iUnESHOz9vVJ/DxlRX2+2j9tihMa3QrCXbmDx9CZfFLiWDnXQ4Z4olAdNs7BUTxrQifr/y+Lxv+duiHM7qm8q9Je9AfH8Y0CYfuDdthJ0RGNNKlFZUcses5fxtUQ5Xj0pn+qhdePd+C2f+Cjz2VTXNx84IjGkFDhRXcPPL2SzZXMj95/Xn5tG9kem/hORMGHSx2+GZds4SgTEu23mghMnTl7B5bxH/76qTmDi0O+z8BrZ/CRMeBY/dKmqalyUCY1y0cc8hfvL8Eg6X+phxw0hO65PmLFj6HETEwklXuxugCQuWCIxxyYrt+7n+hSV4PR5eu/VUBnbv6Cwo2Q8rX4fBl0KsdSpjmp8lAmNc8PGGfG59eRlp8dG8fONIjkvt8N3CFf+AimIYebN7AZqwYonAmBb23qpd3DlzOb07deClG0bSuWOVdz/5/U6zUM+R0G2oe0GasGL3pBnTgt5clstt//iKQT068totpx6bBAA2fQiFOXY2YFqUJQJjWshLn2/hl69/zSm9U3jlxlEkxkV+v9KyFyAuDQZOavH4TPiyRGBMC/jbohx+M3s15wzowvOTR9AhuoZW2ZJ9sH4eDLnc6ZjemBZi1wiMaUaqyv8t2MBTH2zgwqHd+dPlQ4n01nL8tfrfUFkOQ65o2SBN2LNEYEwzUVWmvruOZxdv4rKTezL1R0Pq7lj+69cg7QS7SGxanDUNGdMMVJXfvb2GZxdv4tpTj+Ox+pJA4WbY/gUMvcLeMmpanJ0RGBNifr/ywFurmLlkGzePzuT+8wbU35nMyted8eDLmz9AY6oJyRmBiJwrIt+KyEYRmVLD8utEJF9EVgSGm6osmywiGwLD5FDEY4xbKv3Kr974hplLtnHb2D4NSwKq8PUsyBgNSb3qrmtMMwj6jEBEvMDTwHicjuyXisicGrqcfE1Vb6+2bgrwWyALUGBZYN19wcZlTEvzVfq5+59fM+frPH5xTl9+fnbfhnUruWOZ8+zAGXc1f5DG1CAUZwQjgY2quklVy4FZQENvgv4BMF9VCwM//vOBc0MQkzEtqqLSzx0zlzPn6zzuPbc/vzinX8P7Fv7mNYiIsWcHjGtCkQh6ANurzOcGyqr7kYh8IyJviMiR89+GrouI3CIi2SKSnZ+fH4KwjQmNMl8lP3vlK95dtYtfnz+An43p0/CV/X5YMxv6ToCYjs0XpDF1CEUiqOmwR6vNvw1kqOoQYAEwoxHrOoWq01Q1S1WzOnXq1ORgjQml0opKbn15GQvW7ubhSYO4aXTvxm0gdwkc3m1nA8ZVoUgEuUDVK1w9gbyqFVS1QFXLArN/B05u6LrGtFbF5T5unLGURevzmXrJYK49NaPxG1n7NnijnDMCY1wSikSwFOgrIpkiEgVcCcypWkFEulWZnQisDUzPAyaISLKIJAMTAmXGtGqHy3xcN30pn+cU8MfLhnLlyPTGb0QV1syB3mOtWci4Kui7hlTVJyK34/yAe4HpqrpaRB4GslV1DnCniEwEfEAhcF1g3UIReQQnmQA8rKqFwcZkTHM6UFzBtS8sYfWOAzx15UlcOLR70za082s4sA3O+u/QBmhMI4XkgTJVnQvMrVb2myrT9wH31bLudGB6KOIwprkVHC7jJ88vYeOew/z16uFMGNS16Rtb+zaIF044L3QBGtME9mSxMQ20+2ApVz/3JdsLi/n75CzO6hfkTQtr50DG6dAhNTQBGtNE9q4hYxpge2Exl/3tc3buL+HF60cGnwTyv4W962HAxNAEaEwQ7IzAmHps2H2Ia57/ktIKP6/efArDeiUFv9E1gfsp+l8Q/LaMCZIlAmPqsDL3AJNfWIJHhNduPYX+XUN0d8+6d5x+iTt2q7+uMc3MmoaMqcVnG/dy5bTPiY308vpPTw1dEji0C3augBPsbSqmdbAzAmNq8O7Knfx81goy0uJ46YZRdE2MqX+lhtow3xn3/UHotmlMECwRGFPNq19u5cG3VjGsVxLTrxtBUlxUaHewYR507AFdBoV2u8Y0kSUCYwJUlT/NX8+fF25k7AmdePrq4cRFhfgr4iuHnI9g8I+sJzLTalgiMAbnNdL3/2slry/L5coRvfj9RScSUVsn88HY9hmUH7JmIdOqWCIwYe9gaQW3vfoVH2/Y27gOZZpi/fvgjYbeZzXP9o1pAksEJqxtLyzmxhlL2ZRfxOOXDuHyrGbuKnLDPMg4A6I6NO9+jGkESwQmbC3fto+bX8qm3OfnpRtGctrxac27w4IcKNgII29p3v0Y00iWCExYenNZLvf9eyVdOkYz65ZTOL5zQvPvdMP7zrjv+ObflzGNYInAhBVfpZ+p767juU82c1qfVJ7+8XCSO4T49tDarJ8HqX0hpZG9mBnTzCwRmLBRcLiMn89awScb93LdaRk8cP4AIpvjzqCaVJTA1s9gxI0tsz9jGiEk3wIROVdEvhWRjSIypYbld4vImkDn9R+IyHFVllWKyIrAMKf6usaEwvJt+7jwz5+wZEshj/9oCA9NHNRySQCcJFBZBn3Obrl9GtNAQZ8RiIgXeBoYj9MH8VIRmaOqa6pUWw5kqWqxiPwMeBy4IrCsRFWHBRuHMTVRVV7+YiuPvLOGLh1j+NfPTuPEHoktH0jOQqdv4uNOa/l9G1OPUDQNjQQ2quomABGZBUwCjiYCVf2wSv0vgGtCsF9j6rS/uJz/fuMb3l+zm7EndOLJK4aF/nURDZXzIaSfAlFx7uzfmDqE4ty4B7C9ynxuoKw2NwLvVpmPEZFsEflCRC6qbSURuSVQLzs/Pz+4iE27t3RLIec99TEffruHX58/gOcnN8M7gxrq0C7Ysxr6jHNn/8bUIxRnBDU9gqk1VhS5BsgCqj5Wma6qeSLSG1goIitVNed7G1SdBkwDyMrKqnH7xpT5Knly/gaeXZxDr+Q43vjpaQwNRUcywdj0kTPuPdbVMIypTSgSQS5Q9XHMnkBe9Uoicg7wAHCWqpYdKVfVvMB4k4h8BJwEfC8RGFOftTsPctdrK1i36xBXjezFA+cPJD66FdwYl7MQ4lKh6xC3IzGmRqH4liwF+opIJrADuBL4cdUKInIS8CxwrqruqVKeDBSrapmIpAGn41xINqbBynyV/GXhRp75KIekuCien5zF2QO6uB2WQ9W5PtB7LHisHyjTOgWdCFTVJyK3A/MALzBdVVeLyMNAtqrOAf4XiAdeD7zMa5uqTgQGAM+KiB/nesXUancbGVOnpVsKmfLmN+TkF3HJ8B48eP7AlntArCF2r4aiPdDHmoVM6xWS82ZVnQvMrVb2myrT59Sy3mfA4FDEYMJL/qEypr67jje/yqVHUiwzbhjJWf06uR3W920K3DBn1wdMK9YKGlCNabhyn59XvtjKk/PXU+qr5Gdj+nD72OPp0BquBdQkZyF06g+Jdd1IZ4y7Wum3x5hjqSrzVu/msffWsXlvEaP7pvHQxEH06RTvdmi1qyh1nig++Xq3IzGmTpYITKv3eU4Bf3z/W7K37qNv53heuG4EY07o1Hydx4RK7hLwlVonNKbVs0RgWq3sLYX8af56PsspoHNCNI9efCJXZPVqni4km8PmxSBeOO50tyMxpk6WCEyroqp8tD6fZz7KYcnmQtLio3jwgoFcPSqdmEiv2+E1zqZF0P0kiOnodiTG1MkSgWkVSisqmfN1Hi98uoW1Ow/SLTGGBy8YyFUjexEX1Qb/m5YehB3L4IxfuB2JMfVqg98w055sLyxm1tJtzFyyncKick7oksATlw1l4tDuREW0kSagmmz9DLQSMu36gGn9LBGYFldaUcn8Nbv5Z/Z2Ptm4FwHOGdCF607P4NTeqa3/InBDbF4EETHQa5TbkRhTL0sEpkX4Kv18ubmQt5bv4L1VuzhU5qNHUiy/OLsfl2X1pHtSrNshhtbmxU4SiIxxOxJj6mWJwDSbMl8ln+UUMG/VLt5fs5vConLioyM498SuTBrWndP6pOH1tIOj/+oO58PuVTDuQbcjMaZBLBGYkNqxv4TF6/NZuG4Pn27cS3F5JR2ivIwb0IUfntiVcf07t727fxpry2Jn3HuMm1EY02CWCExQdh8sZcnmQr7YVMCnG/eypaAYgB5JsVwyvAfj+nfmtD5p7f/Hv6pNiyC6I3SzHlhN22CJwDRYma+StTsPsWLbPlZs38+ybfvYXlgCQHx0BKMyU/jJqRmM7ptG387x7eOib1NsXgwZZ4DXvl6mbbD/qeZ7VJX8Q2V8u/sQ3+46xNqdh1idd4CNew7j8zudw3VOiOak9CQmn5rByMwUBnbr2Hae+G1O+7fBvs0w6la3IzGmwSwRhLGDpRVsKyhmS0ERWwuK2by3iJz8w2zKL+JAScXRep0SohnUvSNnD+jMid0TGZaeRLfEdnaXT6hs/tgZZ57pbhzGNIIlgnbI71cOlFSw51AZew6VsudgGbsOlrLrQCk7D5SQu6+EHftLOFTqO2a9zgnR9O7UgfOHdKNf53j6dU2gX5cE0uKjXfokbdDmxRCXBp0GuB2JMQ0WkkQgIucCT+H0UPacqk6ttjwaeAk4GSgArlDVLYFl9wE3ApXAnao6LxQxtQcVlX6KynwcKj0yVHCw1MfBkgr2l1RwoKSC/cXl7CuuYF9ROQVF5RQcLqOwqPxoE05VSXGRdO0YQ8/kWEZlptAjOZb0lDjSUzqQnhrXOvr3bctUnUSQOdq6pTRtStDffBHxAk8D43E6sl8qInOqdTl5I7BPVY8XkSuBx4ArRGQgTh/Hg4DuwAIR6aeqlcHG1RSqil/B5/fj9zvjSr9SUalU+hWf34+v0hlXVCq+SqXC76fC58fnV8p9fsor/c642nSZr5Iyn5/SikpKK5xxSUUlpRWVFJc7Q0l5JcUVPorKKikq81Hm89cZrwh0jIkkpUMUyXGRdE+MYUiPRFLjo0iNj6ZzQmDoGEPXjjHERoXRnTtuKMiBQ3nWLGTanFAcAo4ENqrqJgARmQVMAqomgknAQ4HpN4C/iHNLySRglqqWAZtFZGNge5+HIK7vuXPmcj7ekE+lX1GFSnV+4P1Hx82x1+94BGIivURHeIiJ9BIb6SU60ktclJeEmAg6J0QTF+WlQ3QE8dERxEVFkBATQXyMM58YG0nHmEg6xkaQFBtFfExE+3wgq6068vxAhiUC07aEIhH0ALZXmc8Fqr9g5WidQGf3B4DUQPkX1datsU8/EbkFuAUgPT29SYFmZSSTGBuJ1yN4RPAIzrRH8IozjvAI3sAQEagX6RW8Hg8RHiHCK0R4PUR6nHGEV4j0eIiK8BDpFSK9HqIjjsw709GRXqK8zvKwvaUyHGxeDAndIbWP25EY0yihSAQ1/bJVP7aurU5D1nUKVacB0wCysrKadOx+7akZTVnNmPr5/c4dQ8ef47TZGdOGhOKKVi7Qq8p8TyCvtjoiEgEkAoUNXNeY1i9/LRTvtesDpk0KRSJYCvQVkUwRicK5+DunWp05wOTA9KXAQlXVQPmVIhItIplAX2BJCGIypmUdfX5gtLtxGNMEQTcNBdr8bwfm4dw+Ol1VV4vIw0C2qs4BngdeDlwMLsRJFgTq/RPnwrIPuM2tO4aMCcrmxZCcAUlNu35ljJtCcuO4qs4F5lYr+/mE9O8AABDtSURBVE2V6VLgslrWfRR4NBRxGOMKfyVs+QQGTXI7EmOaxJ56MSZYO1dA2QHrltK0WZYIjAnW5sDzA3ah2LRRlgiMCdamRc67heI7ux2JMU1iicCYYPjKYNsX0NuahUzbZYnAmGDkLgVfiV0fMG2aJQJjgrFpEYgHjjvN7UiMaTJLBMYEY/Mi6H4SxCa5HYkxTWaJwJimKjsEO5ZZs5Bp8ywRGNNUWz8Hv88uFJs2zxKBMU21eRF4o6FX9beuG9O2WCIwpqk2L4JeIyEy1u1IjAmKJQJjmqKoAHattGYh0y5YIjCmKTZ/5Iwzx7gZhTEhYYnAmKbIWQgxidBjuNuRGBM0SwTGNJYq5Hzo3Dbq8bodjTFBs0RgTGPt3QAHd0CfcW5HYkxIBJUIRCRFROaLyIbAOLmGOsNE5HMRWS0i34jIFVWWvSgim0VkRWAYFkw8xrSInIXOuM9Yd+MwJkSCPSOYAnygqn2BDwLz1RUD16rqIOBc4P9EpOrz+L9S1WGBYUWQ8RjT/HIWQkpvp2tKY9qBYBPBJGBGYHoGcFH1Cqq6XlU3BKbzgD1ApyD3a4w7fGWw5WNrFjLtSrCJoIuq7gQIjOvsmUNERgJRQE6V4kcDTUZPikh0HeveIiLZIpKdn58fZNjGNNH2JVBRbInAtCv1JgIRWSAiq2oYGtVTt4h0A14GrldVf6D4PqA/MAJIAe6tbX1VnaaqWaqa1amTnVAYl2z6EMQLGaPdjsSYkImor4KqnlPbMhHZLSLdVHVn4Id+Ty31OgL/AX6tql9U2fbOwGSZiLwA3NOo6I1paTkLoecIiOnodiTGhEywTUNzgMmB6cnA7OoVRCQK+Dfwkqq+Xm1Zt8BYcK4vrAoyHmOaT3Eh5K2wZiHT7gSbCKYC40VkAzA+MI+IZInIc4E6lwNnAtfVcJvoqyKyElgJpAG/DzIeY5rPxg8AtURg2p16m4bqoqoFwNk1lGcDNwWmXwFeqWV9+0aZtmPDPIhLtddKmHbHniw2piH8lbBxARw/3l4rYdodSwTGNERuNpTsg34T3I7EmJCzRGBMQ2yY59w2atcHTDtkicCYhlj/vtMlZez3XqdlTJtnicCY+hzYAbtXWrOQabcsERhTnw3vO+O+P3A3DmOaiSUCY+qzYT4k9oLOA9yOxJhmYYnAmLr4ymDTR9B3Aoi4HY0xzcISgTF12fIxVBQ5icCYdsoSgTF1Wfs2RHaA3me5HYkxzcYSgTG18VfCuv84dwtFxrodjTHNxhKBMbXZ9gUU5cOAC92OxJhmZYnAmNqsfRu80XZ9wLR7lgiMqYmqkwj6jIPoBLejMaZZWSIwpiZ5X8HBXGsWMmEhqEQgIikiMl9ENgTGNb6IRUQqq3RKM6dKeaaIfBlY/7VAb2bGuG/t285L5k74oduRGNPsgj0jmAJ8oKp9gQ8C8zUpUdVhgWFilfLHgCcD6+8DbgwyHmOCpwpr5kDmaIhLcTsaY5pdsIlgEjAjMD0Dp9/hBgn0UzwOeKMp6xvTbPashcIcaxYyYSPYRNBFVXcCBMada6kXIyLZIvKFiBz5sU8F9quqLzCfC/SobUcicktgG9n5+flBhm1MHVa9CeKB/pYITHiot89iEVkAdK1h0QON2E+6quaJSG9gYaDD+oM11NPaNqCq04BpAFlZWbXWMyYofj9880/oPQYSurgdjTEtot5EoKrn1LZMRHaLSDdV3Ski3YA9tWwjLzDeJCIfAScBbwJJIhIROCvoCeQ14TMYEzrbPocD22Dcr92OxJgWE2zT0BxgcmB6MjC7egURSRaR6MB0GnA6sEZVFfgQuLSu9Y1pUd/Mct4tNOACtyMxpsUEmwimAuNFZAMwPjCPiGSJyHOBOgOAbBH5GueHf6qqrgksuxe4W0Q24lwzeD7IeIxpuopSWD3buUgc1cHtaIxpMfU2DdVFVQuAs2sozwZuCkx/BgyuZf1NwMhgYjAmZNa/C2UHYOgVbkdiTIuyJ4uNOeLr1yChG2TaK6dNeLFEYAxA0V7YOB8GXwoer9vRGNOiLBEYA7DyDfD7YMiVbkdiTIuzRGCMKmQ/Dz1Ohq4nuh2NMS3OEoExmxfB3vUw4ia3IzHGFZYIjFnyd4hNgUGXuB2JMa6wRGDC24Fc+HYuDP8JRMa4HY0xrrBEYMJb9gvONYKsG9yOxBjXWCIw4ctXBl/NgH4/gOQMt6MxxjWWCEz4WjMHivJhxM1uR2KMqywRmPDk98MnT0JqX6eDemPCmCUCE56+nQt7VsOZvwKPfQ1MeLNvgAk/qrDoMUjOhBN/5HY0xrjOEoEJPxveh13fwJn3gDeoF/Aa0y5YIjDh5cjZQFI6DLHXTRsDlghMuMlZCDuWwRl3gzfS7WiMaRWCSgQikiIi80VkQ2CcXEOdsSKyospQKiIXBZa9KCKbqywbFkw8xtTJXwkLfgsde8KwH7sdjTGtRrBnBFOAD1S1L/BBYP4Yqvqhqg5T1WHAOKAYeL9KlV8dWa6qK4KMx5jaffUS7FoJEx6GiGi3ozGm1Qg2EUwCZgSmZwAX1VP/UuBdVS0Ocr/GNE7JPlj4CBx3ur1czphqgk0EXVR1J0Bg3Lme+lcCM6uVPSoi34jIkyJS62GaiNwiItkikp2fnx9c1Cb8fPSYkwzOnQoibkdjTKtSbyIQkQUisqqGYVJjdiQi3XA6sZ9Xpfg+oD8wAkgB7q1tfVWdpqpZqprVqVOnxuzahLs9a2HJNDj5Oug2xO1ojGl16r2JWlXPqW2ZiOwWkW6qujPwQ7+njk1dDvxbVSuqbHtnYLJMRF4A7mlg3MY0jL8S3rkbouNh7K/djsaYVinYpqE5wOTA9GRgdh11r6Jas1AgeSAignN9YVWQ8RhzrE+fgm2fwQ8fhw6pbkdjTKsUbCKYCowXkQ3A+MA8IpIlIs8dqSQiGUAvYFG19V8VkZXASiAN+H2Q8Rjznbzl8OGjMOhie3jMmDoE9Xy9qhYAZ9dQng3cVGV+C9Cjhnr22kfTPMqL4c2boUNnOP9PdoHYmDrYi1ZM+6MK702Bgg1w7WyIS3E7ImNaNXvFhGl/vvir0/PYGXdD7zFuR2NMq2eJwLQv6/4D8x6AARNh3INuR2NMm2CJwLQfecvhzZugx3C4ZJp1OGNMA9k3xbQPO7+GV34EcWlw1SyIjHU7ImPaDEsEpu3b9iW8eCFExMK1b0F8fW86McZUZYnAtG05H8LLFzkPi93wHqT2cTsiY9ocSwSmbVKFz/7iNAclZ8D170FSL7ejMqZNsucITNtTegDe+i9Y9w70vwAu+ivEJLodlTFtliUC07ZsXOC8RO5ALkx4FE69zZ4aNiZIlghM23AwD967D9a8BanHw/VzIf0Ut6Mypl2wRGBat8N74LM/w9LnQSudV0mffqd1NWlMCFkiMK1T/nrIfh6WzYDKMjjxUhh7H6T0djsyY9odSwSm9SjZB+vmwvKXYdvn4IlwXh89+pd2W6gxzcgSgXGPKuSvg02L4Nv/wJZPneaf1ONh/MMw9Cp7OMyYFmCJwLScw/mw62vndRA7vnKO+osLnGVpJzht//0vgB4n251AxrSgoBKBiFwGPAQMAEYGOqSpqd65wFOAF3hOVY/0ZJYJzMLpuP4r4CeqWh5MTMZF5cVweBcc3Onc3nlgG+zbAns3On0DHPnRB+chsL4/gIzT4bjTISXTraiNCXvBnhGsAi4Bnq2tgoh4gadxurLMBZaKyBxVXQM8BjypqrNE5G/AjcAzQcZkaqIKfp/TmbvfB/4KZ7qyAirLA+My8JWCr9wZV5SAr8T5gS8vgooiKDsEpQeh7CCU7Hd+3EsKoagAyg99f78dOkNaX+dIP60fdBsKXQdDbFLL/xsYY2oUbFeVawGk7tP4kcBGVd0UqDsLmCQia4FxwI8D9WbgnF00XyJ45y7Y+tl386pBbKyWdY/ZpjayPLDsaJkGqh4pqzr2V5v2f1d2ZNpf6bS5+ytrj7exPJEQ0xGiOzo/5rEpzoXcuFSI7+IMCV0hKR0Se9pbQI1pA1riGkEPYHuV+VxgFJAK7FdVX5Xy7/VrfISI3ALcApCent60SBJ7QqcTqm+5adtygqptQS11GlB+dJkcO310HKgvnu/KPF5n/phpAtNeZ+yJCEx7nB9zT4QzeCPBG+WMI6LBGw0RUc6bPCNjnHFUHETFQ1QHp6613xvTrtSbCERkAdC1hkUPqOrsBuyjpl8NraO8Rqo6DZgGkJWV1bTD29G/bNJqxhjTntWbCFT1nCD3kQtUfS1kTyAP2AskiUhE4KzgSLkxxpgW1BKvoV4K9BWRTBGJAq4E5qiqAh8ClwbqTQYacoZhjDEmhIJKBCJysYjkAqcC/xGReYHy7iIyFyBwtH87MA9YC/xTVVcHNnEvcLeIbMS5ZvB8MPEYY4xpPNGg7pxxR1ZWlmZn1/jIgjHGmFqIyDJVzapebj2UGWNMmLNEYIwxYc4SgTHGhDlLBMYYE+ba5MViEckHtjZx9TScZxjag/byWdrL5wD7LK1Ve/kswX6O41S1U/XCNpkIgiEi2TVdNW+L2stnaS+fA+yztFbt5bM01+ewpiFjjAlzlgiMMSbMhWMimOZ2ACHUXj5Le/kcYJ+ltWovn6VZPkfYXSMwxhhzrHA8IzDGGFOFJQJjjAlzYZsIROQOEflWRFaLyONuxxMsEblHRFRE0tyOpSlE5H9FZJ2IfCMi/xaRNtepsYicG/g/tVFEprgdT1OJSC8R+VBE1ga+Hz93O6ZgiIhXRJaLyDtuxxIMEUkSkTcC35O1InJqqLYdlolARMYCk4AhqjoIeMLlkIIiIr2A8cA2t2MJwnzgRFUdAqwH7nM5nkYRES/wNPBDYCBwlYgMdDeqJvMBv1TVAcApwG1t+LMA/BznFfht3VPAe6raHxhKCD9TWCYC4GfAVFUtA1DVPS7HE6wngf8mZD3UtzxVfb9K/9Vf4PRY15aMBDaq6iZVLQdm4RxstDmqulNVvwpMH8L5wam1P/HWTER6AucDz7kdSzBEpCNwJoE+W1S1XFX3h2r74ZoI+gGjReRLEVkkIiPcDqipRGQisENVv3Y7lhC6AXjX7SAaqQewvcp8Lm30x7MqEckATgK+dDeSJvs/nIMkv9uBBKk3kA+8EGjmek5EOoRq4/X2WdxWicgCoGsNix7A+dzJOKe9I4B/ikhvbaX30tbzWe4HJrRsRE1T1+dQ1dmBOg/gNE282pKxhYDUUNYq/z81lIjEA28Cv1DVg27H01gicgGwR1WXicgYt+MJUgQwHLhDVb8UkaeAKcCDodp4u6Sq59S2TER+Bvwr8MO/RET8OC9zym+p+Bqjts8iIoOBTOBrEQGnOeUrERmpqrtaMMQGqetvAiAik4ELgLNba1KuQy7Qq8p8TyDPpViCJiKROEngVVX9l9vxNNHpwEQROQ+IATqKyCuqeo3LcTVFLpCrqkfOzN7ASQQhEa5NQ28B4wBEpB8QRRt8M6GqrlTVzqqaoaoZOP9ZhrfGJFAfETkXpw/riapa7HY8TbAU6CsimSISBVwJzHE5piYR56jieWCtqv7J7XiaSlXvU9Wege/GlcDCNpoECHynt4vICYGis4E1odp+uz0jqMd0YLqIrALKgclt8Ai0vfkLEA3MD5zdfKGqP3U3pIZTVZ+I3A7MA7zAdFVd7XJYTXU68BNgpYisCJTdr6pzXYzJwB3Aq4EDjU3A9aHasL1iwhhjwly4Ng0ZY4wJsERgjDFhzhKBMcaEOUsExhgT5iwRGGNMmLNEYIwxYc4SgTHGhLn/D6mCWkSTYb9YAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def sigmoid(x):\n",
    "    return 1./(1 + exp(-x))\n",
    "\n",
    "xs = linspace(-6, 6, 100)\n",
    "plot(xs, sigmoid(xs), label='sigmoid')\n",
    "plot(xs, tanh(xs), label='tanh')\n",
    "\n",
    "legend(loc=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These units are grouped into layers, each having multiple units, and with the outputs of the units of one layer feeding into the units of the next layer.\n",
    "\n",
    "This is easiest to show graphically, so we define a function to create a network diagram:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def viewnet(layers, weights=None):\n",
    "    names = ['Input'] + ['Hidden %d' % l for l in range(1,len(layers)-1)] + ['Output']\n",
    "    labels = ['x'] + ['z'] * (len(layers)-2) + ['y']\n",
    "    gr = pydot.Dot(graph_type='graph', rankdir='LR', splines='line', fontname='sans-serif', nodesep=0.5, ranksep=2)\n",
    "    weights = np.array(weights) if weights is not None else None\n",
    "\n",
    "    layernodes = []\n",
    "    \n",
    "    def wbox(s):\n",
    "        return '''<<TABLE BGCOLOR=\"white\" BORDER=\"0\" CELLBORDER=\"0\" CELLSPACING=\"0\">\n",
    "                  <TR><TD><FONT COLOR=\"red\" POINT-SIZE=\"10\">%s</FONT></TD></TR></TABLE>>''' % s\n",
    "\n",
    "    # Create our layers\n",
    "    for l, height in enumerate(layers):\n",
    "        layer = pydot.Cluster(graph_name='Layer%d' % l, label=names[l], margin='20')\n",
    "        \n",
    "        # HSV colour, split the hue among layers\n",
    "        layer.set_bgcolor('%f,.4,1' % (float(l)/len(layers)))\n",
    "        \n",
    "        nodenames = []\n",
    "        \n",
    "        # Create our units\n",
    "        for k in range(height):\n",
    "            name = '%d %s' % (k+1, names[l])\n",
    "            label = labels[l] + str(k+1)\n",
    "            tooltip = ''\n",
    "            xlabel = ''\n",
    "            if l > 0 and weights is not None:\n",
    "                # Show bias on hover\n",
    "                tooltip = 'bias=%+d' % weights[l-1][k,0]\n",
    "                xlabel = wbox('bias %+d' % (weights[l-1][k,0]))\n",
    "                \n",
    "            node = pydot.Node(name, label=label, regular=True, fontname='sans-serif',\n",
    "                              tooltip=tooltip, xlabel=xlabel)\n",
    "            layer.add_node(node)\n",
    "            nodenames.append(name)\n",
    "            \n",
    "        layernodes.append(nodenames)\n",
    "        gr.add_subgraph(layer)\n",
    "    \n",
    "        if l > 0:\n",
    "            # Add edges between previous and this layer\n",
    "            for i1, n1 in enumerate(layernodes[l-1]):\n",
    "                for i2, n2 in enumerate(layernodes[l]):\n",
    "                    # hack to keep the node orders sequential\n",
    "                    # only 'constrain' the edge between middle nodes\n",
    "                    constrain = i1 == floor(len(layernodes[l-1])/2.) and i2 == floor(height/2.)\n",
    "                    tooltip = 'w%d%d' % (i2+1,i1+1)\n",
    "                    wt = ''\n",
    "                    if weights is not None:\n",
    "                        tooltip += '=%d' % weights[l-1][i2,i1+1] # +1 for bias\n",
    "                        wt = '     %d     ' % weights[l-1][i2,i1+1]\n",
    "                    gr.add_edge(pydot.Edge(n1, n2, tooltip=tooltip, dir='forward',\n",
    "                                           constraint=constrain, xlabel=wt))\n",
    "    # remove superfluous tooltips\n",
    "    svg = re.sub('<title>.*</title>', '', gr.create_svg().decode(\"utf-8\") )\n",
    "    return SVG(svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we show a layer with 4 of these sigmoid units, along with an \"input\" layer that represents a 2D sample point. (Not showing the biases)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"355pt\" viewBox=\"0.00 0.00 288.59 355.00\" width=\"289pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 351)\">\n",
       "\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-351 284.5928,-351 284.5928,4 -4,4\" stroke=\"transparent\"/>\n",
       "<g class=\"cluster\" id=\"clust1\">\n",
       "\n",
       "<polygon fill=\"#ff9999\" points=\"8,-84 8,-263 88.2964,-263 88.2964,-84 8,-84\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"48.1482\" y=\"-247.8\">Input</text>\n",
       "</g>\n",
       "<g class=\"cluster\" id=\"clust2\">\n",
       "\n",
       "<polygon fill=\"#99ff99\" points=\"192.2964,-8 192.2964,-339 272.5928,-339 272.5928,-8 192.2964,-8\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4446\" y=\"-323.8\">Hidden 1</text>\n",
       "</g>\n",
       "<!-- 1 Input -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "\n",
       "<ellipse cx=\"48.1482\" cy=\"-200\" fill=\"none\" rx=\"20.2975\" ry=\"20.2975\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"48.1482\" y=\"-196.3\">x1</text>\n",
       "</g>\n",
       "<!-- 1 Hidden 1 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "\n",
       "<ellipse cx=\"232.4446\" cy=\"-276\" fill=\"none\" rx=\"20.2975\" ry=\"20.2975\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4446\" y=\"-272.3\">z1</text>\n",
       "</g>\n",
       "<!-- 1 Input&#45;&#45;1 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "\n",
       "<g id=\"a_edge1\"><a xlink:title=\"w11\">\n",
       "<path d=\"M66.8338,-207.7055C99.2593,-221.0771 166.1232,-248.6504 204.2283,-264.3642\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"203.1305,-267.6973 213.7096,-268.2741 205.7992,-261.226 203.1305,-267.6973\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- 2 Hidden 1 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "\n",
       "<ellipse cx=\"232.4446\" cy=\"-200\" fill=\"none\" rx=\"20.2975\" ry=\"20.2975\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4446\" y=\"-196.3\">z2</text>\n",
       "</g>\n",
       "<!-- 1 Input&#45;&#45;2 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "\n",
       "<g id=\"a_edge2\"><a xlink:title=\"w21\">\n",
       "<path d=\"M68.5753,-200C100.9034,-200 164.0528,-200 201.819,-200\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"202.207,-203.5001 212.2069,-200 202.2069,-196.5001 202.207,-203.5001\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- 3 Hidden 1 -->\n",
       "<g class=\"node\" id=\"node5\">\n",
       "\n",
       "<ellipse cx=\"232.4446\" cy=\"-124\" fill=\"none\" rx=\"20.2975\" ry=\"20.2975\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4446\" y=\"-120.3\">z3</text>\n",
       "</g>\n",
       "<!-- 1 Input&#45;&#45;3 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "\n",
       "<g id=\"a_edge3\"><a xlink:title=\"w31\">\n",
       "<path d=\"M66.8338,-192.2945C99.2593,-178.9229 166.1232,-151.3496 204.2283,-135.6358\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"205.7992,-138.774 213.7096,-131.7259 203.1305,-132.3027 205.7992,-138.774\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- 4 Hidden 1 -->\n",
       "<g class=\"node\" id=\"node6\">\n",
       "\n",
       "<ellipse cx=\"232.4446\" cy=\"-48\" fill=\"none\" rx=\"20.2975\" ry=\"20.2975\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4446\" y=\"-44.3\">z4</text>\n",
       "</g>\n",
       "<!-- 1 Input&#45;&#45;4 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge4\">\n",
       "\n",
       "<g id=\"a_edge4\"><a xlink:title=\"w41\">\n",
       "<path d=\"M63.8612,-187.0406C96.2318,-160.3427 170.5139,-99.0779 208.7486,-67.5435\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"211.2837,-69.9895 216.7714,-60.9266 206.8298,-64.5893 211.2837,-69.9895\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- 2 Input -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "\n",
       "<ellipse cx=\"48.1482\" cy=\"-124\" fill=\"none\" rx=\"20.2975\" ry=\"20.2975\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"48.1482\" y=\"-120.3\">x2</text>\n",
       "</g>\n",
       "<!-- 2 Input&#45;&#45;1 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge5\">\n",
       "\n",
       "<g id=\"a_edge5\"><a xlink:title=\"w12\">\n",
       "<path d=\"M63.8612,-136.9594C96.2318,-163.6573 170.5139,-224.9221 208.7486,-256.4565\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"206.8298,-259.4107 216.7714,-263.0734 211.2837,-254.0105 206.8298,-259.4107\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- 2 Input&#45;&#45;2 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge6\">\n",
       "\n",
       "<g id=\"a_edge6\"><a xlink:title=\"w22\">\n",
       "<path d=\"M66.8338,-131.7055C99.2593,-145.0771 166.1232,-172.6504 204.2283,-188.3642\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"203.1305,-191.6973 213.7096,-192.2741 205.7992,-185.226 203.1305,-191.6973\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- 2 Input&#45;&#45;3 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge7\">\n",
       "\n",
       "<g id=\"a_edge7\"><a xlink:title=\"w32\">\n",
       "<path d=\"M68.5753,-124C100.9034,-124 164.0528,-124 201.819,-124\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"202.207,-127.5001 212.2069,-124 202.2069,-120.5001 202.207,-127.5001\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "<!-- 2 Input&#45;&#45;4 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge8\">\n",
       "\n",
       "<g id=\"a_edge8\"><a xlink:title=\"w42\">\n",
       "<path d=\"M66.8338,-116.2945C99.2593,-102.9229 166.1232,-75.3496 204.2283,-59.6358\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"205.7992,-62.774 213.7096,-55.7259 203.1305,-56.3027 205.7992,-62.774\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a diagram with 2 input units and 4 hidden units\n",
    "# (ignoring output units now by setting to 0)\n",
    "viewnet(layers=[2,4,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be represented by:\n",
    "\n",
    "\\begin{equation}\n",
    "z_j = \\sigma(a_j) = \\sigma(\\sum_{i=1}^{D}w_{ji}x_i + w_{j0})\n",
    "\\end{equation}\n",
    "\n",
    "Where $a_j$ is our \"activation\" which is transformed by the nonlinear \"activation function\" $\\sigma$\n",
    "\n",
    "Or, if we define $x_0$ to be constant 1, we can write it in vector form as:\n",
    "\n",
    "\\begin{equation}\n",
    "\\mathbf{z} = \\sigma(W \\mathbf{x})\n",
    "\\end{equation}\n",
    "\n",
    "(the sigmoid being element-wise)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, with the appropriate weights and biases we can represent various boolean functions. Here an output > 0.5 is true and < 0.5 is false."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W = \n",
      " [[-3.  2.  2.]\n",
      " [-1.  2.  2.]\n",
      " [ 3. -2. -2.]\n",
      " [ 1. -2. -2.]] \n",
      "\n",
      "x =  [1, 0, 1] \n",
      "\n",
      "activation: [-1.  1.  1. -1.]\n",
      "nonlinear transform: [0.26894142 0.73105858 0.73105858 0.26894142]\n"
     ]
    }
   ],
   "source": [
    "biases = [-3, -1, 3, 1]\n",
    "\n",
    "weights = [[ 2.,  2.], # AND\n",
    "           [ 2.,  2.], # OR\n",
    "           [-2., -2.], # NAND\n",
    "           [-2., -2.]] # NOR\n",
    "\n",
    "# concatenate the biases and weights into one array\n",
    "W = zeros((4,3))\n",
    "W[:,0], W[:,1:] = biases, weights\n",
    "\n",
    "print('W = \\n', W, '\\n')\n",
    "\n",
    "# test with x1=0, x2=1\n",
    "x = [0, 1]\n",
    "\n",
    "# set x0 to be constant 1\n",
    "x = [1] + x\n",
    "\n",
    "print('x = ', x, '\\n')\n",
    "\n",
    "print('activation:', dot(W, x))\n",
    "print('nonlinear transform:', sigmoid(dot(W, x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We make a couple of functions to compute a sigmoid and tanh layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid_layer(x, W):\n",
    "    assert x[0] == 1\n",
    "    return sigmoid(dot(W, x))\n",
    "\n",
    "def tanh_layer(x, W):\n",
    "    assert x[0] == 1\n",
    "    return tanh(dot(W, x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using our boolean weights with a tanh unit, we get a positive value for true and a negative value for false."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x1  x2    AND         OR          NAND        NOR\n",
      "0   0   [-0.99505475 -0.76159416  0.99505475  0.76159416]\n",
      "0   1   [-0.76159416  0.76159416  0.76159416 -0.76159416]\n",
      "1   0   [-0.76159416  0.76159416  0.76159416 -0.76159416]\n",
      "1   1   [ 0.76159416  0.99505475 -0.76159416 -0.99505475]\n"
     ]
    }
   ],
   "source": [
    "print('x1  x2    AND         OR          NAND        NOR')\n",
    "for x1 in (0,1):\n",
    "    for x2 in (0,1):\n",
    "        print(x1, ' ', x2, ' ', tanh_layer([1, x1, x2], W))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far so good, but this is all stuff we could do with a Perceptron (a simple threshold unit, ie $\\operatorname{sign}(\\mathbf{w}.\\mathbf{x})$). Why do we need the sigmoidal function?\n",
    "\n",
    "Well, with a single non-linear layer (sigmoidal or perceptron) there are functions we cannot represent, for example no configuration of the weights above would allow us to compute boolean XOR. However, with additional layers the story is different. Indeed, some combination of NAND units is sufficient to compute any boolean function. In fact any boolean function can be computed with just two layers, using a combination of OR and AND units, since any boolean function can be expressed in disjunctive normal form (an OR of ANDs of inputs and their negations, where negation can be expressed by the weights).\n",
    "\n",
    "This gives us greater representational power, but still could be done by multiple layers of perceptron units. The difference is when it comes to learning the weights from samples. There is no efficient learning rule for multiple layers of perceptron units, but because the sigmoid units are smooth we can find their gradient, which gives rise to an efficient learning algorithm called Backpropagation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x1  x2  XOR\n",
      "0   0   [-0.64491267]\n",
      "0   1   [0.48015701]\n",
      "1   0   [0.48015701]\n",
      "1   1   [-0.64491267]\n"
     ]
    }
   ],
   "source": [
    "# Implement XOR, by setting weights such that\n",
    "# the AND and OR columns above are added, then -1\n",
    "#\n",
    "#                bias AND  OR   NAND NOR\n",
    "W_out = array([[-1.0, 0.0, 1.0, 1.0, 0.0]])\n",
    "\n",
    "print('x1  x2  XOR')\n",
    "        \n",
    "for x1 in (0,1):\n",
    "    for x2 in (0,1):\n",
    "        print(x1, ' ', x2, ' ', tanh_layer([1]+list(tanh_layer([1, x1, x2], W)), W_out))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A diagram of our \"handmade\" XOR net."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"355pt\" viewBox=\"0.00 0.00 472.89 355.00\" width=\"473pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 351)\">\n",
       "\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-351 468.8893,-351 468.8893,4 -4,4\" stroke=\"transparent\"/>\n",
       "<g class=\"cluster\" id=\"clust1\">\n",
       "\n",
       "<polygon fill=\"#ff9999\" points=\"8,-84 8,-263 88.2964,-263 88.2964,-84 8,-84\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"48.1482\" y=\"-247.8\">Input</text>\n",
       "</g>\n",
       "<g class=\"cluster\" id=\"clust2\">\n",
       "\n",
       "<polygon fill=\"#99ff99\" points=\"192.2964,-8 192.2964,-339 272.5928,-339 272.5928,-8 192.2964,-8\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4446\" y=\"-323.8\">Hidden 1</text>\n",
       "</g>\n",
       "<g class=\"cluster\" id=\"clust3\">\n",
       "\n",
       "<polygon fill=\"#9999ff\" points=\"376.5928,-84 376.5928,-187 456.8893,-187 456.8893,-84 376.5928,-84\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"416.7411\" y=\"-171.8\">Output</text>\n",
       "</g>\n",
       "<!-- 1 Input -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "\n",
       "<ellipse cx=\"48.1482\" cy=\"-200\" fill=\"none\" rx=\"20.2975\" ry=\"20.2975\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"48.1482\" y=\"-196.3\">x1</text>\n",
       "</g>\n",
       "<!-- 1 Hidden 1 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "\n",
       "<g id=\"a_node3\"><a xlink:title=\"bias=-3\">\n",
       "<ellipse cx=\"232.4446\" cy=\"-276\" fill=\"none\" rx=\"20.2975\" ry=\"20.2975\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4446\" y=\"-272.3\">z1</text>\n",
       "</a>\n",
       "</g>\n",
       "<polygon fill=\"#ffffff\" points=\"177.2964,-240.8518 177.2964,-255.8518 212.2964,-255.8518 212.2964,-240.8518 177.2964,-240.8518\" stroke=\"transparent\"/>\n",
       "<text fill=\"#ff0000\" font-family=\"sans-serif\" font-size=\"10.00\" text-anchor=\"start\" x=\"179.7964\" y=\"-245.3518\">bias -3</text>\n",
       "</g>\n",
       "<!-- 1 Input&#45;&#45;1 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "\n",
       "<g id=\"a_edge1\"><a xlink:title=\"w11=2\">\n",
       "<path d=\"M66.8338,-207.7055C99.2593,-221.0771 166.1232,-248.6504 204.2283,-264.3642\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"203.1305,-267.6973 213.7096,-268.2741 205.7992,-261.226 203.1305,-267.6973\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"114.031\" y=\"-224.8348\">     2     </text>\n",
       "</g>\n",
       "<!-- 2 Hidden 1 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "\n",
       "<g id=\"a_node4\"><a xlink:title=\"bias=-1\">\n",
       "<ellipse cx=\"232.4446\" cy=\"-200\" fill=\"none\" rx=\"20.2975\" ry=\"20.2975\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4446\" y=\"-196.3\">z2</text>\n",
       "</a>\n",
       "</g>\n",
       "<polygon fill=\"#ffffff\" points=\"177.2964,-164.8518 177.2964,-179.8518 212.2964,-179.8518 212.2964,-164.8518 177.2964,-164.8518\" stroke=\"transparent\"/>\n",
       "<text fill=\"#ff0000\" font-family=\"sans-serif\" font-size=\"10.00\" text-anchor=\"start\" x=\"179.7964\" y=\"-169.3518\">bias -1</text>\n",
       "</g>\n",
       "<!-- 1 Input&#45;&#45;2 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "\n",
       "<g id=\"a_edge2\"><a xlink:title=\"w21=2\">\n",
       "<path d=\"M68.5753,-200C100.9034,-200 164.0528,-200 201.819,-200\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"202.207,-203.5001 212.2069,-200 202.2069,-196.5001 202.207,-203.5001\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"113.6972\" y=\"-188.8\">     2     </text>\n",
       "</g>\n",
       "<!-- 3 Hidden 1 -->\n",
       "<g class=\"node\" id=\"node5\">\n",
       "\n",
       "<g id=\"a_node5\"><a xlink:title=\"bias=+3\">\n",
       "<ellipse cx=\"232.4446\" cy=\"-124\" fill=\"none\" rx=\"20.2975\" ry=\"20.2975\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4446\" y=\"-120.3\">z3</text>\n",
       "</a>\n",
       "</g>\n",
       "<polygon fill=\"#ffffff\" points=\"175.2964,-88.8518 175.2964,-103.8518 212.2964,-103.8518 212.2964,-88.8518 175.2964,-88.8518\" stroke=\"transparent\"/>\n",
       "<text fill=\"#ff0000\" font-family=\"sans-serif\" font-size=\"10.00\" text-anchor=\"start\" x=\"177.7964\" y=\"-93.3518\">bias +3</text>\n",
       "</g>\n",
       "<!-- 1 Input&#45;&#45;3 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "\n",
       "<g id=\"a_edge3\"><a xlink:title=\"w31=-2\">\n",
       "<path d=\"M66.8338,-192.2945C99.2593,-178.9229 166.1232,-151.3496 204.2283,-135.6358\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"205.7992,-138.774 213.7096,-131.7259 203.1305,-132.3027 205.7992,-138.774\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"112.031\" y=\"-152.7652\">     -2     </text>\n",
       "</g>\n",
       "<!-- 4 Hidden 1 -->\n",
       "<g class=\"node\" id=\"node6\">\n",
       "\n",
       "<g id=\"a_node6\"><a xlink:title=\"bias=+1\">\n",
       "<ellipse cx=\"232.4446\" cy=\"-48\" fill=\"none\" rx=\"20.2975\" ry=\"20.2975\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.4446\" y=\"-44.3\">z4</text>\n",
       "</a>\n",
       "</g>\n",
       "<polygon fill=\"#ffffff\" points=\"175.2964,-12.8518 175.2964,-27.8518 212.2964,-27.8518 212.2964,-12.8518 175.2964,-12.8518\" stroke=\"transparent\"/>\n",
       "<text fill=\"#ff0000\" font-family=\"sans-serif\" font-size=\"10.00\" text-anchor=\"start\" x=\"177.7964\" y=\"-17.3518\">bias +1</text>\n",
       "</g>\n",
       "<!-- 1 Input&#45;&#45;4 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge4\">\n",
       "\n",
       "<g id=\"a_edge4\"><a xlink:title=\"w41=-2\">\n",
       "<path d=\"M63.8612,-187.0406C96.2318,-160.3427 170.5139,-99.0779 208.7486,-67.5435\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"211.2837,-69.9895 216.7714,-60.9266 206.8298,-64.5893 211.2837,-69.9895\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"159.8049\" y=\"-116.092\">     -2     </text>\n",
       "</g>\n",
       "<!-- 2 Input -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "\n",
       "<ellipse cx=\"48.1482\" cy=\"-124\" fill=\"none\" rx=\"20.2975\" ry=\"20.2975\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"48.1482\" y=\"-120.3\">x2</text>\n",
       "</g>\n",
       "<!-- 2 Input&#45;&#45;1 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge5\">\n",
       "\n",
       "<g id=\"a_edge5\"><a xlink:title=\"w12=2\">\n",
       "<path d=\"M63.8612,-136.9594C96.2318,-163.6573 170.5139,-224.9221 208.7486,-256.4565\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"206.8298,-259.4107 216.7714,-263.0734 211.2837,-254.0105 206.8298,-259.4107\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"157.8049\" y=\"-185.508\">     2     </text>\n",
       "</g>\n",
       "<!-- 2 Input&#45;&#45;2 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge6\">\n",
       "\n",
       "<g id=\"a_edge6\"><a xlink:title=\"w22=2\">\n",
       "<path d=\"M66.8338,-131.7055C99.2593,-145.0771 166.1232,-172.6504 204.2283,-188.3642\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"203.1305,-191.6973 213.7096,-192.2741 205.7992,-185.226 203.1305,-191.6973\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"157.031\" y=\"-148.8348\">     2     </text>\n",
       "</g>\n",
       "<!-- 2 Input&#45;&#45;3 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge7\">\n",
       "\n",
       "<g id=\"a_edge7\"><a xlink:title=\"w32=-2\">\n",
       "<path d=\"M68.5753,-124C100.9034,-124 164.0528,-124 201.819,-124\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"202.207,-127.5001 212.2069,-124 202.2069,-120.5001 202.207,-127.5001\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"111.6972\" y=\"-112.8\">     -2     </text>\n",
       "</g>\n",
       "<!-- 2 Input&#45;&#45;4 Hidden 1 -->\n",
       "<g class=\"edge\" id=\"edge8\">\n",
       "\n",
       "<g id=\"a_edge8\"><a xlink:title=\"w42=-2\">\n",
       "<path d=\"M66.8338,-116.2945C99.2593,-102.9229 166.1232,-75.3496 204.2283,-59.6358\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"205.7992,-62.774 213.7096,-55.7259 203.1305,-56.3027 205.7992,-62.774\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"112.031\" y=\"-76.7652\">     -2     </text>\n",
       "</g>\n",
       "<!-- 1 Output -->\n",
       "<g class=\"node\" id=\"node7\">\n",
       "\n",
       "<g id=\"a_node7\"><a xlink:title=\"bias=-1\">\n",
       "<ellipse cx=\"416.7411\" cy=\"-124\" fill=\"none\" rx=\"20.2975\" ry=\"20.2975\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"sans-serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"416.7411\" y=\"-120.3\">y1</text>\n",
       "</a>\n",
       "</g>\n",
       "<polygon fill=\"#ffffff\" points=\"361.5928,-88.8518 361.5928,-103.8518 396.5928,-103.8518 396.5928,-88.8518 361.5928,-88.8518\" stroke=\"transparent\"/>\n",
       "<text fill=\"#ff0000\" font-family=\"sans-serif\" font-size=\"10.00\" text-anchor=\"start\" x=\"364.0928\" y=\"-93.3518\">bias -1</text>\n",
       "</g>\n",
       "<!-- 1 Hidden 1&#45;&#45;1 Output -->\n",
       "<g class=\"edge\" id=\"edge9\">\n",
       "\n",
       "<g id=\"a_edge9\"><a xlink:title=\"w11=0\">\n",
       "<path d=\"M248.1576,-263.0406C280.5282,-236.3427 354.8103,-175.0779 393.045,-143.5435\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"395.5801,-145.9895 401.0678,-136.9266 391.1262,-140.5893 395.5801,-145.9895\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"299.1013\" y=\"-192.092\">     0     </text>\n",
       "</g>\n",
       "<!-- 2 Hidden 1&#45;&#45;1 Output -->\n",
       "<g class=\"edge\" id=\"edge10\">\n",
       "\n",
       "<g id=\"a_edge10\"><a xlink:title=\"w12=1\">\n",
       "<path d=\"M251.1302,-192.2945C283.5557,-178.9229 350.4197,-151.3496 388.5247,-135.6358\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"390.0956,-138.774 398.0061,-131.7259 387.4269,-132.3027 390.0956,-138.774\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"298.3275\" y=\"-152.7652\">     1     </text>\n",
       "</g>\n",
       "<!-- 3 Hidden 1&#45;&#45;1 Output -->\n",
       "<g class=\"edge\" id=\"edge11\">\n",
       "\n",
       "<g id=\"a_edge11\"><a xlink:title=\"w13=1\">\n",
       "<path d=\"M252.8717,-124C285.1998,-124 348.3492,-124 386.1154,-124\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"386.5034,-127.5001 396.5034,-124 386.5033,-120.5001 386.5034,-127.5001\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"297.9936\" y=\"-112.8\">     1     </text>\n",
       "</g>\n",
       "<!-- 4 Hidden 1&#45;&#45;1 Output -->\n",
       "<g class=\"edge\" id=\"edge12\">\n",
       "\n",
       "<g id=\"a_edge12\"><a xlink:title=\"w14=0\">\n",
       "<path d=\"M251.1302,-55.7055C283.5557,-69.0771 350.4197,-96.6504 388.5247,-112.3642\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"387.4269,-115.6973 398.0061,-116.2741 390.0956,-109.226 387.4269,-115.6973\" stroke=\"#000000\"/>\n",
       "</a>\n",
       "</g>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"298.3275\" y=\"-72.8348\">     0     </text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "viewnet(layers=[2,4,1], weights=[W, W_out])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x1163146d8>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO2dd3hUxdrAf5MEQlEITTqCAgIqBIgUr58gWAAVUMAGAtIECwGvXrHcS6xgAyKKSBNFRRFF8FrpV5EWOkiV3kG6QEiy7/fHnoUl7Cab7Dl7zu7O73nOs3vmTHn37Dnzzsz7zowSETQajUYTvcTYLYBGo9Fo7EUrAo1Go4lytCLQaDSaKEcrAo1Go4lytCLQaDSaKCfObgHyQ+nSpaVq1ap2i6HRaDRhxbJlyw6LSJns4WGpCKpWrUpaWprdYmg0Gk1YoZTa4StcDw1pNBpNlKMVgUaj0UQ5WhFoNBpNlKMVgUaj0UQ5WhFoNBpNlKMVgUaj0UQ5pigCpdQEpdRBpdRaP9eVUupdpdQWpdRqpVQDr2vdlFKbjaObGfJoNBqNJnDM6hFMBFrlcL01UMM4+gAfACilSgKDgcZAI2CwUqqESTJpNBqNJgBMmVAmIv9TSlXNIUo74BNxb36wSCmVoJQqDzQHZorIEQCl1EzcCmWyGXJpnEnz5u7PefPslEKTV1ziYs2BNczZNod5O+YBUKtULWqVrkXtMrWpVboWCYUS7BVSky9CNbO4IrDL63y3EeYv/BKUUn1w9yaoUqWKNVJqNJqL2HZ0Gz//+TNzts1h7va5HD59GIDqJatTKK4QP235iXNZ587HL1u0LM2qNuPVW16lRqkadontCMKpwRMqRaB8hEkO4ZcGiowBxgAkJSXpbdXCEM+LMX/+xefh8KJEG+mZ6aTMS+HN39/EJS4qXl6RNjXa0LJaS26peguVi1cGINOVyfZj21l/aD0bDm/gj8N/MPWPqUxbP43+jfvz4s0v6l5CGBAqRbAbqOx1XgnYa4Q3zxY+L0QyaTQaABFQF9pkS3cvofuMR/jj0B/0SOzBszc9S42SNVDq0nZbXEwc1UtWp3rJ6tx9zd0ADGk5hBfnvMiwhcP4eNXHvHLLK/Rq0Iu4mLBc2izPhGODJ1TuozOArob3UBPguIjsA34GbldKlTCMxLcbYZoIZN4899GsmfvwnGtsJCUFBg4EEdIz03l+1nM0HdeE4wd38cNDPzC+3XhqlqrpUwn4o9xl5RjXdhzL+iyjTpk69Pu+H/U/rM/srbOt+x2aoDBFRSulJuNu2ZdWSu3G7QlUAEBERgM/AG2ALcBp4BHj2hGl1CvAUiOrlz2GY41GYzEicOwYpKaSFnuA7levYd2hdTyyEoZV60xC9ZwcAXOnfvn6zOs2j2/Wf8MzM5/h1km38sGdH9A3qa9JP8CZeBo34dAT8KDcjjzhRVJSkuhlqDUaExBhyr/u5KEiP1L2FIz9Dtq0SYbhwy8aLgqWMxln6PRVJ77f/D3D7xjOgCYDTMvbqThRESillolIUvbw6Bi002g0Pvl+8w90LjaTptvgu8mQcJbzSsDMiqxwgcJ8c/83PPj1gwz8eSBnM88y6KZBwWfsYJykAHJDLzGh0UQp87bPo+NXHal3riT//dxQAnDeZmA2BWML8mXHL3nwugd5bvZzpMxLIRxHJCIR3SPIB07s8mk0eWHJniXcPflurkovwk/DDlK8rzEcNHAgpKYydSrM3zMcMLdnEBcTx6R7JhEfF89L81/ibOZZhrQckidjtMZ8tCLQWIpWms5jzYE1tPq0FVcUvYKZ6e0p3Tvrgk1g+HCmToVTcQn4nuYTPLExsYxvO5742HjeWPAGZzLOMKLVCOuVQTY32UvOoxitCPJAOPoHa6KMXCq7zX9t5rZJt1G4QGFmPTyLCiWqXRxHKTruciuFic3dQVY83zEqhg/u/IBCcYVIXZxK5eKVefrGp80vyENKittDyqPwRNy9n4QE97UoR9sINJbQvLn7mD/ffXjONRbiNScAuFDZGRXd7hO7uXXSrWRJFjMfnkm1EtXc8bK3ikPUSlZKMfyO4dxb+16em/0ci3YvsqYgLzfZ8/fHGALj2DFL7CHhhu4R5IFw9A/WRAnelR1cNN5PcjJZWZl0/qYzR84cYX73+dQpUyfXLEPxfCulGN92PPU/rM/9U+9nxaMrKFm4pNmFuO8HuO+H5x4lm+8mG67oeQT5QCuCwNH3KoR4t3Q9GJXdGwveZNDsQXzc/mO61utqn4x+WLJnCTdNuInWNVrz7f3fWmMvEIEYr0EQlyvqlIC/eQR6aCgf6KURNI7Eu+XrYfhwlu9fwb/n/ptOdTrxcN2H7ZEtFxpVbMSbt73JjI0zGLFohPkFeJSkNxa5yYYjWhFoLEUrzRDio7I7PfAJOn/TmTJFyzD6rtGOdtNMbpxMu2va8eysZ1myZ4l5GXv3lJKT3T2B5OSLbQZRjlYEZpL9gdIPmCZU+Knsnt00ig2HNzCx7Ufmj72bjFKKCe0mUOHyCtw/9X6OnjlqVsZu7yBvm8Dw4e7zhISoGx7yhbYRmIV2T9PYTbZn8MdNP9Bm8p0MkMYMT7HII8cCFu9ezE0f3cRdNe/im/u+Ma8Xo+cRaBuBpWj3NI0TSEk5rwQO/X2IHt/15Noy1zLkxXnml2Vh77dxpcYMbTmUbzd8y5hlY0zL1y432XBA9wjMIgePDf3AaUKJiHDvlHv5YfMPLOm1hHrl6plbQAh6vyJCi09asGr/KjY9uYnSRUqbkm8wRIIHnO4RWI0fjw2tBDShZuLKiXy74Vtea/Ga+UogRL1fpRSnvnyPY6dP8vzs503JU+MfUxSBUqqVUmqjUmqLUuqStWWVUsOVUiuNY5NS6pjXtSyvazPMkMcWtHuaxgEcO3uMZ2Y+w01VbuKppk+ZX4C3oTU11e2X7zFQm9zwKXr6Wiru7s+45eNYumdp7gksIhpmyQetCJRSscD7QGugDvCgUuqiaYsiMlBEEkUkERgJfON1+Yznmoi0DVaenDh+9jhrD641P2PtnqZxCC/Pf5kjZ44wsvVIYpRFHX6Le7/eFe/uzwZTIL0sLYc9jktcpuSvuRQznpRGwBYR2Soi54AvgHY5xH8QmGxCuXnmzs/vpNNXnch0ZZqbsXZP0ziATX9tYuSSkfSs35PEconWFRTK3m96Ma76821OFlvK+OXjzc8/AKJir20RCeoAOgLjvM4fBt7zE/dKYB8Q6xWWCaQBi4D2OZTTx4iXVqVKFckP09ZPE1KQ0UtH5yt9bjS72SXNmnkFuFyWlKPR+OLuz++Wy1+/XPaf3G9dIS6XSHKyCLg/fZ2bRLNm7sPlcsn/Tfg/KfVGKfnr9F/+5crp3ER5whkgTXzUr2b0CHw1d/01DR4ApopIlldYFXFbsR8CRiilrvaVUETGiEiSiCSVKVMmX4K2u6YdN1W5icHzBnMy/aTPOEGN/2n3NI1NzPxzJt9t+o4Xb36RspeVta4gG3q/Sinea/Mex84e44XZL1waIZdVV80imJ6A0+0KZiiC3UBlr/NKwF4/cR8g27CQiOw1PrcC84D6JsjkE6UUb932Fgf+PsA7C98xLd9oMCZpnEumK5OBPw/kqhJXkdw42foCveYrABeUgYUVb92ydXn8hsf5cNmHLNu77EIkPYfHHHx1E/Jy4F7KeitQDSgIrAKu9RHvGmA7xtwFI6wEEG98Lw1sBurkVmbDhg2D6h51mtJJirxWRPae2Hs+zNPtcz85eesGBpNWowmWUUtGCSnIN398Y7colnL0zFG54q0rpPHYxpLlyrpwwXt4ynOYPEyVX5xWN2DV0JCIZAJPAD8D64EpIrJOKfWyUsrbC+hB4AtDGA+1gTSl1CpgLjBURP4IVqbceL3l62RkZZAyL8WU/KLCmKRxJEfPHOXfc/9N86rNaV+rvd3iWEpCoQTevPVNFu9ZzMSVEy9c0HN4gseXdnD6EWyPQESk/w/9JealGFl3cN1F4cFobLu1vSb6GPjTQFEpSlbuW2m3KKbj633KcmVJk3FNpOI7FeVMxhl3oIN7BB6cUjdgobE4LPl3s39zWcHLGDTrkvlv+Ub3BDShxOMu2qtBL/NnEDuUGBXDq7e8yp6Texi7bKyew2MSUbtVZekipXnupud4bvZzzN8+n2ZVmwG6IteED8/MfIbCcYV5tcWrlpVhx/o6njLnz/ctQ4tqLbj5ypt5/bfX6dWgF4V9eTGBo+bwOL1eidoeAbg3wqhUrBJPz3xaz1rUhBVL9yxlxsYZDLppEFcUvcJucUKKUoqXm7/M/lP7GZ02OmReTJFM1K8++vHKj+k+vTuTO0zmgeseMCVPjcZq2k5uy4JdC9ievJ3L4y83Pf/srfJm7g6zLT0Df2Xe+smtrDm4hq39t1K0YNFQiRXW6NVH/dClbhfqla3H87OfJyMrw25xNJpcWbFvBd9t+o6BTQZaogTChZeav8TBvw8yaukou0UJe6K+RwDw/abvuWvyXXzU7iO6J3Y3LV+Nxgru/fJe5mybw44BOyheqLilZTl9Df5Wn7YibW8a25K3ha1SDOU91j2CHGhTow31ytZj6G9DyXJl5Z5Ao7GJ1QdWM23DNAY0GWC5EggHXr7lZf468xcjl4y0W5SwRisC3Man5//veTb+tZFv1n+TewKNxiZe/d+rXF7w8tAsJYHzXaIbVWzEXTXv4u3f3+b42eN2i5MnnLQ0jVYEBh1qd6BmqZq8/tvrhONwmSY4wmF9qHUH1zH1j6n0b9yfEoVL2C2OY3ip+UscPXuU1MWpuUfW+EQrAoPYmFgG/WMQK/ev5MctP9otjkZzCa/++ipFCxZlYJOBuUeOIhqUb0D7Wu0ZtnAYR88ctVucgHHS0jRaEXjRpW4XqhSvwmu/vqZ7BVGCk7rnObHh8Aa+XPslT9zwBKWKlLJbHMeR0iyF4+nHGb5oeO6RNZegFYEXBWIL8MyNz/D7rt/5347/2S2ORnOe1359jcIFCluzD3EEUK9cPTrU7kDq4lROpJ+wW5w84QQ7jFYE2ehZvydli5bl9d9et1sUTQhwUvfcH5v/2sznaz7nsaTHKFM0f5syRQODbhrEifQTjFs+zm5Rwg6tCLLhaXX98ucvLN2z1G5xNBpe/+11CsYW5Okbn7ZbFEeTVCGJ5lWbM2LRCD05NI9oReCDvkl9SSiUwJDfhtgtSshw6th4qHBiTwBgx7EdTFo1iUcbPmrtFpQRwtNNn2bXiV189cdXdosSVpiiCJRSrZRSG5VSW5RSl6zrrJTqrpQ6pJRaaRy9vK51U0ptNo5uZsgTLMXii9G/UX+mbZjGuoPr7BZHE8V4Jkr9s+k/bZYkPGhdozW1S9fmrd/f0g4feSBoRaCUigXeB1oDdYAHlVJ1fET9UkQSjWOckbYkMBhoDDQCBiulHOEg3b9xf4oWKMrQBUPtFsVSwsVrJho5mX6SscvH0unaTlQuXjn3BBpiVAz/bPpPVu5fyZxtc+wWJ2wwo0fQCNgiIltF5BzwBdAuwLR3ADNF5IiIHAVmAq1MkCloShUpRd+kvkxeM5ltR7fZLY4mCpmwYgIn0k/oeQN5pHPdzpQtWpa3F75ttyhhgxmKoCKwy+t8txGWnQ5KqdVKqalKKU/zJtC0tjCwyUCUUhG9jkk4eM1EI1muLFIXp/KPyv+gUcVGdosTVhSKK8STjZ7kpy0/sfbgWrvFCQvMUAS+tgDKPjj3HVBVROoCs4CP85DWHVGpPkqpNKVU2qFDh/ItbF6oWKwinep0YvyK8ZxMPxmSMjXWEw7DX9M3TmfbsW163kA+6ZvUlyIFivDOwnfsFiUsMEMR7Aa8BzArAXu9I4jIXyKSbpyOBRoGmtYrjzEikiQiSWXKhM6XOrlxMifST/Dxqo9zjxzG6J6Asxi2cBjVEqrR7ppAR1k13pQqUooeiT34bPVn7D3ps0pxBE5plJihCJYCNZRS1ZRSBYEHgBneEZRS5b1O2wLrje8/A7crpUoYRuLbjTDH0LhSYxpXbMy7i9/V21mGOXYbxgMtb8meJSzYtYDkxsnExsRaLVbEMrDpQLIki5GLI3do1yyCVgQikgk8gbsCXw9MEZF1SqmXlVJtjWj9lVLrlFKrgP5AdyPtEeAV3MpkKfCyEeYokhsns/nIZn7crBej01jP8EXDKRZfjB71e9gtSlhzVYmr6FC7A6OXjXbc0K7djZLs6B3KyH2HoIysDKqmVuXaMtfyy8O/mFauxh5CvetWXvb/3XV8F9VSqzGgyQDevl17vQTL4t2LaTK+CSPuGEFyk9Ds4RAIdu0JrXcoyyteCrJAbAEeT3qMmVtn6glmGksZuWQkgvBkoyftFiUiaFypMf9X5f8Yvmg4ma5Mu8U5j9O89eLsK9p+smvl8y3F5ilw7BgMHw5KgQh9pu3kleJxvLv4XT68+0MbpNWYRahfOE95ufVETp07xZhlY+hYpyNXJlwZAsmig6eaPsU9X97Ddxu/457a99gtjiPRPYLsiLiVQGoqDBzoPh84kNIjxtDl3DVMWj2JI2ccZ8bQRAAfrfiI4+nHeaqJdhk1k7tq3kWV4lV4b+l7dotyCXb3BDxEdY/Ad0tNgRibW6Smug+A5GT69+/BuA/rMXbZWJ696dmQyqoJf3J64bNcWYxYPIKmlZrSuFLjkMkUDcTFxNG3YV+en/M86w+tp3aZ2naL5Dh0j8AXSrmHhbwZPpzry9WlRbUWvLf0vZAsc2u3J4EmdHy/+Xu2Ht2ql5OwiF4NelEwtiCjlo6yWxRHohUBPrpnxnDQRRjDRMmNk9l9YjfTNkwLoYSaSKZ5c+j+/igqXF5Bj2FbRJmiZbj/2vv5eNXHjnMldQJaEWTHowRSUyE5GVwu96dhM7izehuuLnE1qYtTLRPBaT7GGms5U3gLR0v9zKMNHyUuJqpHay3l8Rse5+S5k0xaPcluURyHVgTZUQoSEtyVv8draPhw93lCArGxcTzZ6El+3/W7Y3Yw04oiPPH8b0uyPoSsOH58rZf+Hy2kUcVGNCzfkPeXvq/3KsiGbn74IiXF3TNQxpp4HmVgnD9S/xFenPsio9JG8VHFj8wtW4R589zlNG9+8bkm8siKOQP1J8CGe4g/V8FucSIapRRPNHqCR6Y/wvwd82letbndIjkG3SPwh1J+z4vFF6Pz9Z35Yu0XHD1z1LwyU1IuuKwCiPDEnwPd4T7QQ0jhzbx50GvYFChyhLrn+jnGlTCSuf/a+ylZuCTvLXGeK6mdaEWQT/ol9eNs5lnzViX1MX9hXv2BdNyT6g7XXdmIZFTaKIr8XYuEY83tFiUqKFygMD3r9+TbDd+y+8Ruu8VxDHqtoSBoOr4pR88cZf3j61HZexD5wdtQ7cHbVuGHUK+dozGHZXuXkTQ2iXdbvcuTjfWSEqFi29FtXP3u1bx484u8fMvLdosTUvRaQxbQt2FfNv61kXnb5+U9cXYF7LFJ+Ji/kJMS0IQvH6R9QJECRehar6vdokQV1UpU486adzJm2RjOZZ2zWxxHoBVBENx37X2UKFSC0ctG5y2hD1sAAwfC4MF+5y/khB5bDj+OnjnK52s+p8v1XSheqLjd4kQdj9/wOAf+PsDXf3xttyiOQCuCIChcoDDdE7vzzfpv2H9qf2CJ/KxlRGoqzJjhd/6CmTaCaDUqO+l3f7zqY85knqHfDf3sFiUquf3q26lesjrvL33fblEcgVYEQfJow0fJdGUyYcWEwBJ4z0tITYWYmAuVf9u2fucv6OGhyMElLkYtHUXTSk1JLJdotzhRSYyK4bGkx1iwawEr96+0WxzbMcVYrJRqBaQCscA4ERma7fpTQC8gEzgE9BCRHca1LGCNEXWniLQlF5xiLPbQ8pOWbDmyha39twa+taCIWwl4cLnOL3l9UaWf/TwI7NoMw26c9rtnbZ3FbZNuY9I9k+hSt4s9Qmg4euYoFYZV4JHERxh1Z3SsQWSZsVgpFQu8D7QG6gAPKqXqZIu2AkgSkbrAVOBNr2tnRCTROHJVAk6kX1I/dh7fyU9bfgosQQ5rGeU0f0ETGXyQ9gGli5SmY52OdosS1ZQoXIL7rr2PT1d/yt/n/rZbHFsxY2ioEbBFRLaKyDngC6CddwQRmSsip43TRUAlE8p1DO2uaUe5y8rxQdoHuUfOZS0jK+cLOG1XJDPJafzfSb9794ndTN8wnR6JPSgUV8geITTn6dOgDyfPneTLdV/aLYqtmKEIKgK7vM53G2H+6Al47wJfSCmVppRapJRq7y+RUqqPES/t0KFDwUlsMgViC9Czfk9+2PwDO47tyDlyLmsZ6R5AZDNu+Thc4uLRpEftFkUD3Fj5RuqUqcOHy6J810ERCeoAOuG2C3jOHwZG+onbBXePIN4rrILxeRWwHbg6tzIbNmwoTmPHsR0S81KMPD/r+cASuFw5n0cBzZq5DzPycHelzMnTKjKzMqXSsEpyx6Q77BZF48WIhSOEFGTFvhV2i2I5QJr4qFPN6BHsBip7nVcC9maPpJS6FXgBaCsi6V6KaK/xuRWYB9Q3QaaQU6V4FdrUaMP4FeMDm6SibQFRx09bfmL3id30adjHblFyx9eExwjl4XoPUyiuEGOXjbVbFNswQxEsBWoopaoppQoCDwAzvCMopeoDH+JWAge9wksopeKN76WBfwB/mCCTLfRL6seBvw8wfcP03COH6Ytmhi++mYvlOWn8PzfGLB9D2aJlubvm3XaLkjP+Jjz6Wfww3ClZuCSd6nTi0zXRazQOWhGISCbwBPAzsB6YIiLrlFIvK6U8XkBvAZcBXymlViqlPIqiNpCmlFoFzAWGikjYKoI7rr6DK4tfmftM4yh70TSw58Qevt/0PY8kPkKB2AJ2i+OfnCY8RvDih30a9uFE+onoNRr7Gi9y+uFEG4GHl+e9LKQgW/7a4juCyyWSnOwe0E5O9n3uQKwYi3fyeL7ZvDL/lZyfCyfh/Ux6Dgc/m2bgcrmk9nu1pfHYxnaLYilYaCPQePFI/UeIUTGMXzHed4ScZhbrBeYiEpe4GLd8HC2rteTqklfbLU7uROHih0opHm34KIv3LGbV/lW2ymLHUihaEZhMpWKVaFOjDR+t/IiMrAzfkcLwRbNiLD57Hk5aC8hMZv45kx3Hd9C7QW+7RQmMnCY8esfJnibMebjew8THxjNm2Ri7RQk5WhFYQO8Gvdl/aj/fb/7ed4RAXjRNxDB2+VhKFylN+1p+p8k4h0AmPEaojatk4ZJ0utY+o7GdOw5qRWABbWq0ocLlFRi73Ic7mo0zi80gp55Afh/cSN5y88CpA0zfOJ1u9boRHxdvtzi5k9uER4hoY/KjDR/lRPoJpqybYrcoIUVvXm8BcTFxPJL4CEN+G8Ku47uoXNxrmoW/Fw30zOIIZOLKiWS6MsNnWAjcLXvvda88z6jn3PO8pqZe2E0vQmxc/6j8D2qXrs2Y5WN4pP4jIS3b08CyY8dBvVWlRWw7uo2r3r2KlGYpDG4++NII3i+ar/MwIrfVPQN9sCNty02XuKg5siaVilViXvd5dotjLuJn9dwIYMSiEQz8eSCr+q6ibtm6IS/fyvdAb1UZYqqVqMbtV9/O+BXjyXJlXRpBzywOD4Iwis7bPo8/j/4ZXr2BQLDBxhXK4cKu9braajS2Y1KkVgQW0rtBb3ad2MUvf/5ityiW4s+jKK9j/46bFRykUXTMsjGUKFSCDnU6WCZiyAlzG1cglCxckg51OvDZms84k3Em4HThbNvSisBC2l7TljJFyvg2GmucTZAzbA/9fYhv1n9D13pdI2u56RCvnmuXI0Gv+r04dvYYX6+Pjj2NtbHYQgrGFqR7YneGLxrO/lP7KXdZObtFspTsrXk7jV955RIZvY34+TCKfrLqEzJcGZE3LAS5G5MjgOZVm1O9ZHXGLR+X6y5y2W1k4fC8Z0f3CCymV4NeZLoymbhyot2iaPJKPif+iQhjl4/lxso3cu0V11oooI2EyMZl16KCSil61u/J/B3z2fTXJusLtBmtCCymZqmaNLuy2fkNSUKJU8YsHTf270WOQw/5NIou2LWAjX9tjMzeQBTRrV43YlUs45f7WS7GIJxWwPWHVgQhoHeD3vx59E/mbZ9ntygag1yVZBBG0XHLx3F5wcvpVKdT/svXXIQdlWv5y8tzV827mLhqov/lYiIEbSMIAR3qdODJH59k7PKxtKjWwvLyImHMMlT4t2Pkb+Lf8bPHmbJuCl3rdaVowaJ+8taEC70b9Gb6xun8d9N/uaf2PTnGzfH/zee8oVA9O1oRhIBCcYV4uO7DjF42mr9O/0WpIqXsFilq8ackfZIPo+jnaz7nTOYZejXolafytZJwJndUv4OKl1dk3IpxuSoCv6SkuD3NPM+Op7eZkOCY9ZlMGRpSSrVSSm1USm1RSg3ycT1eKfWlcX2xUqqq17XnjPCNSqk7zJDHifRq0ItzWef4dPWnlpcVCWOWocbvPcqjUXTcinHUK1uPhuUb+rQ/rFxpiriaEOFZLuanLT+x6/iuvGeQTzfkULvNBq0IlFKxwPtAa6AO8KBSqk62aD2BoyJSHRgOvGGkrYN7a8trgVbAKCO/iOP6stfTuGJjxi4fSzgu6xEpWKkkl+9bzvJ9y+ndoDfKj8JITNRKOtzo2aAnIsJHKz/Ke+Iw2X/EjB5BI2CLiGwVkXPAF0C7bHHaAR8b36cCLZX7TWkHfCEi6SKyDdhi5BeR9GrQi3WH1rF4z+KQlKcrmdAyfvl4CsUV4qHrHwJ0zyxSqJpQlVuvutX/cjG5kQ835FA/O2YogoqAd59ptxHmM4649zg+DpQKMC0ASqk+Sqk0pVTaoUOHTBA79Nx/7f0ULVCUccvH2S1K1GP2i3U64zSfrfmMjnU6UqJwiZCXr7GWXg16sfP4TmZvm533xGGw/4gZisCXWsv+C/3FCSStO1BkjIgkiUhSmTJl8iiiM7g8/nIeuO4Bvlj7BSfTT9otjsZEpv4xlePpx+lV/1Ijsa70w59217SjdJHSeV8uJsi1mUL17JihCHYDXgvuUwnY6y+OUioOKA4cCTBtRNGrQS/+zvibL9Z+YbcoGhMZt3wcNUrW4OYrb7ZbFI0FxMfF05vdFRkAACAASURBVLVuV6ZvmM7Bvw8GnjDEazPlFzMUwVKghlKqmlKqIG7j74xscWYA3YzvHYE54raYzgAeMLyKqgE1gCUmyBQ4Id57tXHFxlx3xXWMW6GHhyKFjYc38uvOX+nVoJdfI7Em/OnZoCcZrgw+WfVJ3hKmpFxsE/AoA4e4joIJisAY838C+BlYD0wRkXVKqZeVUm2NaOOBUkqpLcBTwCAj7TpgCvAH8BPwuIjkwxqTT2zYe1UpRa/6vViyZwmrD6y2rBxN6Bi/YjxxMXF0rdfVblE0FlKnTB1urHwj45aP8+35l1Oj0uH7j5gyj0BEfhCRmiJytYi8ZoT9R0RmGN/PikgnEakuIo1EZKtX2teMdNeIyI9myBOg0LbtvdqlbhcKxhbURuMI4FzWOSaunMjdNe+O+NVlQS+N0btBbzb+tZHfdv528QUbGpVmEr1rDdno31uqSCnurX0vk1ZPytPGFxrn8d3G7zh0+pBeYC5K6FSnE8Xii11sNLaxUWkWes9im/ZenbNtDi0/acln93523u9cE360/qw1aw+uZXvydmJjInIuJJD7vtTRxGPfP8ZHKz9i71N7L7gKe1f+Hhw2aQz0nsW+sdG/t3nV5lxV4io9PBTG7Di2g5+3/EyPxB4RrQQ0F9O7QW/OZp7lszWfXQjM594VTiF6FYHNe6/GqBh61u/J3O1z2XJki6Vlaaxh/Ar3OvU9G/S0WRLr0bOkL1C/fH0alm948XIxYTBpLCeiVxE4wL+3e2J3YlRMrhtfaJxHpiuT8SvG06p6K6oUr2K3OJoQ07tBb1YfWM3SvUttb1SagbYR5HOdcLNoO7ktS/YsYdfAXRSILRCycqOdYJd//m7jd7T9oi3T7p9G+1rtzRLLedj8fjiVE+knKP9OeR667iHGth0bFktNg7YR+Mdm/97eDXpz4O8D/HfTf0NariY4xiwfQ7nLynFnjTvtFsU6QuQSGY4uqcXii/HAtQ8wee1k93IxYTBpLCe0IrCZ1jVaU6lYJT5c9qHdokQFZqzzvvvEbn7Y/AM9EntEbi8uAlwiraZ3w94XLxfj8EljOaF3KLOZuJg4eib24OX/vcK2o9uoVqKa7n47nAkrJuASV2Qbib29YFJTL7hFmugSGe67tXmWixm7fCy9G4b3PBLdI7CblBR6ztiNUsrthRJmMxLDjWC9X7JcWYxfMZ7brrqNq0pcZY2QTiHMXSKtRilF7wa9Wbp3Kav2r7JbnKDQisBOjO535RETaJN+JeNXjCdjYH/d/XYwv/z5CzuP74yOmcQWu0RGgktql7pdiI+Nz9vy1CFe6DIQtCKwEy+X1T5Tt7H/1H7++/N7jpyRGGnkt9IZu3wsZYqUoV2t7JvwRRgR4BIZCkoWLknHOh35dPWnnM44nXsCh65JpBWB3RjKoPUWqHgCxjREKwGHsu/kPmZsnEH3xO4UjC1otzjWEsJ5NuHYE/Cmd4PeHE8/ztQ/puYc0ckGeBEJu6Nhw4YSMbhcIsnJIiCDmyNqMLJt4CPucI2jeP1/rwspyMbDG+0WJXRkfw7D/bm04Pe4XC6pObKm/GP8PwIr33jfzx/JySG7r0Ca+KhTdY8gJ6wey8vW/e4xbTsKxbhVH+nut8NwiYuxy8fSvGpzapaqabc4oSOMXSIvwaJhGY/ReMGuBaw9uDa3yI40wAelCJRSJZVSM5VSm43PS3btVkolKqUWKqXWKaVWK6Xu97o2USm1TSm10jgSg5HHVEIxlpet+10l4Upa12zDhJuKkpFwue0Ph+YCs7fOZtuxbfRp0Cd0hTrQqBi2WDws0z2xO/Gx8XyYlst8IKeuSeSrmxDoAbwJDDK+DwLe8BGnJlDD+F4B2AckGOcTgY55LdfyoSHv7pun25b93OzyDGZsmCGkINPWTzO3DE1QdJrSSUq9UUrOZJwJTYGDB1/8rHmewcGDQ1N+JGLxsEyXb7pIsSHF5GT6ydzLD0W94gP8DA0Fqwg2AuWN7+WBjQGkWeWlGJypCERsG8vLyMqQiu9UlFaftrK0HE3g7D+5Xwq8XEAG/jQwNAU6oMKIWFyui99pE+/lbzt+E1KQscvG+o9ks4K3ShEcy3Z+NJf4jXDvaxwjFxTBRmA1MByIzyFtHyANSKtSpYqlN+s8wTw0QRil/jPn36JSlGw7ui3PaTXm89r/XhNSkA2HNoSuUJuNihGJxffU5XLJ9aOulwYfNhBXTnnaaID3pwhytREopWYppdb6OPLkSK2UKg9MAh4REZcR/BxQC7gBKAk86y+9iIwRkSQRSSpTpkxeis4f4mcsz+W6NF52grEveM80Xq5nGttNliuLD5d9SMtqLbmm9DWhK9ihRsWwxfMeWTgvQilF36S+LN+3nLS9OayO7EADfK6KQERuFZHrfBzTgQNGBe+p6A/6ykMpVQz4HnhRRBZ55b3PUFTpwEe4ewz2k9ND07DhBWXgq5KWIIxSRtoqIz6idXoVPdPYAfyw+Qd2Ht/JYzc8FtqC/TVE9DOQP0I0L6JL3S4ULVCUD9I+MCW/kOGrmxDoAbzFxcbiN33EKQjMBgb4uOaxLyhgBDA0kHJDYiPIPpaXlSWSmBjYuG0wXVAj7fRrEFKQqbX1kICdtPq0lVR4p4JkZGWErlBtI7COEAzL9JnRRwq/WliOnD5iet7BgkU2glJGJb/Z+CxphCcB44zvXYAMYKXXkWhcmwOsAdYCnwKXBVJuyCaUZX9IsrICr+CDtC9kKqTKAOSWbnlMqzGNLX9tEVKQlLkpoS9cew2FLcv3LhdSkBELR9gtyiVYogjsOmydWRxIBW9Cj0BAhtzk7hWsG9hFKwMbePrnpyX2pVjZc2KPPQJE2qzeKKLx2MZS671aORuNbcCfItAzi30hfibySADjtp44+TFKZUvb88cDFJRYRm38VI8Ph5gzGWeYsHIC99S+hwqXV7BHCAcaFTWB0S+pHxsOb2D+jvl2ixIQWhFkx5/Hz+DBgVXwwRilsqUtc9kVPFCvMx/fUIATCYV1RRBCvvrjK46cOcJjScEbicNxK0ZL8dfQiiDuu/Y+ShQqwei00XaLEhB6hzJvxMvjB9wVuHflX7z4pRU8XFrBp6RcvMuYJ24gFXm2tI83epxPVn/CpNaVeNy0H6rJjVFLR1GrdC2aV22e/0y8nwFf59FImGzyHiyFCxSme2J33lvyHgdOHaDsZWXtFilnfI0XOf2w1EaQ2/i+DeO2SWOSpPZ7tfM/3qjHmvNE2p40IQVJXZSa/0wGD5avKiZLs5tdxmPkkq8qJstHVw42Tc6wI8q8oTYc2iCkIK//73W7RTkP2licByychp4fJq6YKKQgc7bOyXti7X2SZ3pO7ylFXisiR88czV8GXhXcVxWTBVwynAvndj9PthJlM6ZbfNxCrhx+pWRmZdotioj4VwTaRpAdT1fVG5sNtfdfdz+lCpfivaXv5S2h91CX0zbCcCjHzh7j8zWf0/n6ziQUSshfJl52oY57UhFiGIB7eLHjriifHRxlM6b7JfVjx/Ed/HfTf+0WJWd8aQenH5b1CBzcdf3XL/+S2JdiZdfxXXlLGGUtsGAZsXCEkIKs2Lci+Mwc1rN0xBBhlD2PGVkZUmV4Fbll4i12iyIiukcQGCHcni+v9E3qi0tcua93np0oa4EFg4gwKm0UTSo1IbFckFtjOK1n6YS9cr17pFas95M9fYD5WenVFRcTx+M3PM7c7XNZfWC1NYWYgFYE2UlJubii9FSkNns0VCtRjbtq3sWY5WNIz0wPPKHTKiQHM3vbbDb9tYl+Sf2Cy8i7wuvf/+IKb8CA0N97pwwRWtnQcoqi83Heq0EvihQoQuqi1NDJkld8dROcfkTUnsV54KfNPwkpyGerPwssgYOHupxIm8/aSNm3ysrZjLPBZzZ4sEjjxiL9+7vvs8vl/t64sT2GeicNyZg9RJXP57xZM/fhuR2e83yRi1NG3+/6Svwr8XLw1MF8FmAO6KGh8Oe2q2+jesnqvL/0/cASOHioy2msP7SeHzb/wOM3PE58XHzwGQ4eDI0bw7vvXtwjW7zYHkO9k4YIzZ4x7f1cp6ZCTMyF4adQ/MYAelz9G/cnPSudD5flcWg3VPjSDk4/orVHICIyfOFwIQVZvnd54ImcYCR0OH1m9DG/xea0VrhTZLGKfBrng+oJeJedy/29fdLtUv7t8pKemR5kYfkH3SOIDLondqdogaKMWDwi8ER6zZocOXz6MJ+s/oSu9bpSpqiJmx45pRXu3UK1aFMW2/H8Rm9C+dsC+K8HNB7AvlP7mPrH1NDIlAe0IggzEgol0KtBLz5f8zm7T+y2W5yIYHTaaM5mnmVAkwHmZmx35eQh0ocIg1R08+a5D1Nk8CZb2XdUv4OapWoyYtEIxGnK11c3welHNA8NiYhsO7pNYl6KkX/98i+7RQl7zmaclXJvl5NWn7YyN2MnGuojeYjQzhn0efiv31v8npCC/L7zd+vl8gF+hob0onNhSNWEqnSq04nRy0bzws0vUCy+mN0ihS1frP2C/af2M7DJwNwj5wV/rXCwrxUeyUOEwSz0GCx5+K+7JXbjhTkvkLo4laaVm1ovW4AoCaKLopQqCXwJVAW2A/eJyFEf8bJw70QGsFNE2hrh1YAvcG9cvxx4WETO5VZuUlKSpKXlsDm0XXg/iL7OTSRtbxo3jL2Bd25/h6eaPmVJGZGOiFD/w/pkujJZ028Nyor/KoTPhMZmAvyvn/7laUYsGsH2AdupVKxSCAUEpdQyEUnKHh6sjWAQMFtEauDeqnKQn3hnRCTRONp6hb8BDDfSHwV6BimPfYR4QktShSSaXdmMEYtGkJGVYUkZkc7c7XNZdWAVA5sMtEYJQGS3wjUXE+B//USjJxCE95cE6AYeAoJVBO2Aj43vHwPtA02o3G9eC8BjQs9Tekch9szcfPrGp9l1Yhdf/fGVJflHOsMWDqNMkTJ0rtvZblFCS/bn0WmGy/wSJr+rakJV2tdqz5jlYzidcdpucYDgFUFZEdkHYHxe4SdeIaVUmlJqkVLKU9mXAo6JSKZxvhuo6K8gpVQfI4+0Q4cOBSm2ydg0oaVNjTbUKl2Lt39/23leCA5n4+GNfL/5ex674TEKxRWyW5zQ4YSlGKwgzH7XgMYDOHLmCBNXTrRbFCAARaCUmqWUWuvjaJeHcqoY41IPASOUUlcDvmpHv7WZiIwRkSQRSSpTxkRfb7OwwWc8RsXwz6b/ZMX+FczdPteyciKREYtGEB8bH/y6QuGETT1XywnD33VTlZtoWqkpby540xlDu75ciQI9gI1AeeN7eWBjAGkmAh1xK4LDQJwR3hT4OZByHek+atPMzTMZZ6TsW2Wl9aetLS0nkjj892Ep/Gph6Tm9p92ihJ5InWEchr/rvxv/K6QgE1dMDFmZWDSzeAbQzfjeDZiePYJSqoRSKt74Xhr4B/CHIdRcQyn4TR8WeLdAQjxzs1BcIZ5s9CQ/bvmRtQfXWlZOJDFq6SjOZJ4xfwJZOOCU2c5mE4a/q02NNtQrW48hvw0hy5VlqyzBKoKhwG1Kqc3AbcY5SqkkpdQ4I05tIE0ptQp3xT9URP4wrj0LPKWU2oLbZjA+SHnsweaZm32T+lKkQBGGLRzmN46Va66HEyfTTzJ80XDuqnkX111xnd3ihB5Po8WbSFhmIgx/l1KK5//veTb+tZFpG6bZK4yvboLTD0cODYnYOnPzie+fkAIvF5A9J/b4vG7KwloRwJBfhwgpyJLdS+wWJfQ4cbazGYTx78rMypSaI2tK/dH1xRUCOdGLzpmML1c1G33GBzQZQJZkXbL5hacnMH+++4jmnsGpc6d4Z+E7tK7emhsq3mC3OKEnUtccCuPfFRsTy6B/DGLF/hX8tOUn2+QIamaxXdg+szglxe2N4HnoPN3ShARb3dUe+vohZmycwdbkrVxR1O3J66n05893fzZr5v4MepGtMOStBW/xr1n/4vcevztqen/Iyd5o8dQB4T4D2tfvCoPfkJGVQfWR1alcrDK/PvKrdZMbsW5mcfThYFe1/zT7D2cyz/DWgrfOh3lWVmzWzH2YstJiGHI64zRv/f4Wt111W3QrAbi0cnzppbDywfdLmM7iLhBbgGdufIYFuxbw685fbZFBK4K8YvduSDlQq3QtOl/fmfeXvs/+U/ttk8OJfJj2IYdOH+I/zf5jtyjOwsENm2iiZ/2eXFH0Cl779TV7BPBlOHD64QhjcT53Q7KaTYc3SexLsTLgxwF2i+IYTp87LeXeLictPm5htyjOJAx98CORob8OFVKQpXuWWlYG2lhsIp5WkzcOcVWrUaoGXet15YO0D9hzYo/d4jiCccvHsf/Ufv5zs+4N+CQMffAjkX439COhUAKv//p6yMvWiiCveHedHbrt379v/jdZksWQ34bYLYrtnM08y9AFQ7n5yptpVrWZ3eI4Ewc3bKKJYvHFeLLRk0zbMI11B9eFtGytCPJKGLiqVStRjR6JPRi7fCw7j++0WxxbmbBiAntP7g19b8CXe3F+4lhNGDRsoonkxslcVvAyBs8bHNJytftofsmvq1qIXNx2Ht9J9Xer06N+D0bfNdr0/MOB9Mx0qo+sTpXiVfjtkd8sdcu7iEDci53kguwkWTS8PP9lBs8bzIIeC7ix8o2m5q3dR80mP65qIVwqt0rxKvRu0JvxK8az7eg20/MPByaunMjuE7v5z83/CZ0SCMQLx2meOikpF9sEPL1crQRs4Z9N/0m5y8rxzMxnCFlD3ZcF2emHpV5DVi0TYcM0+F3Hd0n8K/HS49sepuftdE6mn5Tyb5eXJuOahGTq/kUE4oWjPXU0OTAmbYyQgnz9x9em5osfryHbK/X8HJYpgsGDL34ZPS/r4MHm5G/Dy9//h/4S+1KsbP5rs2VlOJEXZ78opCC/7/zdHgECcS92qAuyxn4ysjKkzvt1pMa7NeRc5jnT8vWnCPTQkIdQdNdtcNMbdNMgCsQW4KX5L1lWhtPYcWwHby98mweve9CeWcSeZ8eb7IbXQOJoopa4mDjeuPUNNh/ZzJhlY6wv0Jd2cPphWY/A6ha7TcMBz858VkhBFu5aaGk5TuH+r+6Xwq8Wlp3Hdoa+8ECGAMN4tUxN6HC5XNJ8YnMp82YZOX72uCl5onsEAWBli927hxFiN70X/u8FKlxegcd/eNz2DTByJft9yON9WbBzAV+u+5JnbnyGysUrmyhYgATiXhwGLsj5Jsj/T3MBpRRv3fYWh04f4s0Fb1pbmC/tEOgBlARmApuNzxI+4twCrPQ6zgLtjWsTgW1e1xIDKTdsewRW2yBy4PPVnwspyAdLP7C8rHwT5P3JcmVJ0pgkqfhORTmVfsoyMQMiEKcDG/evsAQbn+9I5sGpD0rhVwvL7uO7g84LK4zFwJvAIOP7IOCNXOKXBI4AReSCIuiY13ItUQSh6q7b9PK7XC5p9lEzKTG0hBz6+1BIyswTJtz/iSsmCinIpFWTQiCw5iL0cJdlbD2yVQq+UtAU7z+rFEGeNq8H+gCfeZ07RxGIRHyLZs2BNRL7Uqz0ntE7x3i27WYWRI/M4y7aaGwjyXJlhUBYzSVol1jLeOqnp0SlKFm9f3VQ+fhTBEHNLFZKHRORBK/zoyJSIof4c4BhIvJf43wi0BRIB2YbvYt0P2n7GIqEKlWqNNyxY0e+5c4RCc/NLQLlqZ+fYsSiESzutdjvLl2ezWxs2bdAxL20tweXK6D7/+85/+bVX19lYc+FNKnUxEIBNTmSz/9PkzNHzhzh6nev5sbKN/L9Q9/nO598zyxWSs1SSq31cbTLowDlgeuBn72CnwNqATfgHjZ61l96ERkjIkkiklSmTJm8FJ03wnRzi0BJaZ5C2cvK8vgPj+MS10XXbN/WUvLnUulxF33o+oe0ErCTfP5/mtwpWbgk49uO57UW1uxXkKsiEJFbReQ6H8d04IBRwXsq+oM5ZHUfME1EMrzy3mf0WNKBj4BGwf0cTW4Uiy/GW7e9xdK9S5mwYoItMvhUMJ5KJB9eVf+a9S8UiqEth1optiYngvj/NIFxb+17SSyXaEnecUGmnwF0A4Yan9NziPsg7h7AeZRS5UVkn3IvBNMeWBukPJoA6Hx9Z8YsG8OgWYO4t/a9lCxcErgwFGTL0JA/l0rI0aVyyropTFk3hVduecUed1GNm3z+fxpnEKyNoBQwBagC7AQ6icgRpVQS0FdEehnxqgILgMoiF8YjDJtBGUDhdh/tKyKncivXEauPhjmrD6ym/of1ebTho4y6c9RF16xSBJ585893fzZr5qOcPNhodp/YTd0P6lKzVE1+6/EbcTHBtms0QRPhNrZwx5+NIKg3R0T+Alr6CE8Denmdbwcq+ojXIpjyNfmnbtm6PHHDE4xcMpJ7at3DbVffdv6arZvbB2ijcYmL7t9251zWOSbdMykqlICtRvxAiXAbW6Qqush/ezR+eb3l68zaNosu07qw8tGVlL+8vKXlmTn0lLooldnbZjP27rHUKFUjSMk0mgCI4H0b9BITUUzRgkWZ0nEKJ9NP0vmbzs5ffsJgzYE1PDf7Odpe05ae9XvaLY7l2O7NpXHeHhImoxVBlHPtFdfyfpv3mbt9Lq/+79WQlDlvXv57A2czz9L5m84kFEpg3N3jQrfhjCa68V4PKjXVPVfC4yFl8QrCoUBvVRnFeFqVc+cK3b7txqerP2VW11m0qOZc083TvzzNOwvf4fuHvqdNjTZ2ixNSwsJGEOmE+YQ5vVWlxi9KKUbdOYprSl9D5286c+DUAbtF8smcbXMYtnAY/ZL6RZ0S0DiACJ4wpxVBFOJrzPmu2y9jSscpHDt7jC7TujjOXrD35F66fduNmqVq8vbtb9stji0EM6SmCZIInzCnFYHmPNeXvZ6RrUcya+sshvw2xG5xznP49GFum3Qbx84eY3KHyRQpUMQ+YbK/8GFeAWgCJJL3kEDbCKzHwX7HvsacRYSHpz3M5LWT+aLDF3S6tpMdop3n+NnjtPykJesOrePHzj/SvGpz+4SJYPdBTYA4+H0OBG0jCCUe5ZqS4q4oXK4L4QMHOrrSUEox+q7RNK3UlAe+foBPV39qmyynM05z9+S7WXVgFV/f97W9SiDC3Qc1ARKhE+b0hDKz8bQahw27UHHMnw9t28Lx4xfGGB3QkvA33nxZwcv4qctPtJ3clq7TupKemU7PBqH11z+XdY4OUzrw287fmNxhsv3GYe+1c1JT3QdEjPugJsrxtUmB0w/LNqYJluy7MmVliSQmhu1GHafPnZZWn7YSUpD3Fr8XsnIzsjKkw5cdhBRk3LJxISs3IFyui//PMPkvNXkk0rYRNUBvXh8Csk86iY2FlSsvjhNGrcfCBQrz7f3f0vaatjzx4xO88/s7lpfpEhe9v+vN1+u/Ztjtw0LeE8mRCHYf1HjhGdL1/K9hMKQbLFoRmI33EIIvwqziiI+LZ2qnqXSq04mnZz7Na/+zZmMMgD+P/MktH9/CxJUTSWmWwsCmA3NPFCoi3H1QYxCltiBtIzAbX63GxERYtgyeeurC2HIY9QwKxBbg8w6fEx8Xz4tzX2TlgZW8eeubVCtRzZT8XeLi/SXvM2j2IArEFGBC2wl0T+xuSt6modfbjw6i1Bak3UfNJHursXhxmDHDPTyUnOw2ID/1lG93Qyvc0kzOM8uVxZDfhjDktyFkubJ4qulTPHfTc1wef3m+8/zzyJ/0nNGT+Tvm07p6a8bcPYZKxSrlOz/LCXP3QU2AhPlSEv7w5z4alNEW6ASsA1xAUg7xWgEbgS24N6j3hFcDFgObgS+BgoGU61hjsYjI4MEXG4Szstzngwe7z30ZnbKn8RidPWnMkMOMPA12Hd8lD3/zsJCClH2rrIxfPl4yszLzlEdmVqaMXDxSirxWRIoNKSbjl48XV4QY5DRhjrfTRxg6eeQEfozFwSqC2sA1wDx/igCIBf4ErgIKAquAOsa1KcADxvfRQL9AynW0IhDJm8dBdk8jX+f5Kd/sPH2wePdiaTquqZCC1B9dX0YvHS3zt8+Xg6cOXlKpu1wuWXdwnby76F1p/0V7SRiaIKQgd0y6Q3Ye22mKPBpN0ITo3bELf4rAlKEhpdQ84Glx70yW/VpTIEVE7jDOPfsWDwUOAeVEJDN7vJxw7NBQfvEeUvIQ7JikFXn6LEb4ct2XPDvrWXYe33k+vGThktQqXYvapWtzOuM0c7bN4cDf7sXsqiZUpWW1ltxZ407a12qvl5LWOIsInkHub2goFIqgI9BKLuxf/DDQGEgBFolIdSO8MvCjiFznp4w+QB+AKlWqNNyxY0fQcjsKK8YkQzjO6RIXu47vYsPhDWw4vIH1h9ef/4xVsdxS7RZaVG1Bi2otTDMyazSWEaG2oHzvWayUmgWU83HpBRGZHkjZPsIkh3CfiMgYYAy4ewQBlBs++PNPN6NHYGaeORCjYrgy4UquTLiSO6rn2qnTaJxNhC4l4Y9c5xGIyK0icp2PIxAlALAbqOx1XgnYCxwGEpRScdnCowsr/NO1z7tGo8kDoZhHsBSooZSqBuwBHgAeEhFRSs0FOgJfAN2AQJVL5GCFf7r2eddoNHkgKBuBUuoeYCRQBjgGrBSRO5RSFYBxItLGiNcGGIHbg2iCiLxmhF+FWwmUBFYAXUQkPbdyI85YDGExj0Cj0YQ3lhqLQ01EKgKNRqOxGL0fgUaj0Wh8ohWBRqPRRDlaEWg0Gk2UoxWBRqPRRDlaEWg0Gk2UoxWBRqPRRDlaEWg0Gk2UE5bzCJRSh4D8rjpXGvfyFk5Dy5U3tFx5Q8uVNyJVritFpEz2wLBUBMGglErzNaHCbrRceUPLlTe0XHkj2uTSQ0MajUYT5WhFoNFoNFFONCqCMXYL4ActV97QcuUNLVfeiCq5os5GoNFoNJqLicYegUaj0Wi80IpAo9FoopyIVARKqU5KqXVKKZdSyq+rlVKqlVJqFvWRJgAABX5JREFUo1Jqi1JqkFd4NaXUYqXUZqXUl0qpgibJVVIpNdPId6ZSqoSPOLcopVZ6HWeVUu2NaxOVUtu8riWGSi4jXpZX2TO8wu28X4lKqYXG/71aKXW/1zVT75e/58Xrerzx+7cY96Oq17XnjPCNSinTNnUOQKanlFJ/GPdmtlLqSq9rPv/PEMrWXSl1yEuGXl7Xuhn/+2alVLcQyjTcS55NSqljXtcsu19KqQlKqYNKqbV+riul1LuG3KuVUg28rgV/r0Qk4g6gNnANMA9I8hMnFvgTuAooCKwC6hjXpgAPGN9HA/1MkutNYJDxfRDwRi7xSwJHgCLG+USgowX3KyC5gFN+wm27X0BNoIbxvQKwD0gw+37l9Lx4xXkMGG18fwD40vhex4gfD1Qz8okNkUy3eD0//Twy5fR/hvB+dQfe85G2JLDV+CxhfC8RCpmyxX8S946KobhfNwMNgLV+rrcBfgQU0ARYbOa9isgegYisF5GNuURrBGwRka0icg73lpntlFIKaAFMNeJ9DLQ3SbR2Rn6B5tsR+FFETptUvj/yKtd57L5fIrJJRDYb3/cCB3FvnWo2Pp+XHOSdCrQ07k874AsRSReRbcAWIz/LZRKRuV7PzyKgkgnlmiJbDtwBzBSRIyJyFJgJtLJBpgeBySaUmysi8j/cjT5/tAM+ETeLgASlVHlMulcRqQgCpCKwy+t8txFWCjgmIpnZws2grIjsAzA+r8gl/gNc+iC+ZnQNhyul4kMsVyGlVJpSapFnuAoH3S+lVCPcLb0/vYLNul/+nhefcYz7cRz3/QkkrVUyedMTd6vSg6//0ywCla2D8f9MVUpVzmNaq2TCGEKrBszxCrbyfuWGP9lNuVdxQYlmI0qpWUA5H5deEJHpgWThI0xyCA9arkDzMPIpD1wP/OwV/BywH3dlNwZ4Fng5hHJVEZG9SqmrgDlKqTXACR/x7Lpfk4BuIuIygvN9v3wV4SMs+++05JnKgYDzVUp1AZKAZl7Bl/yfIvKnr/QWyfYdMFlE0pVSfXH3ploEmNYqmTw8AEwVkSyvMCvvV25Y+myFrSIQkVuDzGI3UNnrvBKwF/eCTglKqTijVecJD1oupdQBpVR5EdlnVFwHc8jqPmCaiGR45b3P+JqulPoIeDqUchlDL4jIVqXUPKA+8DU23y+lVDHge+BFo9vsyTvf98sH/p4XX3F2K6XigOK4u/uBpLVKJpRSt+JWrM1EJN0T7uf/NKtiy1U2EfnL63Qs8IZX2ubZ0s4LhUxePAA87h1g8f3KDX+ym3KvonloaClQQ7k9Xgri/uNniNsCMxf3+DxANyCQHkYgzDDyCyTfS8YnjcrQMy7fHvDpYWCFXEqpEp6hFaVUaeAfwB923y/jv5uGe/z0q2zXzLxfPp+XHOTtCMwx7s8M4AHl9iqqBtQAlgQhS8AyKaXqAx8CbUXkoFe4z//TBJnyIlt5r9O2wHrj+8/A7YaMJYDbubhnbJlMhlzX4Da8LvQKs/p+5cYMoKvhPdQEOG40dMy5V1ZZwe08gHtwa8p04ADwsxFeAfjBK14bYBNurf6CV/hVuF/ULcBXQLxJcpUCZgObjc+SRngSMM4rXlVgDxCTLf0cYA3uCu1T4LJQyQXcaJS9yvjs6YT7BXQBMoCVXkeiFffL1/OCe6iprfG9kPH7txj34yqvtC8Y6TYCrU181nOTaZbxDnjuzYzc/s8QyjYEWGfIMBeo5ZW2h3EftwCPhEom4zwFGJotnaX3C3ejb5/xLO/Gbc/pC/Q1rivgfUPuNXh5Q5pxr/QSExqNRhPlRPPQkEaj0WjQikCj0WiiHq0INBqNJsrRikCj0WiiHK0INBqNJsrRikCj0WiiHK0INBqNJsr5f7dSaXggWzWYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#TODO...maybe try an example like this\n",
    "\n",
    "xs = 2 * random.random(100) - 1\n",
    "ys = 2 * random.random(100) - 1\n",
    "\n",
    "training = array([xs, ys]).T\n",
    "targets = ys > sin(4 * xs) + randn(100)\n",
    "\n",
    "plot(linspace(-1,1),sin(4 * linspace(-1,1)), 'g')\n",
    "\n",
    "\n",
    "#scatter(xs, ys, c=targets)\n",
    "scatter(xs[targets], ys[targets], c='b', marker='+')\n",
    "scatter(xs[~targets], ys[~targets], c='r', marker='x')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}