Week6/Recitation_Week6_NeuralNets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1><center>Week 5: Introduction to neural Networks</center></h1>\n",
    "\n",
    "<h3><center>CSCI-UA 9473 - Introduction to Machine Learning</center></h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<img src=\"neuralNetImage.jpeg\" alt=\"Drawing\" style=\"width: 400px;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perceptron learning rule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This week, we will start working with neural networks. For each of the exercises below you can use the method of your choice but you should display the final boundary of your classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1. \n",
    "As a first exercise, load the binary dataset below and code a few steps of the perceptron learning rule. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy.io as sio\n",
    "data1 = sio.loadmat('perceptron_data_class1.mat')\n",
    "data2 = sio.loadmat('perceptron_data_class2.mat')\n",
    "\n",
    "data1 = data1['perceptron_data_class1']\n",
    "data2 = data2['perceptron_data_class2']\n",
    "\n",
    "# put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2.\n",
    "\n",
    "__2a.__ Load the data below. Using the neural_network module from scikit-learn and its MLPClassifier model, learn a classifier, for the dataset below using \n",
    "\n",
    "- One hidden layer with a linear activation function and \n",
    "    - One neuron\n",
    "    - Two neurons\n",
    "    \n",
    "    \n",
    "    \n",
    "- One hidden layer with a non linear activation function (take Relu for example or a binary step)\n",
    "    - One neuron\n",
    "    - Two neurons\n",
    "\n",
    "How many neurons, hidden layers do you need to learn the distribution of the data? Do you have an idea why?\n",
    "\n",
    "Try increasing the number of neurons and hidden layers. Then try different values of the learning rate. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWsAAAD4CAYAAAAqw8chAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeLUlEQVR4nO2dbYiuW1nH/2t7NB1N1HOsTJ2ZgrAkSD0D+VIhahElGUEQjGFR7C9RJw2img8VsSkoQr8UDFZa58mQk2ho9IIK9cmY7RE7pyMFnT3jKc0j0ev+kLJXH+553DNznpf7Xuu61nVd9/r/4GZmnnnue73/17Wu9XKnnDMIIYT45pp1BAghhGyHYk0IIQGgWBNCSAAo1oQQEgCKNSGEBOAejYfed999eX9/X+PRhBAyS27evPnFnPML1/1fRaz39/dxcnKi8WhCCJklKaXTTf+nG4QQQgJAsSaEkABQrAkhJAAUa0IICQDFmhBCAkCxJoSQFSwWwP4+cO3a8HOxsI0PxXpOWNQujTC9tRLSHYsFcP06cHoK5Dz8vH7duCrmnMWv+++/P5PGPPhgzjs7OQ91a7h2dobPI4VpkQ5lHnww5729nFMafkomRePZmvGVRDOee3uXq+Dy2tuTC+MqAE7yBl2lWM8Fi9qlEaZFOjZQKwiafU+0vlJSXLX79JRWV8OUZJ6/ij7F2to0sAjfonZphCnxTKH8lxAEzb4nUl8pLa7afTot601ICZz1MNoqfFrWA4L5L5E8zT7Ua1+5Cumqom2bWDTjGGItmTPWw2ir8OmzHhDMfwlBoGU9IC2uLZpZ6wFyDLGWzHkLd4CX8C3cL95muATzX6Ja0mc9IC2u1gNoDWKItaTA9WpZkwHB/JcSBK4G0etYIqxaGUsMsZYUOOsu1zr83hHO/7kJgiXMy83EEGtpgbOuFdbh9w7znwRkm1in4TuyHBwc5MkvH1gsgKMj4OwM2N0FbtwADg/F40ZWwLwnxJyU0s2c88G6/6u8KaaIw0MKhAXLfbW3bw9/L/fVAiwPQhzBs0EioHlWxtHRXaFecvv28DmZBVrVp/URLt0fGbPJR1J6me9gnBNR9tXSTyyK9z1irefRW4ZnVZURYoKRrCfCvtoIa9wCEWGPWOsVqq3Cs1zMRbEei1ehiLCv1pPgz4AIe8Ra7/1qFZ7lNgmK9Rg8C0WEfbXe92UbUpK1EfaIzdWyttyA3KdYT20hnoXCc0eyxPuJRwKUiG5p0UXYIzZXnzUt65aUlKpzoXDrolnixZWiRGvRjbJHrHW13BaeRHzosx6DVMmXtBDHQhGG2vJzPIIorR41NoD3/tkbktWHq0E2IZnTJS3EsVB0heR6NcHWViq6tAHaMYe8jiHWkjldM/aUauA0i+xQ6Hi9uDPmhHQT8e7JHEMMsZbMaesWYh3+ujj10nkomFg1RdpT1o9Fo4nQso5oWeds20K81RqPnYcmSiZWj6KrlWaNJjKHah5DrOeQ00u8jcdqWkZEhfLWWQZFs0lqNZGI1fUiImIN4O0AHgXwCID3AXjmpu+brgaxxptYlLaMqB1o1Hg7Q7Mae2siXqgWawAvBvA4gGed//1+AD+26R7zTTGWaIhFTUdW2jIityhHHb+jqExCc4DovT8Nu3TvXKw/C+AFGM6//jCA79l0T9dinbP8ypKaml16v3RrjapaFXhY+1t6n3Zf7bU6hN8UA+ABAP8D4EkAizXfuQ7gBMDJ7u6ufsp6QaLVlLSMCHuenSOVhaXZV7uKpcMii73dHMDzAXwMwAsBPB3ABwG8ddM93VvWklhNWEq21sgulQqkis7Kk+XV+tXE80FOY94U8yYAj+ecn8w5fwnABwC8duzLDUglu7vTPpfi8BA4Pgb29oCUhp/Hx2Wv+jo7m/b5TJAqutLsq832w0Pg1i3gzp3hZw9vebNqbmMYI9ZnAF6dUtpJKSUAbwTwmG60CpjrO39u3AB2di5/trMzfK6NVGv13AIUkSq60uzrNNurpMCyuW1lk9m9vAD8KoDPYFi690cAvmrT95u7QebuYIs+Hp1J+ZQUg+VJcDPJ9kl5KJHmsKtBSq7mYt2pTzQUVkonhLXwtV4NIoFFRxVZCvoQ6xavvnIiGpOJGndrdbxCZBGwQKr4pua7tw3EU+hDrDVbkjPRmETkuDtTx8giYIFU8U3Nd2fVZhLbxHrMBGMbWs0KTA3n6Ai4ffvyZ7dvD597J3LcS5cyKE009zpZV4rUAqCp+e56grCWTUpeepkc5DRmuF8STmSTKnLcS0wkxZFE5EFKKRanHKyKw9R8j+r5Qwg3SKuxS0k4kcdVkXdFlLRS5bKSyo4IYmJ1ysG6Z3nPLwliiHUrC7C3V35F3288tZUGGEl4yNYxWJ1y0DMxxNqzZZ2zr1o3NS5eTvJpkYcBRkFRDkgK0O+pwXXWm2hlbkQxa9bRMv4RX7UWoHyjHD0aoN9TIfype1Mv1y8fsOg2pcJs2YIkw2oZb0+joBVoZgUPSqwv/tCn7pVcPHXvApK1vuXYNGq8FbHcOj6G3o8gl8hbz6fuUay1iWqh5hxzRKBEhNUNM8jmKiTST8u6ZyL6fqWJGu8LRBDC6Nlc24lJNDX6rHsm4qoKDaLG+5wonpyo2SwhkpIbcbgapCVeam10c4fknGNY1pZ4mNiL3tT6FGtvpeal4yDFeKtSnvA0sRe5qfUp1jSDnkpJLY5c8xVgdqwm+sSeF/oU6ygOxlaUmD40JclIok/seWGbWPs5IlUSyfMs5/Bux5KjUq2OV63N7zmUVzAkmpvk+5lnyyYlL73MLWupbtpbd186Di8xfSxGJ7X57a28rkTNkwtFMj6Osz0U6NINkrNMbfTkSKtpESXpsDj0qja/PZXXBaTFrLZqa4irt84oInHE2mNpe/J91whRK591rQrU5ren8rqAtzM7nPZp3RNDrL2OozzV6loharEaxNoy9lReF5DsQySS6LRP654YYu20kbnqRLzm0UUkOpQZ+qwli05CaCNUpR6JIdaeu3pN98yUZzsVoktIbUOrdcg6c6dJFh13+s2XGGLdY1df6hN2JkSX6EAFSotAqugkFzp5rkq1RExfDLHuoJE/hbl2UBFbyUi8VNMZZ/ElajrG2nKyyOMYYp1zu9zxUtM9u37ISubav3qk9UpVqbBriCPWLfBiGuXMlh8Q9q8yjLGXappHbTlZNc1tYj3P7ebrsNpCvYobN4Cdncuf7ewMn08h6vbqgPGWPMWgVxYL4Pp14PR0kMDT0+Hvq8V/drb6/nWfX6S2nGrCVmWTkpdebi1rb6aRx61oLQga76DRdsVYq7X1HjCpsGsA3SAXmJvrQTM9mr79wOXgZcpDG610jrWXJJbcl8afPmsPzM000hopaOeTtxEOuYRm8U/ppy07Rq4G8cCcTCMtC1Xb8g1sWbfEqqpqD9jmZC9J0qdYz0mQN6FV87Ut385abOmxLFZZ1KL4e2ieU+lPrDVquefapRG3Fpav5zwVpLQ6Wg4+OPCxoT+xlq5pnVmBOec+06xEaXW0dOuz+G3YJtbzW2ctvUjS09rsVvAdS2KUVkfLNd0sfp/4EWupTRLStbxW/ANu/gAwtMxbt4A7d4afJS21NO1R82wFpdVRas9UKRLFHxmXVXCT2b28ADwPwEMAPgPgMQCv2fR904OcpMdwlqvzI1Oa9pnlWU1yrE/465XQ66wBvBfAT57//gwAz9v0ffMjUiVrq+WJMpEpTXvpfY4VqmXULIRGOn3WRRl2ByOA5wJ4HEDa9t3lNauXD+RcXnu8p0uT0rSX3Dcza7yG1kIjnfWWOxeXWDVbCbF+BYC/A/AeAA8DeDeAZ6/43nUAJwBOdnd3p8VyrhboXNM1hpaWdc/5fIXWQiOd9R68jpEt6wMAXwbw7ed/vwvAr226hy8fOGeu6RpDS591zyOYK7QWGumsr3meVNrD+qwBfB2AWxf+/k4AH9l0j+uXD7RmrukaQ6sZsoYK5b04WwuNJ8tasuMIezYIgL8F8LLz338FwG9u+r75dvPISE+OelYWKRopVJSBUuQJzZ7n86XE+hXn/uhPA/gggOdv+j7FuhDPSxi900ChtMUgat/qZTVI9CovItZTL4p1IZJqEN3McIimazy60HghaoeX83ax9rODsTUetyhJbpV3+26iuGhuAe/xVAMN5rzzMrZY12xnHvMiOMkwxyCpBlLPkkqvx85xIppbwNm3kq1sMrtLryZuEIuZCO2xqjeftVR8LPZcK6EVHXqtCGbrs7ZY4xPtnOfaZ0ml12vn6IiOkkrWMF+xtlg9XxqmM+twNFIzap47R0dErSZEhm1iHddnXeOTLXU+loRZ4x+3RsrvXfqczhy5WpNj2tMF0s9vMb0xNgxXUy2blLz0cu+zXt4/1YwpCTOydWjts46cd06INM3SIr5TwmjtmkIoN8hUAbUYN04NM/q5FVJ5rN050oewEu3+ztN2c+kwWtsKccR6rjMstA7rGCPCBnWntm9o1bdo2wqeDnKSDqO1nRVHrOcqanPthDzRuO5IeOBaVQla1uVh0LJeR3R3wSY4RNelcd2pbcQtRYA+6/Iw6LNex1wtazKe0k7t3ntX151771WJZm3f0Nou0bYVvBzkpBFGSzsrjljTXbAdzZpj7YStKf/GYh3JsiZxiCPWOdNdsAnNzsyDE9ZiR2ohHrKLzI9YYl1DqdC37CBqwtI0xzyYih7e5zQB64EImR99iHWpqdLSxKkNS9N69OCErRFcmqpkBsQS61Jzo7Sht7TIPFivnuNmsSOVEEfEEeuaxlpq2bX0ddaGNXef9fI5FFzSKXHEusY668Gyznneq0EI6Zw4Yl1jefbgsyaEzJptYu3niNSa4zgPD4HjY2BvD0hp+Hl8vP2MydL7SmgZFiFkdqRB0GU5ODjIJycn025anvt88a2hOzsUNEJIF6SUbuacD9b9349lTcuTEELWco91BC5xeEhxJoSQFfixrAkhhKyFYk0IIQGgWBNCSAAo1oQQEgCKNSGEBIBiTciSxQLY3weuXRt+LhbWMbJFMj+85m1NvFqnadP2xtLL5DxrQmqIehyA1pkskvmhfQhZafpr4qWQJoQ5G4QQS7QO9dI+fEtLBCXzQzNva9JvcXjcBijWxA2uD+bTOC5X21rXPDVSMj+0jiKuTX9NvBTStE2s6bN2Tiu3mHY4y6NfTk+HWn16Ovy9LZxmbsGag8TWcXR0+awbYPj76Kj8mRc5O5v2+RQk80Mjb4H69NfESytNm9ik5KVXz5a1pPXYyo3aIpwSI6ipG1kjMO2XW2ha1hF81rXpp8/ap1i3GIJLl1+rdyO0CKdEt5q/B1e6kmgnQLs3k7Y8pBugRPprJygF0xRWrCNaqNJts9Vbx1qEU5I3Ld+6pkKLiud6IqABM0p/SLGOaqFKi8ucLOuSMm1uWWswIzEhuoiJNYCnAXgYwIe3fbdWrKNaqNLxnpPPehnOFN2KuvSZCNFZRycp1u8A8MctxDqqhaohLq3qq9d24TVeRJkOe2oRsQbwEgAfBfCGiJZ16/fiUlxIEZ7fXt86jFn4wKYhJdYPAbgfwOvXiTWA6wBOAJzs7u5WRTqyhUo6Q6piaW/JbjHRKRmG9PBaWgAUBKVarAG8GcDvnP++VqwvXt5Wg0SHedGQKZktKVCalmQLK1U6DMnnSXckSp2fhFj/OoAnANwC8HkAtwE8uOkej+uso9Kh686OqZktKSias+AtZtg1LGGvHaFS5ye6dK+lZU0GPLruZmvpT81sSYGiZf1UpCqadEei1PltE2ueDSKM9FkWmsc/lFB6xkcIpma25PkQN24AOzuXP9vZGT6vRfPZmmEcHgK3bgF37gw/Dw/LniN9jofFuSDANMt67NXasvZi6Wm4LKQMFqk88mjpizE1cRq+UK4GkacXn3XJ1VKsrY8HuIjWSFAifVJ1q/UW8KbtvySjvAoUuUwPq0FKrhKxLk17rUBGELLaeiHZibS0rE0mVym+xIgQYl3TKGsFMqqQTUGyE2kpoF7zkyjReUcZQqxrGmVtg44qZFPQ2BHaok2FP3WPjIfbjGOIdU2jrC3jqEI2Ba+dyDZoWXdE5AN8hAgh1hJ+51KBdFx2onjsRLYRtmwsM1si7NJn1IQd9WhMQUKItXWj9C5k3uO3jdrONFTaLSuz5dIh6yHu2IoypVNoXPlCiHXOARtlI6w7slrCxd/T0pupSIRd+gzLZVlT7h0bT4OKG0asyWocj9pGESr+Eg3UclZUIuzSZ0iE3WL97tgyNqi4FOvgtGz7GqObUCs6LC1TCSJb1jVMrWRjKrrGM7fQtVjPwbUSfZI8lGUtZR3SZ9023RqVTMNa30K3Yu3FV1rbYbRKh5aoeimHUUhlQtQVGR7CLkGjkmn4wbfQrVh7sOik6lCLNqDprggzwvHSs3iJRyQ0KpnGCpMNbBPrNHxHloODg3xyciL+3Clcuzbk2FVSGk5cbMH+/nCE6FX29oYTHz0RKa6qLBbA0dFwLOru7nDEZ+nRnKWwMGIhVF4ppZs554N1/5/tedZWR85exNtZ1JtoceRxCKTOUK4hUsXRRPpweC0aNZ7ZinVJ/knXDQ8dxlgOD4Hj48EYSGn4eXxso1UXidJeRYlUcbSI9JaLVo1nk4+k9PLgs855mhvLeo6CPJVu86/bhF/Aw6RTYxBtgtFqMkpzNUSL9ISZxJtAh+31Ll4K1GplSKgF+jKEEmtLgyJy3fBoiEloTeQymQWWa6477KlDiXX0zV9WeIu7VOfhLV3dYbmbUdoC8TJS2UAosba0pDxap2PxZoFK7i2JWibN0BQhy3NCcpZLW5CKFEqsrS2pAJ3vSqzz7SrSb9+JWCZN0BahiOeERIjPGkKJdZAO0B2S+SYhjkHaxmpKMmBus+JLIp4TsgpvQ881hBLrnGlJlSJ1JITU9nhPbXU0JRGPMiveelWHp4YcxHoIJ9bEDsk67amtjqYkAyLMikfoPTVHNNrpF6rs3Yh1SHFwRpDRoh4lGRBhVty7ZdliRKMlEIIdQRdi3dJw0Cxz687Ge5tWJ5plnfO4iuO9F46Y7wrx6EKsW5WbVqeg+dwpHUCE0bIq0XzWY/EibOuINqJRikcXYh39LfYaz62ZyLe28E3jY7UaRDOhNR1KiwLwZFlPTS8t62m0Mhy0OgWN53o3psYQwWgVoUVCSzuhFgXgZURjHI8uxLpVnYpkWXsZJdYwhw5nFF4TOjVeNVa4h/XtpeXA1SDTaDFai+Sz9tr+pzCHDmcUUi/qlW4AU9dxRx8GGVe4bsS6FVFWg8yh7cyhwxlFbUK1CntKvOZQWMZpoFh3jLfJwqnMocMZRW1CNSfbxsZrDsMg4woXSqyjiwuRJ2SdaO1/9fBq+jlY1jmbVrgwYu11Qpxsh/l6AQvrzINQdjMM0iOMWGvXtznUJQ1RrH3mHPJVFAvhLF1y5q0yRaW31SDaLi+pNiQhbiX3e1lWehUPRp0rrHy3UyqWhx7Ww1I9CTytswbwUgAfB/AYgEcBPLDtHo+WtdTqqJpyqblfI38knjmHeSVRIvRe1nG02HyiJfSedjACeBGAV53//tUA/hHAyzfd49FnLZGntc+ouV9DFCWead3uxenBL2Tdw5ZUmpqKVlomY+qC57NBAHwIwHdv+o7H1SASbai2XGru92pZR9Cm0UglxuNw/SLWPWzrg5tK0ju2LniyrC99GdgHcAbguSv+dx3ACYCT3d3dyREtYWqbqG1Dlpa1V5/18jmetWk01iLWCusetrVlXSL0Y8Pz5LP+yheB5wC4CeCHtn23xaYYi/pm6bNe3s8JfEWs3QOr0Cogy4Jv7bMuEfoWr0y7gohYA3g6gL8E8I4x328h1lZGkNVqkIiES6s3y9raAt4Wt9YNoeVSKoO6IDHBmAD8IYB3bvvu8moh1h6NIHKXkC4Wb+LorfNY4i2fxlDiM22cRgmx/g4AGcCnAXzq/Pq+TffM2bIm4wg7eelpOODVIuml8TWuC9vEOg3fkeXg4CCfnJyIP/ciiwVw/Tpw+/bdz3Z2gONj4PBQNWgygmvXhhZ8lZSAO3fGPWN/Hzg9ferne3vArVs1sQuC1wyQKFzyFFJKN3POB+v+f61lZCQ5PByEeW9vqCN7exTqdSwWQ7u/dm34uVjoh7m7O+3zVZydTft8dty4MVggF9nZGT6fimQlkChcMp1NZnfpxSNS/WDlXuxyK7vX5TrSlSCizzoAiHI2CNHBUvB62Az4FTxHVqMSePLtX8RrvEZAse4cqTkqqzYQpu15HgZ4naiUxnOHOYJtYu3KZ23hW507Eu7F5WTu6enQAk5Ph79blM/h4TCXdufO8NPtnIRnB3svPuajo8srDoDh76Mj3XBbCdcmJS+9vBzkJOXus9wE48GV4NloFKc0wz1nUlSLc2pZGOw6dLndfMrl4YhUiTz0sL3cw8aSklF0GPfFRWoy3LsgRisQzV2HkmXl9SCnsZeHlw9I5KHlwU1SaZBgajy869ZaajM8miBuQzI9U59VUhZjK55kw/J8ROqYy4NlLZGHlkeiSqVBgqni66WTmYyXDPeAZI9b8qzSshjTKUiWc4+WtbQ1RstalimGUVjN85Th1kjmRcmzNMtC8tk9+qyX6ZYcddFnfflZrUboGvMPTeJu7b/x5EaR7HFLJz20ykJjk5BAuYUSa2m4GuTuM1pqkPUIugrLBeWeHP3WlnXOumXhqWM8p2uxJgMWo3upthDSM1GSeOmhuace3ltH5BSKNYnrQ84B414qTJJbTT36zhxast7YJtZhj0gl4/F60uYYwsW9NMJSCQ2XYWTJbI9IJeORPGmzNeHiXrrtXCqhnre9kyoo1h0Q+ezvcHEvPYdDKqFzPAeEhwYNbPKRlF70WZMxzNKNaT2Z1ir8VoXn1QevADjBSCSQrufWmqaKtShoh9+y8KRWyQSocKHE2rqOr0MqXhrpa5FnGvU85JI8MtCy8KRWyQSocGHE2utoRypeGoLXyljQqOfhluSRu7QsPKnKF+AtHGHE2utoRypeGoLXyljQaJtuDR0P215LqQl3yr0tC0+qQUvEWdk6CiPWXkc7UvHSELxWBo5G23TpQvRwoEwpNeFOvbc2rKkdipcdmcqdVBix9jbakY5XZMtaS3/czVFIZKjVkKEm3JJ7S0XXeqVMTYVTto7CiLWn0Y5GvCL7rJdhuRJWDTwcgl5KTbiRh2gtoWV9Fy+jHY14ST5H+5ndQstaN87RZ5Xps5aHAkaKoM9aN87RLeucuRqEEDdwNUisBfszYptY89Q9Qkg7Fgvg6Gg4WGp3dzioyu1BL23ZdurePS0jQwjpnMNDinMhPHWPEEICQLEmhJAAUKwJISQAFGtCCAkAxZoQQgKgsnQvpfQkgBVv7XTLfQC+aB0JxzB/1sO8WQ/zZj2r8mYv5/zCdTeoiHU0Ukonm9Y39g7zZz3Mm/Uwb9ZTkjd0gxBCSAAo1oQQEgCK9cCxdQScw/xZD/NmPcyb9UzOG/qsCSEkALSsCSEkABRrQggJQNdinVJ6aUrp4ymlx1JKj6aUHrCOkzdSSk9LKT2cUvqwdVw8kVJ6XkrpoZTSZ87rz2us4+SFlNLbz9vTIyml96WUnmkdJ0tSSr+fUvpCSumRC5+9IKX01ymlfzr/+fxtz+larAF8GcDP5Zy/BcCrAfxUSunlxnHyxgMAHrOOhEPeBeAvcs7fDODbwDwCAKSUXgzgZwAc5Jy/FcDTAPyIbazMeQ+A773y2S8A+GjO+ZsAfPT87410LdY558/lnD95/vt/Y2hwL7aNlR9SSi8B8P0A3m0dF0+klJ4L4LsA/B4A5Jz/L+f8H6aR8sU9AJ6VUroHwA6AfzWOjyk5578B8O9XPn4LgPee//5eAD+47Tldi/VFUkr7AF4J4BPGUfHEOwH8PIA7xvHwxjcCeBLAH5y7iN6dUnq2daQ8kHP+FwC/BeAMwOcA/GfO+a9sY+WSr805fw4YjEYAX7PtBoo1gJTScwD8KYCfzTn/l3V8PJBSejOAL+Scb1rHxSH3AHgVgN/NOb8SwP9ixDC2B859r28B8A0Avh7As1NKb7WN1TzoXqxTSk/HINSLnPMHrOPjiNcB+IGU0i0AfwLgDSmlB22j5IYnADyRc16Owh7CIN4EeBOAx3POT+acvwTgAwBeaxwnj/xbSulFAHD+8wvbbuharFNKCYPf8bGc829bx8cTOedfzDm/JOe8j2GC6GM5Z1pIAHLOnwfw2ZTSy84/eiOAfzCMkifOALw6pbRz3r7eCE6+ruLPALzt/Pe3AfjQtht6f2Hu6wD8KIC/Tyl96vyzX8o5/7ldlEgQfhrAIqX0DAD/DODHjePjgpzzJ1JKDwH4JIbVVg+j823nKaX3AXg9gPtSSk8A+GUAvwHg/Smln8DQwf3w1udwuzkhhPinazcIIYREgWJNCCEBoFgTQkgAKNaEEBIAijUhhASAYk0IIQGgWBNCSAD+H4FA+BuDA/lWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import scipy.io as sio\n",
    "import numpy as np\n",
    "\n",
    "data1 = sio.loadmat('data_Week6_class1_XOR.mat')\n",
    "data2 = sio.loadmat('data_Week6_class2_XOR.mat')\n",
    "\n",
    "data1 = data1['data_Week6_class1_XOR']\n",
    "data2 = data2['data_Week6_class2_XOR']\n",
    "\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.scatter(data1[:,0], data1[:,1], c= 'r')\n",
    "plt.scatter(data2[:,0], data2[:,1], c= 'b')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# put your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__2b.__ Keep the dataset from above. try to change the intialization of the training algorithm. Plot the resulting classifier for a couple of different initializations. What do you see?\n",
    "\n",
    "Do it for a small network first. Then repeat those experiments for larger architectures. I.e. increase the number of neurons and the number of layers. What do you see when you change the initialization?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neural_network import MLPClassifier\n",
    "\n",
    "# put your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3. \n",
    "\n",
    "__3a.__Load the data below. Try to build the best neural network you can for this dataset. Split the data between a training and a test set and evaluate the models you built. What is the best validation error you can get?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAApi0lEQVR4nO2df4hm11nHv2ens7TTtGR3EsNiOjMoIpRQahNswVJia0Vj8EdBUUaoIKwMFmJL0OiCTYWhduuP+k+FVRfavmND8VdDKWiIFStIy2yT5gcxzT+7S2ObRNOiy/7nHv9479u988695+fznPOc+z4feJmZO++995xzz/me5zznOecaay0URVGU9jhROwGKoihKGirgiqIojaICriiK0igq4IqiKI2iAq4oitIoryl5s9tuu83u7OyUvKWiKErzXLp06b+stbcvHy8q4Ds7Ozg8PCx5S0VRlOYxxlwZOq4uFEVRlEZRAVcURWkUFXBFUZRGUQFXFEVpFBVwRVGURlEBV5QRDg6AnR3gxIn5z4OD2ilSlKMUDSNUlFY4OADOngWuX5//feXK/G8A2N2tly5F6aMWuBLEqlmj587dFO8F16/PjyuKFFTAFS8La/TKFcDam9bolEX86tW449JZtQ54VVABV7ysojW6tRV3XDKr2AGvCirgipepWaMh7O8DGxtHj21szI+3xip2wKuCCrjiZUrWaCi7u8CFC8D2NmDM/OeFC21OYK5iB7wqqIArXqZkjcawuwtcvgzcuDH/2aJ4A6vZAa8KKuCKlylZo6vIqnbAq4DGgStB7O6qYLfK4rmdOzd3m2xtzcVbn2f7qIArygqgHfA0UReKoihKowQLuDFmzRjzhDHmC93fDxtjXjTGPNl97uNLpqILMRRFWSbGAn8AwHNLx/7UWvvW7vNFwnQpPVZ6IUZoz+X6nvZ+ylSx1no/AO4E8DiAdwP4QnfsYQAPhpy/+Nx9991WiWd729q5dB/9bG/XThkzs5m1GxtHM72xMT8e+r3Qa4SmZ3vbWmPmP1OuoSgJADi0Q9o8dPDYl4C/AXA3gHuXBPwygKcAXARwauTcswAOARxubW2VzfVEMGZYwI2pnTJmQnsu1/dCr+ETZ4qOYOwe2jEoHpIFHMD9AD7Z/d4X8DsArGHuhtkHcNF3LbXA02jWAs8VptCey/W9kGuEiHPuQxi7x94e3QhBmSw5Av5RAN/srO1vA7gOYLb0nR0Az/iupQKeBqUXoBi5iZ7NrF1bK2OBh3wndxg0do/QPCorTZYL5XtfPmqBn+kd/yCAR3znq4CnU2SUnXqTofNyLNYh8ef0gYeIc64FPnaPsc+E/GPqIcqHQ8A/A+Dpzgf+aF/Qxz4q4IJJtZjHzssRJpe1OpaefqexsGoXauFTkJD75Y4oxu4xJuxDHUODStjk6FEgJAKe+1EBF0yqhcnhGkh1V1B2QkPnxwjo8neHfN3r69a+5jXH77m+Tj+JWkn8m52/EYYKuOImVTRdroFUwaHuTELdNlT+aNeEZV9ENzeH77e5SZ+3SmbwykZQEaMCrrhJFQiXCOX41FMEJ1ctqNQmtCxj7peTtopmsFrgNIwJuO6F4mAyC/hCMsKx52jqhtqp+9fmbnxNtXF26BsUYu6Xk7aKb3TQrWyZGVJ1rk9LFvhkJl9iMpJiMadYhlz+WAo/McVDj1k8FPNsQtwyQ+dWNoMbnHsVB9SFEsdkhn5UKxFzr9+/D2fPmKsWFGrD1Wnu7d3006+tWfue9+RvNaA0gQp4JJOZfKFaiThG7LmT6Rk9UJudQ+U88RBE5SYq4JFMRmeoViK6iBGHyfSMhRl7RlRlqQIvmjEB10nMEZqbfBmbqAzJSO4kV8xkpb5hN42YCcfYslzp/YobZ0jVuT4tWeDWNmSU+NwYqSsROYYb6o9NI3QlJ+ekq1INqAulEVJ6Da6d8rhEtZmeURA5USg+XIux9BmJQAW8BWotYFncm2ojK+UIZEXEVdYh/nUdJVVFBbwFOJaQcwrsCrlDSi8qZWMoI669YNSlIgIV8Bag3sSJ+2UBBX2nOf3Q2Lmua/b/t7lp7cmTacUoyr3s6k36GaaMblFIUAFvgdwNi5bViFs9CoUEhlqxocblWN928uTNrV1Ctu8OKcbQIiriiQqtD6J6HcVaFfA2oB5vcwtsoYYecpuxohvba2ts48GYD9U+UsXcLDG9iSi/j5It4Ji///IJ3Hyhw2kAjwF4ofs5+FLj/kcFPABKU4xbYAs19JwX5nB+XLsFxLhfihm8MTdK8TspbFAI+IcA/HVPwM8DeKj7/SEAH/NdY6UEXEJFLyGwTPnsX3bMWt7c9Lttxz65FnjMnlPr6/O0jhWR6z4x5eQt/tz6oJZ5NbIEHMCdAB4H8O6egD+P7jVqAM4AeN53nZUR8JCKXkrgC3YkoZOCvmSEBEWsrx+3bMdEPtQH7rufS4QXpFjTY53J2pq7jIbcQ149zakPoZkrbLxIsJW4yRXwvwFwN46+E/O7S9/5zsi5ZwEcAjjc2toql+Oa+Cr6BC0ZX4ADxX5Xa2v+l9mM3d/nDdjcnIt0//yFZR8jCinTDrEWuK+DY5tr5N4YLYEJNqVBkgUcwP0APtn9Hi3g/c/KWOC+ij7BWX5XlmKzG6ITHIsHKSy5lEcbe47P388W7ReS0MJ1e4JNaZAcAf8ogG8CuAzg2wCuA5ipC8WBr1ZNcEc+V5ZisytQJ4JJsQhjz/H5/Nk8GiEJLVy3J9iUBiEJI1yywD++NIl53nf+ygi4r6JLVR8PLiGgtMBDpxCkDp1TBDPmHJcFzl5OvoSqBc4Ch4BvdhObL3Q/T/vOXxkBt9Y/oydVfWzcgpi+P5nKBz6WhpTvTJExH/jiPdJ9iguc+sBZ0IU8nHCbXESEimLMgpjlBSkUUSiKn0V5AjejWIbKtYqLYXl2OCR8h+h2U61bKuBcNGIChCYzdkHM1HyNLRHyTKu6GBppGy3QrICL710bccKFJjN2UYywbK4UIc90b2/4O3t78/+ztq9G2kYLNCngTXTgjUyDhyZzrM2NLYgR9SxWjJwtBhZizfpMG2kbLTAm4KLfiXnuHHD9+tFj16/Pj4uhkXc8hiZz7BWaf/ZnwIULwPY2YMz854UL7tdfehl7j6cSRMgzdb3ulL19jSXw9GmiGyiiLfAmOnABw4ScyUmW6A7qBFHf1/Wd2MzHfJ/YX5HjA3ftBUPWvmaz4f0O1tfH8y7eZ1oHtOhCEedCC1mTXbjSFRVmygSlBIf7hDc3eDy2U4ktfIaOPqVYfB/S9hUSvuRKrPrprLWNCrio5ykgMUONVVwnlztbOmT+UYVb+L5DtaZ96PsVH1S/3vh2YSSv0jHPWVxllkOTAm6toBFV5co1pmFjDbGamyl3tjRV/HI2UFl8J9Zn5wrZSS0XZlxJ7i/aImtzMc9ZSBlJpFkBF0PlyjXWDsaSxbrSjmIpdcyIJjfcIvQ7VBa4MQKWRMYleZEM8oFmzAWFlJFEVMBTcfkpmCrXkEbGxGe75oiSE7A4TrlJSaipF9KwuXzgrj1lXQ8mpMMChte/M+IrAhYNDX3OAtyUUlEBT8E3A8RQucbqcMj+131NIE9AjMM9ZQzuOie0YcdGoQwt8+7nc1mch+7pehDLaZnNEt/CQIurmHyDHXaX5nIZFe7gpKICnoLP8maoWDELacY+WV4dl0hzuZFCrWdK5aAyRUN9W7GdIFe+PbiSV8RAVit8EBXwFCr4vV23XG7LMRFaJAng8lHW8H367hn67F1ul9xOMMTlQyzuFAOwLNQPPogKeAoVKlPMLZOMlZxJSC7rqMYEse+esQ+iX6ZD56V0gpXM4bEqUuQxaSTKICrgKVQYzs1mx9/NSLZwjWqij3pIL9ECz3n2VMJbYzSUmK22btIeKuCpMPkgXYEey6uPT54kui3nJGQONfyenH53qk7QZc27rHwm1Adej2QBB/BaAF8F8HUAzwL4SHf8YQAvAniy+9znu1aTAs4Aq5+xVIgBx6RiyU6D+54U1/ZFQVWwVF2GB1lR1qgLwskRcAPglu73dQBfAfCOTsAf9J3f/6iAz2EL9MiJrIixfNRKKoerV1/+LJ5BYQGcenWQ0J+QuFAAbAD4GoC3T1bACzwtNtdmjl+XanJN4cG1kms51rywmhapDpVUVErnlCXgANY6N8k1AB/rjj0M4DKApwBcBHBq5NyzAA4BHG5tbZXNdSxMT6uvj67NhLKDC0LMd4oQA40UKE9uXDpj58q++KeiikqxVags8FsBfAnAXQDu6IT9BIB9ABd954u3wBmeVqgbkyTQIyf9K2iB5whLcYMwVMQqdK7s0Y4V65sUW4UsCgXAh5ddJwB2ADzjO1e8gDM8rRD3peut4lHktJZCPvBaorl87t5emaIiJaQAYsUusyeazdy7A5Bob0UVlWKr5Exi3g7g1u731wH4MoD7AZzpfeeDAB7xXUukgPcr8Jh/I+NpxWxCRSIEIQ2SIpQgoeGX6l+WzxsSGN8eVK7s+SzOqhNeBSejx0aX/e1LSLR3rMDX1tgLunkfOIC3AHii83U/A+D3u+OfAfB0d/zRvqCPfcQJeIh/I/NppYTysvbujfoTU85NicJbbFngKiJXp0y151YWoTfINC9DTnfV/+C8F2invttPIgol9yNOwAv07Kkiwkaj/sSUc1M7T18RuaqNr2ilWHTW2mzzOHTOnGRDT+aRsnRUwIco5FsbikLZ3ibYjCrFNBDoT1xb83t6Utqsz301tmFgSFTFkAiHdMgh/SerxUcohKG2gC+UPVp3pcwsFkQFfIjKMxRZ1ljqyRXz7LLGQrZkibXeXKKxsTGfyBwSylSRDTkvtXMgEfGQQo14G0hsWsl0V8rMYkFUwIdgbC1DDTz0WBCplbhwnoe+E2L4UXi3QibZQs8LnTD1neebAGX1DoT4lCI33ompv2S6K8oPVQYV8DEYxqtD9evkyeO7DGbVuRxzplCeUzbZo8jecrpSssp13lg5DYU1ZlmpQ4SGRDFZsqS6K2FmsSAq4AWJmTxLbivChpExycmJXpjCKDnG/VLcAifrLYZZMd0lY0zAT0Ah5+pVnu8eYX8f2Ng4emxjY368AmP5GDoeknRh2SNldxe4fBm4cWP+c3fXXQ/I8j1UqENsbbn/f3AA7OwAJ07Mfx4cRCXj2rV5T3HlCvDAA9Gn+8lMX1MMqTrXR4wFzmwGFLHArRVlzsRazKH+ciHZYyc2QieZfqFubh7ffH59/fiLnpfPz1iFu+xGXLgXyfI4Uf841IXS4XJCEqlFaOx3S/Uq1bfbSv5q4ys/ts7MJ+gxs7AeXIYN2dKLifreVMAXjD3gsaDgREL8mmyWFbG5GirOTVjMghM5lrRinSNFHKSD0DnUapP7glEBXxCzOQlBr13EIGBu4ZMxamLKSZDQFyv/EPFjssDJ8jaZynoUFfAFMbWIoNcuYj0xV9rJGDUxSwcF+YNCyz+7zwldwUTsAyerW7OZe2vEhlEBXzBUAX1b0xHcktV/ydgBWSvQqEkt0FAlFJZhZl29yd7e8I329o5+L6NCL2vsiRNERZ26aqsRVMD7LFfAnM2hM26bdfmQmdJMwen3D8RTBO4b+kJTUp9VqDA3sEhqWZtI+pwKHRfZYEdYp0uNCrgPZp8n+ajc5woimIQdG6iwuIRDCyinoXLfg9H14vMOkLi5KvnKSJreZPx8w6iAV4bcQHBNxhIobHGDhtM6Xg6Vc8U5L74vcKMw1+VbtcBdRAm7sLRTowJeCTY39dQmLrn806linLLSiPwhH8VVRCTGv6DJ2+ikCEo7B8kCDuC1AL4K4OsAngXwke74aQCPAXih+3nKdy0xAl4oRIzcTb1sSZLujnWUogbNbBa+DV9sQ+XKSIXJcF9WSKp1yIqtAm0n6bEJCv2kJkfADYBbut/XAXwFwDsAnAfwUHf8IQAf811LhIAX7KlJ3dRD6T550u8OSKRYMbl6OYoYba6hxNjDZZztrW5kFkzAxF3a0ZC4UABsAPgagLcDeB7dezABnAHwvO98EQJe0LQkdVNXihBgN2jG8kW1CQhXufkebsym5RHfj/k6+fMrWAfJblXQKue8VZaAA1gD8CSAawtLG8B3l77znZFzzwI4BHC4tbVFl6NUCnbtpPW9gklSpO5z54vLaqR4uMxRK+SXLlgHQ0PSnRQcMXDfisoCvxXAlwDcFSrg/c+qWeCkD7WwBZ6d9lD1L5Evjp6I4uEy5p3l0q1Z4K2l1wFZFAqADwN4sFkXiqvhMS3CILlkYQdoVoWMSWt1x24GuQ+X0aJluXTIsyKq8CTpH7rA4kMM9+AkZxLzdgC3dr+/DsCXAdwP4ONLk5jnfdcSIeDWDleyFoTE1TiIO5+sCpkS6jfR6AEnrVng1vrrIFEbIkn/WGTT2lp0eoqk10GOgL8FwBMAngLwDIDf745vAni8CyN8HMBp37VSBLxY2y4w3GLLC0Pnk1UcGkIQRmEfuDGRPuRYCNsQSdEUtMCb8IHnfmIFvKhRzCw6rHlh6Hyy0lshYqZZGC2Uvb1Ce9gsIG5D2UXDHeFEnV4HTQp40WEgs+iwXp6p80mukC24o2oieTGMgBuyzhs1Wh+bFPCiEzHMOxKyGvgSLd5V9Wv7mPJiGIK8kRfPbBa+ylcwTQp48VAoRtEhzctyOgtth6sQMKHQtkEy2xBLmicwJ9OkgLe+GGHBbEb4ohDXCEJKCKRa3+PE1L/McmzRk+UrnqQiGWp8aoHzC7i1DFogYEEMkPGikJj0ZxRecuNvUTVKEvr8iMqRtS9luLhvgBxdJLPZ8HvcTp5sqk42K+DktLQgZohQCy4zn8npluiPD6DYoCH0uUgvR6Z25LpsUpGMnbS5mZXO0qiA99nbuzmxsbbGGhxL7rEJrcWZApCc7or+xlQRLj5oCEmodL8tYwczVjxJRRJj8Ah2+6mAL2jdAg9Nf6YAtGaB5zxWkcYuU6LIdKpCB0NqgfdPasDtpwK+QIAPPLtuhIwgMvPZmg88J7sijV2GciS9ZIVeL9kH7jtJZA9+FBXwBa1vyxpai4licluJQgl5rGPJiplXLJat5dCl5Fnvm5CHslboqJOege8kkT34UVTAFzTQ2zopFIXCQsU4e5fehGhRUb1iuhm5TkmrX6k0oAkq4Asq+rtI6ntha4GsjTKXu+/yIQLvymfRNs50swZ0qg5DoYbr66I6JBXwPhUsBzL9KtgKW/OZuh6rq98TFxTCuLeNcJ3ywtJ0Z7N5XHi/YITFiauAh8Ik7mT6VXAEQaq5lf2MrnBgcWHZjBEoxXWKsD2xVf0GhiYq4CEwiiOpfhUaQZCmubIFPvZoQ1dZT8EHXlyniPPBlv4pT2ICeFP3HsznADwL4IHu+MMAXuxedvwkgPt81xIj4LnhCAm3GrosaeNhEPViUQsEaQ+diFy+TUzbLbj+i+V5Ftcp4vbElv4pW+Dd+y7f1v3+BgDfAPDmTsAf9J3f/4gQcFdLZ9iQfmw7YlILjsliI73sWFgc0U1S22BMCKHwtR5eiusUcXtiS38DD5fypcafB/DeZgXcVQuIa4jL8ia14BhbJokh6GogRGlP1YrQttuAkeZlKK/r6/O+lMUbR1xorDorPCSSRMAB7AC4CuCNnYBf7t6VeRHAKd/5IgTcF45AWEPGbkVe+aT78FwNmSjtOVoR4hqp8nIEBkHpX3Zz8/ikJqnhyaC4bFEogsXbWgIBB3ALgEsA3tf9fQeANQAnAOwDuDhy3lkAhwAOt7a2imZ6kNyAYIJbkVtw0vc7dqkfkZWWqhUiLfBCQ/oieSIWR3KtZXQ/UqYzS8ABrAP4RwAfGvn/DoBnfNcRYYEX9Hf5fOAkFtxQbNhibCzFknApBeHzSGk0In3ghMqaGhtPBnMYoTGZrkimwAXqupIziWkAfBrAJ5aOn+n9/kEAj/iuJULArS06ZIqNQolOmivAmSkvSXujuGr00IWFhUqOzcFWTZQHX7GzW+CFwggX3s8kGHoxjnLNEfB3ArCdr/t7IYMAPgPg6e74o31BH/uIEfAKhIa5Rdf3Qs7Z7LYYI8gFzd2QxlY8SIFIAUK8haz5KhRGmCWOY+7HDAOIo0nqQh4B+DQsqb4Xcs4W9QEXvFmIiBWPQCFS1pwdGkkoFEaYJY4MAi7KAqf8iBLwCjPPvlsm1fdC5qEvbaTFKWzDripBPsuJSnhpdfXQR4YwwrFnkZwnhocrygdO+REj4BUC91ktvRD1zFTYQvOQ/ptVoHpyEgu4aDUfm8cgTsDe3nHNlVjXREWhUH3ECHiFFlnV10pwYdclyIuzQgcrOjkZBVxkoOkqIIYwQtLJ5L294bJl3SchHhXwPhXGxDHRDuQNjjDWeihtLMVZwcUlNjkMWzyQ5qWQQcTSkVYfXoWhAt5HqAXOBnOH5QuTlBKOnkNVASeOCycXwUIGEUsbkr6KuUMFvI8QH/ii7pAKwpDSMPceQ3kT4v0goboLhTABLFUh9KKZvSCL1qoF3qCAW1s1CqUv3qSCMNbQ9/bYFcjVTwhsD1GIaONE9ZXN3cWyyOEoDBF/AnrnMFTAfRQUdDZB8IWKFMhfVV8/EyLCCIkKiM3dxbLI4SgsAm5t4Y3e01gJAU+u84V7YTZBEODPqxptk4Gr7rS6kCf00kWeA0HdrDZ6EMDkBTzrORRuoVUs8EKwxrtXSnPxNl5gzqK4u4sgT1X995WZvIBnPQfXJgsMVI33LuC78N0i1pLiTnLoqKGYy6fQSIr1NssFRjAP03IETS6TF/Cs5zDWgrO2OXPjE4Qsd9DYiUKGizGdbYkki2vDhaxCttu4JtMze0Fyd7Va4DIEPOs5zGYMmyykwyZaQiprTP5ik5zS8QkplpsU6LWWVzS2UM9YikWIUeNj8gKe/RyGKlwlM4xNUASZmqFCG5Pk1Dogsg0z+mxcE5kk+5wz1TPWEYPwkKjJC7i1mc9BkBnGprOC8hhKTJJzsieqDTMnxjWBSdJ5MdWzVdiyYYyVEPAsBJlhxX2TQiuttXFJFjTASKfAM3LN2ZPUNaY8kLeLhtpDzht53gTgSwCeA/AsgAe646cBPAbghe7nKd+1RAu4tcMz50O9M3OvzVqvctJeyVoJvW2DA4zjFMiEzwLP6vT6MYqL2UaiukLeLhqqMDkCfgbA27rf3wDgGwDeDOA8gIe64w8B+JjvWuIFvI9rJr1Arx3al7DST8Tm5vGXJwuzVsQaVDEdX4FhhG8xT7KGFXgApFEoDQ3ZyFwoAD4P4L0Anl+8B7MT+ed95zYl4GO986L2FOy1qwhTSCsXaK2Ic2nGPrxCVmHfUCbbl6fAAiS1wDMEHMAOgKsA3gjgu0v/+87IOWcBHAI43NraKpbhbEIchYV67Sr1LGScLdRaEYXv4TEseHEx1MGRdXrMFu3YXijqAw8T71sAXALwvu7vIAHvf9QCT6PKSC+0AxNorYjC9fAYF7wMwa5XjJbGbMZkQ4gbsg2TJeAA1gH8I4AP9Y5N24VS2QfeR6wFHpPvRhoKOa6HV/jBst+OsYdwVces9DewE6G1GQIOwAD4NIBPLB3/+NIk5nnftZoScGvHRWd5GRvJ6gd3MkT4wNfX53mNFeGGhqrkuPJeeGhV5HZMHbVrQJh8i0beh2ltnoC/E4AF8BSAJ7vPfQA2ATzehRE+DuC071rNCfgYFQSpigFLddOGJotI8YXUMbsclh9d8cdAWGnH0p61F/iYO3RtLeOix6EoBl3IQ4kAQWrKI9FQuFYWsWGXTIaACO8fcd729hjeYjVm0gMZFz0KVTGogFNSWZCa80gI6PDYSQ27ZOiJXcVdrOMnfOZDRWsMgaejgAVOVQwq4JRUFiQRDTSG2ey4NXryZFjicjNUqkAEhV2KeAUcYf7ZmlsBHzjVs1ABp6SyCeya0BFpmc9m8wnQfsLW1/0Jyy1nqucU0gkICrtk8Re7GDORCfLPFj64gDkKRS1wqVQ0dblC1NmylFqLc2t/6vkpWwhQh10GJm/oOQ0NeEL7zCTG8p7ptPZ5pVrwwKkPXDnGWKVwWSohjZ7Nek8dR4aeN5a5lPvm+LKpwi4dhD4n8lWLLlyjjwyLwNUnihhZBqJRKMoxYsLENjf9jZ7Vrc9pgbsULeW+Ob7sAqOy0CwV9YMzVR6W2O9GUQHnRsDs4ZiWhVhjrA0+1bwPOc83oxt7X0G+7KEqFfqcis6zMw3fViF4KRQVcE4ExfWlNnr2xpLawfnO82Uu9r4Ffdkucjpj1/lsyWYwYAQ1q+qogHMi3FQI9USkRPpVH3hQl31BX7ar3HLcYaH3kMZQelvLAxcq4D3IK4XwlYYhlkxKpJ8IC4kjEcyqEZJk3yaGYkSNKDEi6pJgVMA7WCpKAytrUi2+lPm+4gMPIWUcSki5iSlbF4SNqYn8VkQFvIOlooxV5Apbz6aSMogQPvCoiqtPCSk3URbpWGYIG5PWJTcq4B1sFSUmrk+gWdG0BS4Mn/iGlpuIgYUrM4SNSeuSGxXwjqIVpSGzIsXiE2UlMpEior461lS5uTJD2JiaKpMKqIB3FK0ooeEfBV8O4SJFrMbOcV1LhGUZQGpdCXWRtFAGSa+Ey5jIbKJMKqAC3qNYRfFV8KHQDyB8pz6huLIdGhET83y4nmeqgTkpd0DIcEJVl52cN/JcBPAygGd6xx4G8GL/DT2+61hBAl4UVwV3LRppsrXPyRl1xxp1VEbg0GNK9YBNyh0wqcyUgaNPyxHwdwF424CAP+g7d/mzkgLuwrVsW6CfPBSX8PlEMdZ6pbB2c1c9jl1zMobppDLDC1d/l/tW+h0VcAbUAj+W3Virl2KemGLVYzOoGLPC5T4bE/ATSOcDxpinjDEXjTGnxr5kjDlrjDk0xhy+8sorGbebIPv7wPr68eMnT87/BwAHB8DODnDixPznwUHJFCaxvw9sbBw9trExP+7638HBPJtDbG3lHXcV49Wrw9d49VXgwgVgexswZv7zwgVgd3f4++I5OADOngWuXJnrypUr87+X61ShOtdg1fYyVpfGjmczpOrLHxy3wO8AsAbgBIB9ABdDrqMW+ACuKJSG/Y+xUShDWaXygVPFZTdPaFRUgTrXcNV2UtoCTxLw0P8tf1TAI2lqxUceY1ldW8uPQvEVI8sbzyUiYlvKorcpTis+8DO93z8I4JGQ66iAR9Lcmut0ONc8xYYyG0P+akQZhKhmocVnDa1xi6ZkFIrXB26M+SyAfwfww8aYbxpjfh3AeWPM08aYpwD8eCfiCjUhDt5z54Dr14/+//r1+fGGiPVxU117qPisBb74xfz7spHqPHZNQCzgfBABlzt9mvQ2VdjdBS5fBm7cmP9knTMZUnWuj1rgkYRY1xMxZTgHEoW28yhD7koon3lY0Ade9MXLjQNdidkouQ7ehuB05Y9du7niK7HRSqE5laIvXm6cMQE38/+V4Z577rGHh4fF7rcSLELD+n6AjY15vBsw9xFcvTofs+7vNxwDx4Or+EQW1YkTc51bxpj5mH1nZx4euMz29nw8LwhfVpSbGGMuWWvvWT6eEweuSGB3dzhYGQiL+Q1likG7GC++bPEOKa+UMvX5qIsHIqdTyN0+bYbMcq6PulAKQukbmEikSzEogtPH2Nsbfq6LsJmGfEJarcKB+sBXjNjZuZRNtwSKgghCysu1fj9nzqOCKua4zCewhKEIKuCrRozo+hp9c6EalQkpL9dGZi7xFbbZuFrRZVABXzViWpZP7MfCBTY36dNcOgyFgxwLPPW8SqMhYcmZLCrglakyVAy9qc+qSxXwmEzXCgTnINUHPvYRvPJWB2dlUAGviLA2dxzqPV6tjc90rMsnpjfkMBNDFsT0/7+3N7yDV/9YaGC0IMexWuBlUAGviPhKzrFdX+w5oZ1ESm9IOaGbkobQ7wvt6X0LOwUmeXKogFekiWEmdSuNEWSXP3hZ8GM7htlsvqVhjHXvy2tsGjhHF8zkrtxXaFABr4h4CzwEDreFzw881EnE9Iau66dO6MamIeX7gphE3Z0AKuAVWclhZo4lu1CIVIH1fde1yTjHntkNq2DDfc+kUAGvzEoOM32Z5p4cTbl+6siBwgcukIb7nkmhAq7II1UdQnvDlOvHTDjG9MiMPTh3+Hyjfc+kUAFX5MGtDqnXb2i4lFOEodlsqDgmS7KAA7gI4GUcfaXaaQCPAXih+3nKdx2rAq4Mwa0OE1efnEGMWtbtMCbg3v3AjTHvAnANwKettXd1x84DeNVa+4fGmIc6Af8d386Huh+4otCSuqd2Q9uGK8jYD9xa+68AXl06/HMAPtX9/ikAP5+bQEVR4kndU7uhbcMVB6kvdLjDWvstAOh+ft/YF40xZ40xh8aYw1deeSXxdoqiDBHynuIh9GUK04D9jTzW2gvW2nustffcfvvt3LdTlJUi9Y1CqcKvyCJVwF8yxpwBgO7ny3RJUriY6FvRVp7d3bnf+saN+c+Q18GxvUpOKUqqgD8K4P3d7+8H8Hma5ChcLF7eS/WKTCWf2h1qivArsgiJQvksgHsB3AbgJQAfBvAPAD4HYAvAVQC/aK1dnug8hkah1EOjDmSx6FCvX795bGNDrWBlmJwolF+x1p6x1q5ba++01v6Vtfa/rbXvsdb+UPfTK95KXTTq4CalLF/Xfc6dOyrewPzvc+d40qJME/ZJTEUGU4g6oBDeUq4k3320Q1UoUAFfEVqPOqAS3lKWr+8+U+hQlfqogK8IrUcdUAlvKcvXd5/WO1RFBirgK0TLUQdUwlvK8vXdp/UOVZGBCrjSBFTCS235jvnlQ+7TcoeqyEAFXGkCKuGltHxdfnm1sJUSeOPAKdE4cCWHg4O5z/vq1bnlvb9fVxA1tl4pxVgcuAq4oiSSupWrosSSvJBHUZRhNBRQqY0KuKIkoqGASm1UwBUlEZ2oVGrzmtoJUJSW2d1VwVbqoRa4oihKo6iAK4qiNIoKuKIoSqOogCuKojSKCriiKEqjFF2JaYx5BcDA4mMntwH4L4bkcNBSWoG20ttSWoG20ttSWoG20kuV1m1r7e3LB4sKeArGmMOhJaQSaSmtQFvpbSmtQFvpbSmtQFvp5U6rulAURVEaRQVcURSlUVoQ8Au1ExBBS2kF2kpvS2kF2kpvS2kF2kova1rF+8AVRVGUYVqwwBVFUZQBVMAVRVEaRayAG2MuG2OeNsY8aYwR9xofY8xFY8zLxphnesdOG2MeM8a80P08VTONC0bS+rAx5sWufJ80xtxXM419jDFvMsZ8yRjznDHmWWPMA91xceXrSKu48jXGvNYY81VjzNe7tH6kOy6uXAFnesWV7QJjzJox5gljzBe6v1nLVqwP3BhzGcA91lqRAfvGmHcBuAbg09bau7pj5wG8aq39Q2PMQwBOWWt/p2Y6u3QNpfVhANestX9UM21DGGPOADhjrf2aMeYNAC4B+HkAvwZh5etI6y9BWPkaYwyA11trrxlj1gH8G4AHALwPwsoVcKb3pyCsbBcYYz4E4B4Ab7TW3s+tCWItcOlYa/8VwKtLh38OwKe63z+FeUOuzkhaxWKt/Za19mvd7/8L4DkA3w+B5etIqzjsnGvdn+vdx0JguQLO9IrEGHMngJ8B8Je9w6xlK1nALYB/MsZcMsacrZ2YQO6w1n4LmDdsAN9XOT0+PmCMeapzsYgYNi9jjNkB8CMAvgLh5buUVkBg+XZD/CcBvAzgMWut6HIdSS8gsGwBfALAbwPov9KatWwlC/iPWWvfBuCnAfxm5wZQ6PhzAD8I4K0AvgXgj6umZgBjzC0A/hbAb1lr/6d2elwMpFVk+Vpr/89a+1YAdwL4UWPMXZWT5GQkveLK1hhzP4CXrbWXSt5XrIBba/+z+/kygL8H8KN1UxTES51PdOEbfblyekax1r7UNY4bAP4Cwsq383n+LYADa+3fdYdFlu9QWqWXr7X2uwD+BXN/sshy7dNPr9Cy/TEAP9vN3T0C4N3GmBmYy1akgBtjXt9NCMEY83oAPwngGfdZIngUwPu7398P4PMV0+JkUak6fgGCyrebvPorAM9Za/+k9y9x5TuWVonla4y53Rhza/f76wD8BID/gMByBcbTK7FsrbW/a62901q7A+CXAfyztfZXwVy2IqNQjDE/gLnVDcxfvPzX1tr9ikk6hjHmswDuxXy7yJcAfBjAPwD4HIAtAFcB/KK1tvrk4Uha78V8CGoBXAbwGwtfXW2MMe8E8GUAT+OmP/H3MPctiypfR1p/BcLK1xjzFswn0tYwN94+Z639A2PMJoSVK+BM72cgrGz7GGPuBfBgF4XCWrYiBVxRFEXxI9KFoiiKovhRAVcURWkUFXBFUZRGUQFXFEVpFBVwRVGURlEBVxRFaRQVcEVRlEb5f7etExUNtNg6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "import scipy.io as sio\n",
    "data1 = sio.loadmat('neural_net_ex2_class1.mat')\n",
    "data2 = sio.loadmat('neural_net_ex2_class2.mat')\n",
    "\n",
    "data1 = data1['neural_net_ex2_class1']\n",
    "data2 = data2['neural_net_ex2_class2']\n",
    "\n",
    "plt.scatter(data1[:,0], data1[:,1], c= 'r')\n",
    "plt.scatter(data2[:,0], data2[:,1], c= 'b')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# put your answer here \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__3b.__ With the same dataset, add additional features to your model, e.g. $\\sin(x), \\sin(y)$ or other monomials. Can you improve your classifier ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4.\n",
    "\n",
    "Let us go back briefly to a simple dataset to make sure we understand how things work. As a first exercise, we will code a one hidden layer neural network that outputs a binary 0/1 variable indicating the class of our data. Throughout this exercise, we will use the notation $z^{\\ell+1} = \\sigma(a^{\\ell+1})$ to denote the output of any neuron from the $(\\ell+1)^{th}$ layer and where $a^{\\ell+1} = \\sum_{k} w_{\\ell+1,k} z_k$ is the combination from the previous layer that is fed to the neuron.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"diagramNeuralNet3.png\" alt=\"Drawing\" style=\"width: 450px;\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the function ['minimize'](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html) from scipy. Check the documentation of that function. We will set the 'jac' parameter of the optimizer to 'True', which implies that the (objective) function that we provide as an input should return both the value of the loss and the value of its gradient. \n",
    "\n",
    "For this first exercise, you are asked to write a function $f(W)$ wich takes as arguments a vector $W$ containing all the parameters of your network (as a first step consider building a network with only a few hidden units, in order to make sure the model is working). As indicated above, the function should return (1) the value of the binary cross entropy for the given set of weights and (2) the value of the gradient derived through Backpropagation (as we set the value of 'jac' to True). \n",
    "\n",
    " We will split the writing of your function into several steps. Once you have coded each step, you should gather them together in a single $fun(W)$ body that you will then pass as input to the minimize function. \n",
    "\n",
    "### Exercise 4.a.\n",
    "\n",
    "Start Load the data using the lines below and plot it using scatter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.optimize import minimize\n",
    "\n",
    "import scipy.io as sio\n",
    "data1 = sio.loadmat('pointsClass1Week6.mat')\n",
    "data2 = sio.loadmat('pointsClass2Week6.mat')\n",
    "\n",
    "from numpy import linalg as LA\n",
    "\n",
    "data1 = data1['pointsClass1Week6']\n",
    "data2 = data2['pointsClass2Week6']\n",
    "\n",
    "\n",
    "# put your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4.b \n",
    "\n",
    "As indicated above, we want to apply the network to the simple binary dataset that you loaded above. We want to build an architecture similar to the one shown above, except that, since we only consider a 2D dataset, we only need 2 inputs. we want our activation function to be all sigmoid. Start by defining the function sigmoid and the gradient of this function. Once you are done, check your derivative. Also make sure you can compute the entrywise sigmoid on any numpy array if the input to your function is a numpy array. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4.c\n",
    "\n",
    "Now that we have the sigmoid and its gradient, we will code the loss. As you might remember from previous labs, the MLPClassifier from scikit-learn optimizes the 'log-loss function' (a.k.a Binary cross entropy) which reads for a set of $N$ prototypes $x_i$ with binary $0/1$ targets, \n",
    "\n",
    "$$−\\frac{1}{N}\\sum_{i=1}^N (y_i \\log(p_{W}(x_i))+(1−y_i)\\log(1−p_{W}(x_i))) $$\n",
    "\n",
    "Here the probability $p_{W}(x_i)$ is the output of your network. Instead of minimizing this function directly, we will consider its $\\ell_2$ regularized version\n",
    "\n",
    "$$−\\frac{1}{N}\\sum_{i=1}^N (y_i \\log(p_{W}(x_i))+(1−y_i)\\log(1−p_{W}(x_i)))  + \\lambda \\sum_{j\\in\\text{weights}\\setminus \\text{bias}} W^2_j$$\n",
    "\n",
    "Where the $W_j$ encode the weights. Code that function for a given labeled dataset $X,t$ such as given above, a set of weights stored in the vector $W$ (you can use a list or a dictionnary if you want but ultimately, you will need to store them in a numpy vector for use with the optimization routine).  Note that one typically does not regularize the bias terms. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4.d. \n",
    "\n",
    "Towards Backpropagation. The example above is relatively simple so that backpropagation is not really needed. Compute the gradient of the Binary cross entropy loss with respect to the weights.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4.e. \n",
    "\n",
    "Combine all your previous steps into a single function and pass this function to the 'minimize' routine. To choose the initial value for the weights, a common heuristic is for the n^th layer to be initialized unformly at random in the interval $[-\\varepsilon_n, \\varepsilon_n]$ with $\\epsilon_n$ defined as $\\sqrt{\\frac{2}{\\text{size layer}_{\\ell-1} + \\text{size layer}_{\\ell} }}$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus 1 \n",
    "\n",
    "Change the activation to the Relu and reapeat the steps above. How do you compute the gradient in this case?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 5. \n",
    "\n",
    "Coding a One hidden layer neural network is good to warm up but to really understand backpropagation, we will not add a few hidden layers. Still relying on the code that you developed above and using backpropagation, train a depth 4 neural network with 10 neurons in each layer on the binary dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 5a. \n",
    "\n",
    "In this exercise, because of the multiple layers of the network, you will get to really code backpropagation. To do this, follow the steps below "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(see [Bishop Pattern Recognition and Machine Learning](http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf)) for more details.\n",
    "\n",
    "- Take any of the sample (Once you have completed one step of backprop, you should re-apply it to the next sample to get the gradient contribution for that next sample and so on). Forward propagate that sample through the network and compute all the activations $z = \\sigma(a_i)$ and input $a_i$ for each unit. \n",
    "\n",
    "- Evaluate the $\\delta_k = y_k(x_n) - t_{n,k}$ for all the output units.\n",
    "\n",
    "- Backpropagate the $\\delta_k$ using the formula\n",
    "\n",
    "$$\\delta_j = \\sigma'(a_j) \\sum_{k} w_{k,j} \\delta_k$$\n",
    "\n",
    "- Finally, compute the derivatives as \n",
    "\n",
    "$$\\frac{\\partial \\text{loss}}{\\partial w_{ij}}  = \\delta_j z_i$$\n",
    "\n",
    "Note that when coding the backpropagation algorithm, you don't need to account for the regularization part of the loss as you can just add the gradient of the regularization to the result of the backpropagation algorithm.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.optimize import minimize\n",
    "\n",
    "\n",
    "# put your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 6. \n",
    "\n",
    "Extending to multiple classes. Load the data using the lines given below, visualize it with scatter. How can you extend the Binary cross entropy function to the multiclass problem?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import scipy.io as sio\n",
    "data1 = sio.loadmat('xWeek6Ex3PointsClass1.mat')\n",
    "data2 = sio.loadmat('xWeek6Ex3PointsClass2.mat')\n",
    "data3 = sio.loadmat('xWeek6Ex3PointsClass3.mat')\n",
    "\n",
    "from numpy import linalg as LA\n",
    "\n",
    "data1 = data1['xWeek6Ex3PointsClass1']\n",
    "data2 = data2['xWeek6Ex3PointsClass2']\n",
    "data3 = data3['xWeek6Ex3PointsClass3']\n",
    "\n",
    "\n",
    "# put your code here\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}