L05/code/adaline-sgd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "STAT 453: Intro to Deep Learning and Generative Models (Spring 2021)  \n",
    "Instructor: Sebastian Raschka (sraschka@wisc.edu)  \n",
    "\n",
    "Course website: http://pages.stat.wisc.edu/~sraschka/teaching/stat453-ss2021/  \n",
    "GitHub repository: https://github.com/rasbt/stat453-deep-learning-ss21"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Author: Sebastian Raschka\n",
      "\n",
      "Python implementation: CPython\n",
      "Python version       : 3.9.2\n",
      "IPython version      : 7.20.0\n",
      "\n",
      "torch     : 1.9.0a0+d819a21\n",
      "pandas    : 1.2.2\n",
      "matplotlib: 3.3.4\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -v -p torch,pandas,matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ADALINE with Stochastic Gradient Descent (Minibatch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](figures/adaline-concept.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we are implementing ADALINE \"by hand\" without using PyTorch's autograd capabilities. In Lecture 06, we will use \"automatic differentiation\" (also known as \"autodiff\"; or autograd in PyTorch) to implement Adaline more compactly. (The reason why we don't use autodiff here is that it is useful to understand what's going on under the hood.) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load & Prepare a Toy Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x1</th>\n",
       "      <th>x2</th>\n",
       "      <th>x3</th>\n",
       "      <th>x4</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>6.3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.9</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>6.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>6.2</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.4</td>\n",
       "      <td>2.3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>5.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.8</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      x1   x2   x3   x4  y\n",
       "145  6.7  3.0  5.2  2.3  1\n",
       "146  6.3  2.5  5.0  1.9  1\n",
       "147  6.5  3.0  5.2  2.0  1\n",
       "148  6.2  3.4  5.4  2.3  1\n",
       "149  5.9  3.0  5.1  1.8  1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('./datasets/iris.data', index_col=None, header=None)\n",
    "df.columns = ['x1', 'x2', 'x3', 'x4', 'y']\n",
    "df = df.iloc[50:150]\n",
    "df['y'] = df['y'].apply(lambda x: 0 if x == 'Iris-versicolor' else 1)\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assign features and target\n",
    "\n",
    "X = torch.tensor(df[['x2', 'x3']].values, dtype=torch.float)\n",
    "y = torch.tensor(df['y'].values, dtype=torch.int)\n",
    "\n",
    "# Shuffling & train/test split\n",
    "\n",
    "torch.manual_seed(123)\n",
    "shuffle_idx = torch.randperm(y.size(0), dtype=torch.long)\n",
    "\n",
    "X, y = X[shuffle_idx], y[shuffle_idx]\n",
    "\n",
    "percent70 = int(shuffle_idx.size(0)*0.7)\n",
    "\n",
    "X_train, X_test = X[shuffle_idx[:percent70]], X[shuffle_idx[percent70:]]\n",
    "y_train, y_test = y[shuffle_idx[:percent70]], y[shuffle_idx[percent70:]]\n",
    "\n",
    "# Normalize (mean zero, unit variance)\n",
    "\n",
    "mu, sigma = X_train.mean(dim=0), X_train.std(dim=0)\n",
    "X_train = (X_train - mu) / sigma\n",
    "X_test = (X_test - mu) / sigma"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAWj0lEQVR4nO3df4xV5Z3H8c+XcQyTSKA4JAIXBdFQLTCYnYqEVDelzWjjr9KW1DZa2lq6TSt27dJCIEqptjY0bnZjU4qhNmRZFVM7ddXt1MpumjWLLRSXH0GKmFLuaLZ03FEbxjgMz/5xZ2BmmGHmznnuOc9zzvuVGL1nhnO/597rx+v5nu9zzDknAEC8xmVdAAAgGYIcACJHkANA5AhyAIgcQQ4AkTsviydtbGx0M2fOzOKpASBau3fv/otzbsrg7ZkE+cyZM7Vr164snhoAomVmR4fazqkVAIgcQQ4AkSPIASBymZwjH0p3d7fK5bLefffdrEsJxvjx41UqlVRfX591KQACFkyQl8tlTZgwQTNnzpSZZV1O5pxz6ujoULlc1qxZs7IuB0DAgjm18u677+rCCy8kxHuZmS688MLw/w9l73bpH+dK6ydV/r53e9YVAYUTzDdySYT4IMG/Hnu3S/+2Uuruqjx+61jlsSTNX5ZdXUDBBPONHBF6YcOZEO/T3VXZDiA1BPkI1q9frx/84Ac12ffu3bs1b948XXbZZVq5cqWiWxv+rXJ12wHUBEGeoa985SvavHmzDh8+rMOHD+uXv/xl1iVVZ2Kpuu0AaiLaIG/d067FD+7QrNXPavGDO9S6pz3xPrdu3ar58+erqalJt99++1k/f+SRR/TBD35QTU1N+sQnPqETJ05Ikp588knNnTtXTU1NuvbaayVJBw4c0NVXX60FCxZo/vz5Onz48IB9vfHGG3r77be1aNEimZnuuOMOtba2Jj6GVC25V6pvGLitvqGyHUBqgmp2jlbrnnateWqfurp7JEntnV1a89Q+SdKtV00f0z4PHDigBx54QC+++KIaGxv15ptvnvU7S5cu1Ze+9CVJ0rp167Rlyxbddddd2rBhg9ra2jR9+nR1dnZKkjZt2qS7775bn/3sZ/Xee++pp6dnwL7a29tVKp355loqldTenvw/Rqnqa2i+sKFyOmViqRLiNDqBVEUZ5BvbDp0O8T5d3T3a2HZozEG+Y8cOffKTn1RjY6MkafLkyWf9zv79+7Vu3Tp1dnbqr3/9q1paWiRJixcv1vLly7Vs2TItXbpUkrRo0SI98MADKpfLWrp0qS6//PIB+xrqfHjwV6kMZf4yghvIWJSnVl7v7Kpq+2g450YM0uXLl+vhhx/Wvn37dN99952+xnvTpk26//77dezYMS1YsEAdHR36zGc+o6effloNDQ1qaWnRjh07BuyrVCqpXD7TFCyXy5o2bdqY6wdQXFEG+bRJDVVtH40lS5Zo+/bt6ujokKQhT6288847mjp1qrq7u7Vt27bT248cOaKFCxdqw4YNamxs1LFjx/Taa6/p0ksv1cqVK3XzzTdr7969A/Y1depUTZgwQTt37pRzTlu3btUtt9wy5voBFFeUQb6qZY4a6usGbGuor9Oqljlj3ucHPvABrV27Vtddd52ampp0zz33nPU73/nOd7Rw4UJ99KMf1fvf//4z9axapXnz5mnu3Lm69tpr1dTUpCeeeEJz587VggUL9Morr+iOO+44a38/+tGPdOedd+qyyy7T7NmzdcMNN4y5/kJjuhQFZ1lcu9zc3OwG31ji4MGDuuKKK0a9j9Y97drYdkivd3Zp2qQGrWqZM+bz4yGr9nUpnMHTpVLlypmb/plz98gdM9vtnGsevD3KZqdUuTolj8GNKp1rupQgR0FEeWoFOI3pUoAgR+SYLgUIckSO6VKAIEfk5i+rNDYnzpBklb/T6ETBRNvsBE5juhQFxzfyEdRyGdu1a9dqxowZuuCCC2qyfwDFQJBn6KabbtJvf/vbrMsAELl4g7wG03xpLmMrSddcc42mTp2auG4AxZb4HLmZzZC0VdJFkk5J2uyc+6ek+z2nGtwrMu1lbAHAFx/fyE9K+oZz7gpJ10j6qpld6WG/w6vBvSJHu4zthz70Ic2bN0/btm3TgQMHJJ1ZxvaRRx45HdiLFi3Sd7/7XX3/+9/X0aNH1dAw9gW9AOBcEge5c+4N59zve//5HUkHJdV2dr4G03xpL2MLAL54PUduZjMlXSXppSF+tsLMdpnZruPHjyd7ohpM86W9jC0A+OItyM3sAkk/k/R159zbg3/unNvsnGt2zjVPmTIl2ZPVYJovi2Vsv/nNb6pUKunEiRMqlUpav379mOsHUFxelrE1s3pJz0hqc849NNLv+1jGVnu3F+JekSxjC+SEh8yq2TK2VjmxvEXSwdGEuDdM8wGIRQ2utOvPx6mVxZJul/RhM3u596+PedgvAORDDa606y/xN3Ln3H9J8nL799FcOVIkWdy9CUAN1Hjd/GAmO8ePH6+Ojg7Cq5dzTh0dHRo/fnzWpQBIqsbr5gez+mGpVFK5XFbiSxNzZPz48SqVuEFCKgrSPEdGltw79L1lPa2bH0yQ19fXa9asWVmXgSKqcSMKOP05qtGXhWCCHMgMN3BGGmp4pV0w58iBzHADZ0SOIAe4gTMiR5AD3MAZkSPIAW7gjMjR7AQklnxA1PhGDgCRI8gBIHIEOeBLDW4IDg8K8L5wjhzwgenQMBXkfeEbOeBDjZcpxRgV5H0hyAEfmA4NU0HeF4Ic8IHp0DAV5H0hyAEfQpoOLUBzb9RCel9qiGYn4EONlykdtYI090YtlPelxiyLO/I0Nze7Xbt2pf68QO7949xKeA82cYb09/vTrwdemdlu51zz4O2cWgHypCDNPQxEkAN5UpDmHgYiyIE8KUhzDwMR5ECesCRvIXHVCpA3LMlbOHwjB4DIEeQAEDmCvMiYAARygXPkRcUEIJAbfCMvqoIs7wkUAUFeVEwAArlBkBcVE4BAbngJcjP7iZn92cxYlScWTADml48m9jP3SN+eLK2fWPn7M/d4LxP++PpG/lNJ13vaF9LABGA+9TWx3zomyZ1pYlcT5s/cI+3aIrmeymPXU3lMmAfL2zK2ZjZT0jPOubkj/S7L2AI14mMZ229PPhPi/VmddN+byepDIpkvY2tmK8xsl5ntOn78eFpPCxSLjyb2UCF+ru3IXGpB7pzb7Jxrds41T5kyJa2nBYrFRxPb6qrbjsxx1QqSCWE61EdjLoTj8FGHjyb23yyvbvtwQnlNC4DJToxdCNOhfY25Pn2NOUm68aHR7SOE4/BVh497VPa9brt/Wnk9ra4S4qN9PaVwXtOC8NLsNLPHJP2tpEZJ/yvpPufcluF+n2ZnToRwf0gfjbkQjiOkOnzI07EEZLhmp5dv5M6523zsB5EJYTrUR2MuhOMIqQ4f8nQsEeAcOcYuhOlQH425EI4jpDp8yNOxRIAgx9iFMB3qozHn6ziSNl2X3CvVnT9wW935cU7bhvDZKBCCHGMXwnTojQ9JzV888w3c6iqPq2nM+TgOX9OQg3tWngb2UhfCZ6NAvE12VoNmJ3InT01XBCvzyU4g1/LUdEV0CHLAhzw1XREdghzwwVfTdVz9wG3j6uNtEDLZmRomOwEffExDSpLZuR/HgsnOVNHsBEKRp2Znno4lIDQ7gdDlqdmZp2OJAEEOhCJPzc48HUsECHJkj6ZYha9pyBBeTyY7U0WzE9miKXaGjyVoQ3k9fRwLRo1mJ7JFU8wvXs9co9mJMNEU84vXs5AIcmSLpphfvJ6FRJAjW3lqitFkREZodiJbeWmK0WREhmh2Aj7QZEQKaHYCtUSTERkiyAEfaDIiQwQ5Mnfk0S/r5Pr3yd03USfXv09HHv1ydTugyYiCI8iRqSOPflmX/vFxnadTMpPO0yld+sfHRx/mfU3Gt45JcmeajGmHOfeoRIZodiJTJ9e/T+fp1NnbNU7nrf+/kXdAkxEFQrMTQapzZ4f4ubafhSYjQJAjWz029EdwuO1nockIEOTI1tFLlmnw2T3nKttHhSYjwGRn1fZuz8/UXADHMvvzP9aRR6VLjm5XnTulHhunozOXafbnfzy6HQQ0ydi6p10b2w7p9c4uTZvUoFUtc3TrVdNTrwPFQ7OzGoPHsKXKt78Yr07I07EEoHVPu9Y8tU9d3T2ntzXU1+l7S+cR5vCGZqcPL2wYGHxS5fELG7KpJ4k8HUsANrYdGhDiktTV3aONbYcyqghF4iXIzex6MztkZq+a2Wof+wxSnq6QyNOxBOD1zq6qtgM+JQ5yM6uT9ENJN0i6UtJtZnZl0v0GKU9XSOTpWAIwbVJDVdsBn3x8I79a0qvOudecc+9JelzSLR72G548XSGx5F6drBs/YNPJuvFxHksAI/qrWuaoob5uwLaG+jqtaplT1X5a97Rr8YM7NGv1s1r84A617mn3WSZyykeQT5fUf7Su3LttADNbYWa7zGzX8ePHPTxtBnI0ht3as1iru+9U+VSjTjlT+VSjVnffqdaexVmXVp1ARvRvvWq6vrd0nqZPapBJmj6poepGZ1/DtL2zS05Se2eX1jy1jzDHiBJftWJmn5LU4py7s/fx7ZKuds7dNdyfifaqlRxZ/OAOtQ9x/nb6pAa9uPrDGVQ0Rjka0c/Ne4KaqeVVK2VJM/o9Lkl63cN+UUO5ac7lqGmbm/cEqfMR5L+TdLmZzTKz8yV9WtLTHvaLGspNcy5HTdvcvCdIXeIgd86dlPQ1SW2SDkra7pw7kHS/qC1fzbnMLblXqjt/4La687Np2iZsutIwxVh5GdF3zj0n6Tkf+0I6+ppwuRgpH2qxlrR5uPmyj/dk8IRpX8O0//6RP4zoI26hNDsDqYOGab4xoo98CqXZGUgdNEyLiSBH3EJpdgZSBw3TYiLIEbdQJlQ9Tf0mbVSuapmj+nE2sIxxRsM05whyRC2YCVUPU7/eJjtthMdp1YHU0OxE1PLU3PNxLKHsA7VBsxO5lKfmno9jCWUfSBdBjqjlqbnn41hC2QfSRZAjaqta5qi+blBzry7O5p6Pyc5Q9oF0cfNlxG9wm6fKtk8o05A+JjtD2QfSRbMTUaO5hyKh2YlcorkHEOSIHM09gHPkVWvd056fc4d7t0svbKisBzKxVJlCjOy2data5mjVk/+j7lNnThFWO8noYx+StK51nx576Zh6nFOdmW5bOEP33zqvqn34EMpnNJQ6ioAgr0IoTTEvPCy7GoyEk4w+9rGudZ/+ZeefTj/uce704zTDPJTPaCh1FAWnVqqwse3Q6Q9mn67uHm1sO5RRRQm8sOFMiPfp7qpsj8jGtkPq7hnYsO/ucVW9Jz728dhLQyxhe47ttRLKZzSUOoqCIK9CrppigSy7mlQozc6eYa7+Gm57rYTyGQ2ljqIgyKuQq6ZYIMuuJhVKs7POhj4XM9z2WgnlMxpKHUVBkFchV/dUDGTZ1aRCmWS8beGMqrbXSihTmaHUURQ0O6uQq3sq9jU0E1y1EsKxhDLJ2HzJZP3rzj/pVL9t43q3pymUqcxQ6igKJjtTlqcpwjwdS1K8FkgDk52ByFMTKE/HkhSvBbJEkKcsT02gPB1LUrwWyBJBnrI8NYHydCzrWvdp9prnNHP1s5q95jmta91X1Z/P02uB+NDsTFmemkB5ORYfU5l5eS0QJ5qdKLzZa54bcnCnzkxHvvexDCoChkazExhGKFOZwFgR5Ci8UKYygbEiyFF4vqYys55yRXER5Ci85ksmq27cwG/fdeOsqqnMvinX9s4uOZ2ZciXMkQaCHIW3se2Qek4NPB/ec6r6pXBZthVZSRTkZvYpMztgZqfM7KxOKhCDUJbCBcYq6Tfy/ZKWSvqNh1qATISyFC4wVomC3Dl30DnH/zsikaRNwqR/3tdSuPV1A8+z19dVf99PYCxSm+w0sxWSVkjSxRdfnNbTInBJl8L1sZSut6nMwZedcxk6UjLiZKeZ/VrSRUP8aK1z7he9v/Ofkv7BOTeqcU0mO9En6fKvoSwfG0odyLfhJjtH/EbunPtIbUoCkjcJQ2kyhlIHionLD5GppE3CUJqModSBYkp6+eHHzawsaZGkZ82szU9Z4fIxvRfKBGAIx5K00RjK8rGh1IFiStTsdM79XNLPPdUSPB+NtRDuc+mrjhAajaEsHxtKHSgmlrGtgo+GVihNsTwdC1AULGPrQZ4mAPN0LEDREeRVyNMEYJ6OBSg6grwKviYAQ2iK5elYgKLjnp1V8NHQCqUplqdjAYqOZicARIJmJwDkFEEOAJEjyAEgcgQ5AESOIAeAyBHkABA5ghwAIsdAUKRa97QziANAEkEepVCWwgUQBk6tRGhj26HTId6nq7tHG9sOZVQRgCwR5BFi+VgA/RHkEWL5WAD9EeQRYvlYAP3R7IwQy8cC6I8gj9StV00nuAFI4tQKAESPIAeAyBHkABC5aM6RM5IOAEOLIsgZSQeA4UVxaoWRdAAYXhRBzkg6AAwviiBnJB0AhhdFkDOSDgDDi6LZyUg6AAwvUZCb2UZJN0l6T9IRSZ93znV6qOssjKQDwNCSnlp5XtJc59x8SX+QtCZ5SQCAaiQKcufcr5xzJ3sf7pRUSl4SAKAaPs+Rf0HSE8P90MxWSFohSRdffLHHpy0mJl0B9BkxyM3s15IuGuJHa51zv+j9nbWSTkraNtx+nHObJW2WpObmZjemaiGJSVcAA40Y5M65j5zr52b2OUk3SlrinCOgU3CuSVeCHCiepFetXC/pW5Kuc86d8FMSRsKkK4D+kl618rCkCZKeN7OXzWyTh5owAiZdAfSX9KqVy5xzM5xzC3r/+jtfhWF4TLoC6C+KyU4MxKQrgP4I8kgx6QqgTxSLZgEAhkeQA0DkCHIAiBxBDgCRI8gBIHIEOQBEjiAHgMhxHXmBsRQukA8EeUGxFC6QH5xaKahzLYULIC4EeUGxFC6QHwR5QbEULpAfBHlBsRQukB80OwuKpXCB/CDIC4ylcIF84NQKAESOIAeAyBHkABA5ghwAIkeQA0DkzDmX/pOaHZd0NPUnHptGSX/JuoiUcKz5xLHmxyXOuSmDN2YS5DExs13Oueas60gDx5pPHGv+cWoFACJHkANA5AjykW3OuoAUcaz5xLHmHOfIASByfCMHgMgR5AAQOYJ8FMxso5m9YmZ7zeznZjYp65pqxcw+ZWYHzOyUmeXyMi4zu97MDpnZq2a2Out6asXMfmJmfzaz/VnXUmtmNsPM/sPMDvZ+fu/OuqY0EeSj87ykuc65+ZL+IGlNxvXU0n5JSyX9JutCasHM6iT9UNINkq6UdJuZXZltVTXzU0nXZ11ESk5K+oZz7gpJ10j6ao7f17MQ5KPgnPuVc+5k78OdkkpZ1lNLzrmDzrk834H5akmvOudec869J+lxSbdkXFNNOOd+I+nNrOtIg3PuDefc73v/+R1JByUVZrF9grx6X5D071kXgTGbLulYv8dlFehf+CIws5mSrpL0UsalpIY7BPUys19LumiIH611zv2i93fWqvK/cNvSrM230RxrjtkQ27gGNyfM7AJJP5P0defc21nXkxaCvJdz7iPn+rmZfU7SjZKWuMgvvh/pWHOuLGlGv8clSa9nVAs8MrN6VUJ8m3PuqazrSROnVkbBzK6X9C1JNzvnTmRdDxL5naTLzWyWmZ0v6dOSns64JiRkZiZpi6SDzrmHsq4nbQT56DwsaYKk583sZTPblHVBtWJmHzezsqRFkp41s7asa/Kpt2n9NUltqjTEtjvnDmRbVW2Y2WOS/lvSHDMrm9kXs66phhZLul3Sh3v/HX3ZzD6WdVFpYUQfACLHN3IAiBxBDgCRI8gBIHIEOQBEjiAHgMgR5AAQOYIcACL3/xSB0KCk21RJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(X_train[y_train == 0, 0], X_train[y_train == 0, 1], label='class 0')\n",
    "plt.scatter(X_train[y_train == 1, 0], X_train[y_train == 1, 1], label='class 1')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAASiElEQVR4nO3dfWxVdZ7H8c93O53QRCLRkghcXFAMPkCL2QoSoiaDBp2IOtUho0aXrA+J2REnzuJqIErwYTQYJ2ucSDBODFnWsWa0OqtrV2V3zJpRp4gLdJFFnbjcajKdGnwINdb63T9ueSot3Idzen6/e96vhOD59eac7zmBD8ff+d7fMXcXACBef5V1AQCA2hDkABA5ghwAIkeQA0DkCHIAiNz3sjhoc3Ozz5gxI4tDA0C0tmzZ8hd3nzxyPJMgnzFjhrq7u7M4NABEy8w+Hm2cqRUAiBxBDgCRI8gBIHKZzJGPZnBwUMViUV9//XXWpQRjwoQJKhQKamxszLoUAAELJsiLxaImTpyoGTNmyMyyLidz7q7+/n4Vi0XNnDkz63IABCyYqZWvv/5aJ554IiE+zMx04okn8n8oQL3Y1iH9co60ZlLp920die06mDtySYT4CFwPoE5s65B+t0IaHChtf76ntC1JLctq3n0wd+QAULdeX3swxPcbHCiNJ4AgP4Y1a9bo4YcfTmXfW7Zs0dy5czVr1iytWLFCrA0P1KnPi5WNV4ggz9Att9yiDRs2aPfu3dq9e7deeeWVrEsCkIbjC5WNVyjaIO/c2qtFD27WzDtf0qIHN6tza2/N+9y4caNaWlrU2tqq66677oifP/HEEzrnnHPU2tqqK6+8Uvv27ZMkPfvss5ozZ45aW1t1/vnnS5J6eno0f/58zZs3Ty0tLdq9e/dh+/r000/1xRdfaOHChTIzXX/99ers7Kz5HAAEaPHdUmPT4WONTaXxBAT1sLNcnVt7dddz2zUwOCRJ6t07oLue2y5JuuLsaVXts6enR/fff7/efPNNNTc367PPPjviM+3t7brpppskSatXr9aTTz6pW2+9VWvXrlVXV5emTZumvXv3SpLWr1+v2267Tddee62++eYbDQ0NHbav3t5eFQoH/zUuFArq7a39HyOMsK2jNA/5ebF097P47kQeLgEV2f9nLqU/i1EG+bquXQdCfL+BwSGt69pVdZBv3rxZV111lZqbmyVJJ5xwwhGf2bFjh1avXq29e/fqq6++0pIlSyRJixYt0vLly7Vs2TK1t7dLkhYuXKj7779fxWJR7e3tOu200w7b12jz4XSpJCzlTgGgIi3LUvtzF+XUyid7ByoaL4e7HzNIly9frscee0zbt2/XPffcc6DHe/369brvvvu0Z88ezZs3T/39/brmmmv04osvqqmpSUuWLNHmzZsP21ehUFCxePBBR7FY1NSpU6uuH6NIuVMACEWUQT51UlNF4+VYvHixOjo61N/fL0mjTq18+eWXmjJligYHB7Vp06YD4x9++KEWLFigtWvXqrm5WXv27NFHH32kU045RStWrNBll12mbdu2HbavKVOmaOLEiXrrrbfk7tq4caMuv/zyquvHKFLuFABCEWWQr1wyW02NDYeNNTU2aOWS2VXv86yzztKqVat0wQUXqLW1VbfffvsRn7n33nu1YMECXXTRRTr99NMP1rNypebOnas5c+bo/PPPV2trq5555hnNmTNH8+bN0/vvv6/rr7/+iP09/vjjuvHGGzVr1iydeuqpuuSSS6quH6NIuVMACIVl0bvc1tbmI18ssXPnTp1xxhll76Nza6/Wde3SJ3sHNHVSk1YumV31/HjIKr0uOMTIOXKp1Cmw9FHmyBElM9vi7m0jx2t+2Glm0yVtlHSSpO8kbXD3f6p1v8dyxdnT6jK4kaCUOwWAUCTRtfKtpJ+7+7tmNlHSFjN71d3/J4F9A7VJsVMACEXNc+Tu/qm7vzv8319K2imJW2UAGCeJPuw0sxmSzpb0dpL7BQCMLbEgN7PjJP1W0s/c/YtRfn6zmXWbWXdfX19ShwWA3EskyM2sUaUQ3+Tuz432GXff4O5t7t42efLkJA4LAFACQW6lr0M+KWmnuz9Se0lhSXMZ21WrVmn69Ok67rjjUtk/gHxI4o58kaTrJP3AzN4b/vXDBPZb95YuXap33nkn6zIARC6JrpX/cndz9xZ3nzf86+UkijuqFN5/N57L2ErSueeeqylTptRcN4B8i3L1wzRWtRvvZWwBIClRrrWSxqp25S5je95552nu3LnatGmTenp6JB1cxvaJJ544ENgLFy7UAw88oIceekgff/yxmpqqX9ALAI4mziBPYVW78V7GFgCSEmeQp7Cq3XgvYwsASYkzyFN4/10Wy9jecccdKhQK2rdvnwqFgtasWVN1/QDyK9plbPPyLkaWsQWwX2rL2GaGVe0AQFKsUysAgAOCCvIspnlCxvUAUI5ggnzChAnq7+8nvIa5u/r7+zVhwoSsSwEQuGDmyAuFgorFolji9qAJEyaoUOBFwQCOLpggb2xs1MyZM7MuAwCiE8zUCgCgOgQ5AESOIEdtUlhOGEBlgpkjR4RSWE4YQOW4I0f1UlhOGEDlCHJUL4XlhAFUjiBH9VJYThhA5QhyVC+F5YQBVI4gR/ValklLH5WOny7JSr8vfZQHnVmjkyh36FpBbVhOOCx0EuUSd+RAPaGTKJcIcqCe0EmUSwQ5UE/oJMolghyoJ3QS5RJBjvoWQwdHkjXSSZRLdK2gfsXQwZFGjXQS5Q535KhfMXRwxFAjgkeQo37F0MERQ40IHkGO+hVDB0cMNSJ4BDnqVwwdHDHUiOAR5KhfLcukwvzDxwrzw3oQSJcJEkDXCurXv94u/en3h4/96fel8Usfyaam0dBlghpxR476teWpysaBSBHkqF8+VNk4ECmCHPXLGiobByKVSJCb2a/N7M9mtiOJ/QGJ+JvllY0DkUrqjvwpSRcntC8gGZc+IrXdcPAO3BpK2yE96AQSkEjXiru/YWYzktgXkKhLHyG4UffGbY7czG42s24z6+7r6xuvwwJA3Ru3IHf3De7e5u5tkydPHq/DAkDdo2sFACJHkANA5JJqP3xa0h8kzTazopndkMR+AQDHllTXytVJ7AcAUDmmVgAgcgQ5AESOIAeAyBHkABA5ghwAIkeQA0DkCHIAiBxBDgCRI8gBIHIEOQBEjiAHgMgR5AAQOYIcACJHkANA5AhyAIgcQQ4AkSPIASByBDkARI4gB4DIEeQAEDmCHPVtW4f0yznSmkml37d1ZF0RkLjvZV0AkJptHdLvVkiDA6Xtz/eUtiWpZVl2dQEJ444c9ev1tQdDfL/BgdI4UEcIctSvz4uVjQORIshRv44vVDYORIogR/1afLfU2HT4WGNTaRyoIwQ5ahNyV0jLMqn1GskaStvWUNqu5UFnGucb8jVEFOhaQfVC7wrZ1iH9979IPlTa9qHS9snnVldfGucb+jVEFLgjR/VC7wpJur40zjf0a4goEOSoXuhdIUnXl8b5hn4NEQWCHNULvSsk6frSON/QryGiQJCjeqF3hSRdXxrnG/o1RBQI8iTlrfugZZm09FHp+OmSrPT70kfDeUiXdH1pnG/o1xBRMHcf94O2tbV5d3f3uB83VSO7D6TSnRV/KQEkxMy2uHvbyHHuyJNC9wGAjBDkSaH7AEBGEglyM7vYzHaZ2QdmdmcS+4wO3QcAMlJzkJtZg6RfSbpE0pmSrjazM2vdb3ToPgCQkSTuyOdL+sDdP3L3byT9RtLlCew3LjF0H+Stq0bK5zkjd5JYa2WapD2HbBclLUhgv/FpWRZWcB8qj2t65PGckUtJ3JHbKGNH9DSa2c1m1m1m3X19fQkcFhXJY1dNHs8ZuZREkBclTT9kuyDpk5EfcvcN7t7m7m2TJ09O4LCoSB67avJ4zsilJIL8j5JOM7OZZvZ9ST+R9GIC+0WS8thVk8dzRi7VHOTu/q2kn0rqkrRTUoe799S6XyQsj101eTxn5FIiL5Zw95clvZzEvpCS/Q/3Xl9bmlo4vlAKtBof+nVu7dW6rl36ZO+Apk5q0sols3XF2dMSKDgBKZ0zEBrWWkHVOrf26q7ntmtgcOjAWFNjg37RPjecMAfqCGutIHHrunYdFuKSNDA4pHVduzKqCMgnghxV+2TvQEXjANJBkKNqUyc1VTQOIB0EOaq2cslsNTU2HDbW1NiglUtmZ1QRkE+JdK0gn/Y/0EyyayXoLhggUAQ5anLF2dMSC9qRXTC9ewd013PbDxwHwOiYWkEw6IIBqkOQIxh0wQDVIcgRDLpggOoQ5AhGGl0wnVt7tejBzZp550ta9OBmdW7trbVMIDg87EQwku6C4eEp8oIgR1CS7II52sNTghz1hKkV1C0eniIvCHLULR6eIi8IctQtlhBAXjBHjrqVxhICQIgIctS1JB+eAqFiagUAIkeQA0DkCHIAiBxBDgCR42EnUAFefIEQEeRAmVi7BaFiagUoEy++QKgIcqBMrN2CUBHkQJlYuwWhIsiBMrF2C0LFw06gTKzdglAR5EAFWLsFIWJqBQAiR5ADQOQIcgCIHEEOAJHjYWeO5HGdkDyeM/KHIM+JPK4TksdzRj4xtZITeVwnJI/njHyqKcjN7Mdm1mNm35lZW1JFIXl5XCckj+eMfKr1jnyHpHZJbyRQC1KUx3VC8njOyKeagtzdd7o7/58agTyuE5LHc0Y+jdvDTjO7WdLNknTyySeP12ExLI/rhOTxnJFP5u5H/4DZa5JOGuVHq9z9heHP/Kekf3D37nIO2tbW5t3dZX0UADDMzLa4+xHPI495R+7uF6ZTEgAgCbQfAkDkam0//JGZFSUtlPSSmXUlUxYAoFw1Pex09+clPZ9QLYgQX4EHssdX9FE1vgIPhIE5clSNr8ADYSDIUTW+Ag+EgSBH1fgKPBAGghxV4yvwQBh42Jmg0Ds4Vndu19Nv79GQuxrMdPWC6brvirlV74+vwANhIMgTEnoHx+rO7frnt/7vwPaQ+4HtWsM8hPMD8oyplYSE3sHx9Nt7KhoHEA+CPCGhd3AMjbE42ljjAOJBkCck9A6OBrOKxgHEgyBPSOgdHFcvmF7ROIB45PZhZ9IdJqF3cOx/oJlk1wqAMBzzxRJpyPrFEiM7TKTS3fMv2ucGE7wAMNJYL5bI5dRK6B0mAFCJXAZ56B0mAFCJXAZ56B0mAFCJXAZ56B0mAFCJXHathN5hAgCVyGWQS6wRAqB+5HJqBQDqCUEOAJEjyAEgcrmdI0eYQn85BxAighzBCP3lHEComFpBMFg6AagOQY5gsHQCUB2CHMFg6QSgOgQ5gsHSCUB1eNgZsLx1cLB0AlAdgjxQee3gYOkEoHJMrQSKDg4A5SLIA0UHB4ByEeSBooMDQLkI8kDRwQGgXDzsDBQdHADKRZAHjA4OAOWoaWrFzNaZ2ftmts3MnjezSQnVBQAoU61z5K9KmuPuLZL+V9JdtZcEAKhETUHu7v/u7t8Ob74lqVB7SQCASiTZtfJ3kv5trB+a2c1m1m1m3X19fQkeFgDy7ZgPO83sNUknjfKjVe7+wvBnVkn6VtKmsfbj7hskbZCktrY2r7TQvK07AgDlOmaQu/uFR/u5mf2tpEslLXb3igO6HHlddwQAylFr18rFkv5R0mXuvi+Zko7EuiMAMLZa58gfkzRR0qtm9p6ZrU+gpiOw7ggAjK2mLwS5+6ykCjmaqZOa1DtKaLPuCABEstYK644AwNii+Io+644AwNiiCHKJdUcAYCxRTK0AAMZGkANA5AhyAIgcQQ4AkSPIASByltLyKEc/qFmfpI/H/cDZaJb0l6yLCBDXZWxcm7Hl/dr8tbtPHjmYSZDniZl1u3tb1nWEhusyNq7N2Lg2o2NqBQAiR5ADQOQI8vRtyLqAQHFdxsa1GRvXZhTMkQNA5LgjB4DIEeQAEDmCPGVm9mMz6zGz78yMtimVXhFoZrvM7AMzuzPrekJhZr82sz+b2Y6sawmNmU03s/8ws53Df59uy7qmkBDk6dshqV3SG1kXEgIza5D0K0mXSDpT0tVmdma2VQXjKUkXZ11EoL6V9HN3P0PSuZL+nj83BxHkKXP3ne7OW6IPmi/pA3f/yN2/kfQbSZdnXFMQ3P0NSZ9lXUeI3P1Td393+L+/lLRTEi8oGEaQY7xNk7TnkO2i+AuJCpjZDElnS3o741KCEc0bgkJmZq9JOmmUH61y9xfGu57A2Shj9MCiLGZ2nKTfSvqZu3+RdT2hIMgT4O4XZl1DRIqSph+yXZD0SUa1ICJm1qhSiG9y9+eyrickTK1gvP1R0mlmNtPMvi/pJ5JezLgmBM7MTNKTkna6+yNZ1xMagjxlZvYjMytKWijpJTPryrqmLLn7t5J+KqlLpQdWHe7ek21VYTCzpyX9QdJsMyua2Q1Z1xSQRZKuk/QDM3tv+NcPsy4qFHxFHwAixx05AESOIAeAyBHkABA5ghwAIkeQA0DkCHIAiBxBDgCR+3/ipQDT/WGwwgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(X_test[y_test == 0, 0], X_test[y_test == 0, 1], label='class 0')\n",
    "plt.scatter(X_test[y_test == 1, 0], X_test[y_test == 1, 1], label='class 1')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implement ADALINE Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Adaline1():\n",
    "    def __init__(self, num_features):\n",
    "        self.num_features = num_features\n",
    "        self.weights = torch.zeros(num_features, 1, \n",
    "                                   dtype=torch.float)\n",
    "        self.bias = torch.zeros(1, dtype=torch.float)\n",
    "\n",
    "    def forward(self, x):\n",
    "        netinputs = torch.add(torch.mm(x, self.weights), self.bias)\n",
    "        activations = netinputs\n",
    "        return activations.view(-1)\n",
    "        \n",
    "    def backward(self, x, yhat, y):  \n",
    "        \n",
    "        grad_loss_yhat = 2*(yhat - y)\n",
    "        \n",
    "        grad_yhat_weights = x\n",
    "        grad_yhat_bias = 1.\n",
    "        \n",
    "        # Chain rule: inner times outer\n",
    "        grad_loss_weights = torch.mm(grad_yhat_weights.t(),\n",
    "                                         grad_loss_yhat.view(-1, 1)) / y.size(0)\n",
    "\n",
    "        grad_loss_bias = torch.sum(grad_yhat_bias*grad_loss_yhat) / y.size(0)\n",
    "        \n",
    "        # return negative gradient\n",
    "        return (-1)*grad_loss_weights, (-1)*grad_loss_bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Training and Evaluation Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "####################################################\n",
    "##### Training and evaluation wrappers\n",
    "###################################################\n",
    "\n",
    "def loss(yhat, y):\n",
    "    return torch.mean((yhat - y)**2)\n",
    "\n",
    "\n",
    "def train(model, x, y, num_epochs,\n",
    "          learning_rate=0.01, seed=123, minibatch_size=10):\n",
    "    cost = []\n",
    "    \n",
    "    torch.manual_seed(seed)\n",
    "    for e in range(num_epochs):\n",
    "        \n",
    "        #### Shuffle epoch\n",
    "        shuffle_idx = torch.randperm(y.size(0), dtype=torch.long)\n",
    "        minibatches = torch.split(shuffle_idx, minibatch_size)\n",
    "        \n",
    "        for minibatch_idx in minibatches:\n",
    "\n",
    "            #### Compute outputs ####\n",
    "            yhat = model.forward(x[minibatch_idx])\n",
    "\n",
    "            #### Compute gradients ####\n",
    "            negative_grad_w, negative_grad_b = \\\n",
    "                model.backward(x[minibatch_idx], yhat, y[minibatch_idx])\n",
    "\n",
    "            #### Update weights ####\n",
    "            model.weights += learning_rate * negative_grad_w\n",
    "            model.bias += learning_rate * negative_grad_b\n",
    "            \n",
    "            #### Logging ####\n",
    "            minibatch_loss = loss(yhat, y[minibatch_idx])\n",
    "            print('    Minibatch MSE: %.3f' % minibatch_loss)\n",
    "\n",
    "        #### Logging ####\n",
    "        yhat = model.forward(x)\n",
    "        curr_loss = loss(yhat, y)\n",
    "        print('Epoch: %03d' % (e+1), end=\"\")\n",
    "        print(' | MSE: %.5f' % curr_loss)\n",
    "        cost.append(curr_loss)\n",
    "\n",
    "    return cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Minibatch MSE: 0.500\n",
      "    Minibatch MSE: 0.341\n",
      "    Minibatch MSE: 0.220\n",
      "    Minibatch MSE: 0.245\n",
      "    Minibatch MSE: 0.157\n",
      "    Minibatch MSE: 0.133\n",
      "    Minibatch MSE: 0.144\n",
      "Epoch: 001 | MSE: 0.12142\n",
      "    Minibatch MSE: 0.107\n",
      "    Minibatch MSE: 0.147\n",
      "    Minibatch MSE: 0.064\n",
      "    Minibatch MSE: 0.079\n",
      "    Minibatch MSE: 0.185\n",
      "    Minibatch MSE: 0.063\n",
      "    Minibatch MSE: 0.135\n",
      "Epoch: 002 | MSE: 0.09932\n",
      "    Minibatch MSE: 0.093\n",
      "    Minibatch MSE: 0.064\n",
      "    Minibatch MSE: 0.128\n",
      "    Minibatch MSE: 0.099\n",
      "    Minibatch MSE: 0.079\n",
      "    Minibatch MSE: 0.157\n",
      "    Minibatch MSE: 0.080\n",
      "Epoch: 003 | MSE: 0.09693\n",
      "    Minibatch MSE: 0.131\n",
      "    Minibatch MSE: 0.146\n",
      "    Minibatch MSE: 0.050\n",
      "    Minibatch MSE: 0.095\n",
      "    Minibatch MSE: 0.106\n",
      "    Minibatch MSE: 0.072\n",
      "    Minibatch MSE: 0.102\n",
      "Epoch: 004 | MSE: 0.09658\n",
      "    Minibatch MSE: 0.107\n",
      "    Minibatch MSE: 0.204\n",
      "    Minibatch MSE: 0.149\n",
      "    Minibatch MSE: 0.054\n",
      "    Minibatch MSE: 0.060\n",
      "    Minibatch MSE: 0.056\n",
      "    Minibatch MSE: 0.069\n",
      "Epoch: 005 | MSE: 0.09657\n",
      "    Minibatch MSE: 0.068\n",
      "    Minibatch MSE: 0.111\n",
      "    Minibatch MSE: 0.092\n",
      "    Minibatch MSE: 0.115\n",
      "    Minibatch MSE: 0.157\n",
      "    Minibatch MSE: 0.074\n",
      "    Minibatch MSE: 0.087\n",
      "Epoch: 006 | MSE: 0.09650\n",
      "    Minibatch MSE: 0.057\n",
      "    Minibatch MSE: 0.070\n",
      "    Minibatch MSE: 0.133\n",
      "    Minibatch MSE: 0.127\n",
      "    Minibatch MSE: 0.062\n",
      "    Minibatch MSE: 0.153\n",
      "    Minibatch MSE: 0.103\n",
      "Epoch: 007 | MSE: 0.09683\n",
      "    Minibatch MSE: 0.102\n",
      "    Minibatch MSE: 0.110\n",
      "    Minibatch MSE: 0.101\n",
      "    Minibatch MSE: 0.065\n",
      "    Minibatch MSE: 0.126\n",
      "    Minibatch MSE: 0.124\n",
      "    Minibatch MSE: 0.076\n",
      "Epoch: 008 | MSE: 0.09648\n",
      "    Minibatch MSE: 0.120\n",
      "    Minibatch MSE: 0.056\n",
      "    Minibatch MSE: 0.100\n",
      "    Minibatch MSE: 0.102\n",
      "    Minibatch MSE: 0.106\n",
      "    Minibatch MSE: 0.075\n",
      "    Minibatch MSE: 0.144\n",
      "Epoch: 009 | MSE: 0.09740\n",
      "    Minibatch MSE: 0.073\n",
      "    Minibatch MSE: 0.071\n",
      "    Minibatch MSE: 0.084\n",
      "    Minibatch MSE: 0.152\n",
      "    Minibatch MSE: 0.099\n",
      "    Minibatch MSE: 0.108\n",
      "    Minibatch MSE: 0.118\n",
      "Epoch: 010 | MSE: 0.09636\n",
      "    Minibatch MSE: 0.058\n",
      "    Minibatch MSE: 0.070\n",
      "    Minibatch MSE: 0.145\n",
      "    Minibatch MSE: 0.081\n",
      "    Minibatch MSE: 0.093\n",
      "    Minibatch MSE: 0.127\n",
      "    Minibatch MSE: 0.115\n",
      "Epoch: 011 | MSE: 0.09638\n",
      "    Minibatch MSE: 0.123\n",
      "    Minibatch MSE: 0.091\n",
      "    Minibatch MSE: 0.085\n",
      "    Minibatch MSE: 0.093\n",
      "    Minibatch MSE: 0.091\n",
      "    Minibatch MSE: 0.143\n",
      "    Minibatch MSE: 0.081\n",
      "Epoch: 012 | MSE: 0.09718\n",
      "    Minibatch MSE: 0.096\n",
      "    Minibatch MSE: 0.076\n",
      "    Minibatch MSE: 0.149\n",
      "    Minibatch MSE: 0.092\n",
      "    Minibatch MSE: 0.116\n",
      "    Minibatch MSE: 0.093\n",
      "    Minibatch MSE: 0.091\n",
      "Epoch: 013 | MSE: 0.09638\n",
      "    Minibatch MSE: 0.095\n",
      "    Minibatch MSE: 0.104\n",
      "    Minibatch MSE: 0.107\n",
      "    Minibatch MSE: 0.120\n",
      "    Minibatch MSE: 0.102\n",
      "    Minibatch MSE: 0.045\n",
      "    Minibatch MSE: 0.124\n",
      "Epoch: 014 | MSE: 0.09685\n",
      "    Minibatch MSE: 0.121\n",
      "    Minibatch MSE: 0.051\n",
      "    Minibatch MSE: 0.095\n",
      "    Minibatch MSE: 0.122\n",
      "    Minibatch MSE: 0.030\n",
      "    Minibatch MSE: 0.158\n",
      "    Minibatch MSE: 0.121\n",
      "Epoch: 015 | MSE: 0.09745\n",
      "    Minibatch MSE: 0.080\n",
      "    Minibatch MSE: 0.119\n",
      "    Minibatch MSE: 0.091\n",
      "    Minibatch MSE: 0.095\n",
      "    Minibatch MSE: 0.044\n",
      "    Minibatch MSE: 0.092\n",
      "    Minibatch MSE: 0.180\n",
      "Epoch: 016 | MSE: 0.09693\n",
      "    Minibatch MSE: 0.054\n",
      "    Minibatch MSE: 0.075\n",
      "    Minibatch MSE: 0.184\n",
      "    Minibatch MSE: 0.105\n",
      "    Minibatch MSE: 0.121\n",
      "    Minibatch MSE: 0.066\n",
      "    Minibatch MSE: 0.096\n",
      "Epoch: 017 | MSE: 0.09699\n",
      "    Minibatch MSE: 0.076\n",
      "    Minibatch MSE: 0.050\n",
      "    Minibatch MSE: 0.198\n",
      "    Minibatch MSE: 0.105\n",
      "    Minibatch MSE: 0.054\n",
      "    Minibatch MSE: 0.136\n",
      "    Minibatch MSE: 0.099\n",
      "Epoch: 018 | MSE: 0.09672\n",
      "    Minibatch MSE: 0.158\n",
      "    Minibatch MSE: 0.099\n",
      "    Minibatch MSE: 0.087\n",
      "    Minibatch MSE: 0.070\n",
      "    Minibatch MSE: 0.103\n",
      "    Minibatch MSE: 0.112\n",
      "    Minibatch MSE: 0.081\n",
      "Epoch: 019 | MSE: 0.09638\n",
      "    Minibatch MSE: 0.095\n",
      "    Minibatch MSE: 0.093\n",
      "    Minibatch MSE: 0.111\n",
      "    Minibatch MSE: 0.147\n",
      "    Minibatch MSE: 0.083\n",
      "    Minibatch MSE: 0.102\n",
      "    Minibatch MSE: 0.065\n",
      "Epoch: 020 | MSE: 0.09635\n"
     ]
    }
   ],
   "source": [
    "model = Adaline1(num_features=X_train.size(1))\n",
    "cost = train(model, \n",
    "             X_train, y_train.float(),\n",
    "             num_epochs=20,\n",
    "             learning_rate=0.1,\n",
    "             seed=123,\n",
    "             minibatch_size=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate ADALINE Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot Loss (MSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAl4UlEQVR4nO3de3xcZ33n8c9PI2lsacaxpRnLjm+yHSeOCRASxyShpCGhJUBLQiklgaYppBtCCUl2my3ZpUtpt8uLslD6AgJpKCmBpQlQQkmpyaVJIIXcrNxvdixfYju2JUuyo5t1nd/+cY7kiTySz8gajTTzfb9e8zrnPOcyPx2N9JvnPOd5jrk7IiIiUVUUOwAREZldlDhERCQvShwiIpIXJQ4REcmLEoeIiOSlstgBTIdUKuWNjY3FDkNEZFZ54okn2tw9Pba8LBJHY2MjTU1NxQ5DRGRWMbNXcpXrUpWIiORFiUNERPKixCEiInlR4hARkbwocYiISF6UOEREJC9KHCIikhcljgk8sLmFb/yiudhhiIjMKEocE3jo5Ta+8eC2YochIjKjKHFMIJ2M090/xOGB4WKHIiIyYyhxTCCdiAPQ1t1f5EhERGYOJY4JpJNB4mjtUuIQERmhxDGBlGocIiJHUeKYwEiN44BqHCIio5Q4JlCfqAZU4xARyabEMYGqWAULaqpU4xARyVLQxGFmF5nZFjNrNrMbc6xfa2aPmFm/md2QVb7MzB40s5fM7AUzuy5rXZ2Z3WdmW8PpgkL+DOlkXDUOEZEsBUscZhYDbgLeDawDLjOzdWM26wCuBb40pnwI+DN3PxU4G/hk1r43Ave7+xrg/nC5YFKJuGocIiJZClnj2AA0u/t2dx8A7gAuzt7A3VvdfRMwOKZ8n7s/Gc53AS8BS8LVFwO3hfO3AZcU7CdgpMYxUMi3EBGZVQqZOJYAu7OW93Dkn39kZtYIvAV4LCxqcPd9ECQYYOE4+11lZk1m1nTgwIF833aUahwiIq9XyMRhOco8rwOYJYAfA9e7e2c++7r7Le6+3t3Xp9PpfHZ9nXQyzuHBYXr6hyZ9DBGRUlLIxLEHWJa1vBTYG3VnM6siSBrfd/c7s1a1mNnicJvFQOsUxDqukU6AqnWIiAQKmTg2AWvMbKWZVQOXAndF2dHMDPg28JK7/92Y1XcBV4TzVwA/naJ4cxrpBKg7q0REApWFOrC7D5nZNcA9QAy41d1fMLOrw/U3m9kioAmYB2TM7HqCO7DeBFwOPGdmT4eH/J/uvhH4AvBDM7sS2AV8sFA/A0Aq7ASoGoeISKBgiQMg/Ee/cUzZzVnz+wkuYY31K3K3keDu7cCFUxjmhEaHHVGNQ0QEUM/xY6qrqcYM2lTjEBEBlDiOqTJWQX1ttWocIiIhJY4Igr4c6gQoIgJKHJGkk3HVOEREQkocEaQTcbVxiIiElDgiSIU1Dve8Or6LiJQkJY4I0ok4A0MZujTsiIiIEkcUqaQ6AYqIjFDiiCCdmAOoL4eICChxRDJa49CdVSIiShxRpMMRclXjEBFR4ohkQU01sQpTjUNEBCWOSCoqLBh2RDUOEREljqhSCT17XEQElDgiSyf17HEREVDiiCydjOspgCIiKHFEFlyq0rAjIiJKHBGlk3EGh53XDg8WOxQRkaJS4ohIzx4XEQkocUSkZ4+LiASUOCIa6T2uGoeIlDsljohGahzqyyEi5U6JI6IT5lZRFTPVOESk7ClxRGRmo7fkioiUMyWOPKQS6j0uIqLEkQcNOyIiosSRl1SiWpeqRKTsKXHkIZ2M094zQCajYUdEpHwpceQhnYgznHEO9uqWXBEpX0oceUip97iIiBJHPo48e1w1DhEpXwVNHGZ2kZltMbNmM7sxx/q1ZvaImfWb2Q1j1t1qZq1m9vyY8s+Z2atm9nT4ek8hf4ZsR2ocfdP1liIiM07BEoeZxYCbgHcD64DLzGzdmM06gGuBL+U4xHeAi8Y5/Ffc/fTwtXGKQj6m0WFHVOMQkTJWyBrHBqDZ3be7+wBwB3Bx9gbu3urum4CjHnLh7g8RJJYZIxmvpLqyQm0cIlLWJkwcZlZhZudO8thLgN1Zy3vCsqlwjZk9G17OWpBrAzO7ysyazKzpwIEDU/KmZkY6EadNnQBFpIxNmDjcPQN8eZLHtlyHnOSxsn0TWA2cDuxjnPjc/RZ3X+/u69Pp9BS8bSCVjKvGISJlLcqlqnvN7ANmlisRTGQPsCxreSmwN89jHMXdW9x9OExq3yK4JDZt0hqvSkTKXJTE8d+AHwEDZtZpZl1m1hlhv03AGjNbaWbVwKXAXccRKwBmtjhr8f3A8+NtWwjppIYdEZHyVnmsDdw9OZkDu/uQmV0D3APEgFvd/QUzuzpcf7OZLQKagHlAxsyuB9a5e6eZ3Q6cD6TMbA/wl+7+beCLZnY6wWWvncDHJxPfZKUTcTp6BhjOOLGKfCthIiKz3zETB4CZvQ84L1z8hbv/LMp+4a2yG8eU3Zw1v5/gElaufS8bp/zyKO9dKKlknIxDe08/C5NzihmKiEhRHPNSlZl9AbgOeDF8XReWlSU9e1xEyl2UGsd7gNPDxmjM7DbgKeConuDlQM8eF5FyF7UD4Pys+RMKEMeskVKNQ0TKXJQax+eBp8zsQYK+GecB/6OgUc1gR2ocShwiUp4mTBxmVgFkgLOBswgSx6fDRu2yVBuvZG5VTDUOESlbEyYOd8+Y2TXu/kOmoA9GqUgn46pxiEjZitLGcZ+Z3WBmy8ysbuRV8MhmsFSiWjUOESlbUdo4PhZOP5lV5sCqqQ9ndkgn4+xo6yl2GCIiRRGljeNGd//BNMUzK6QScR7fMaNGfBcRmTZRRsf95ETblKN0Ms7B3kEGhzPFDkVEZNqpjWMSRvpytKsToIiUIbVxTEJ2X45FJ2i8KhEpL1FGx105HYHMJuo9LiLlbNxLVWb251nzHxyz7vOFDGqmW5hU4hCR8jVRG8elWfNjhxi5qACxzBqjNQ51AhSRMjRR4rBx5nMtl5W51TES8UrVOESkLE2UOHyc+VzLZUfDjohIuZqocfzN4bPFDZib9ZxxA8r+ViINOyIi5WrcxOHusekMZLZJJ+Ns2d9V7DBERKZd1Ac5yRipRFw1DhEpS0ock5ROxOnsG6J/aLjYoYiITCsljklK6dnjIlKmlDgmKR325WjT5SoRKTPjNo6bWRcT3Hbr7vMKEtEskVLvcREpUxPdVZUEMLO/BvYD3yO4FfcjQHJaopvBsgc6FBEpJ1EuVb3L3b/h7l3u3unu3wQ+UOjAZrpUohpQjUNEyk+UxDFsZh8xs5iZVZjZR4Cyv5UoXhlj3pxKjVclImUnSuL4MPAHQEv4+mBYVvY07IiIlKMoz+PYCVxc+FBmH3UCFJFydMwah5mdbGb3m9nz4fKbzOwvCh/azBfUONSPQ0TKS5RLVd8ieB7HIIC7P8vrn9UxLjO7yMy2mFmzmd2YY/1aM3vEzPrN7IYx6241s9aRhJVVXmdm95nZ1nC6IEoshaAah4iUoyiJo8bdHx9TNnSsncwsBtwEvBtYB1xmZuvGbNYBXAt8KcchvkPuB0bdCNzv7muA+8Plokgn43T3D3F4oOzvFRCRMhIlcbSZ2WrCzoBm9vvAvgj7bQCa3X27uw8AdzCmrcTdW919E2FtZsy6hwgSy1gXA7eF87cBl0SIpSBGe4+rgVxEykiUxPFJ4B+AtWb2KnA9cHWE/ZYAu7OW94Rlx6vB3fcBhNOFuTYys6vMrMnMmg4cODAFb3u0kU6AuiVXRMrJhHdVhZebPuHu7zSzWqDC3aM+hCLX42Wn7cmB7n4LcAvA+vXrC/K+o88eVzuHiJSRCWsc7j4MnBnO9+SRNCCoYSzLWl4K7M07wqO1mNligHDaOgXHnBQNOyIi5eiY/TiAp8zsLuBHQM9IobvfeYz9NgFrzGwl8CrBnVhT0XHwLuAK4Avh9KdTcMxJqdewIyJShqIkjjqgHbggq8yBCROHuw+Z2TXAPUAMuNXdXzCzq8P1N5vZIqAJmAdkzOx6YJ27d5rZ7cD5QMrM9gB/6e7fJkgYPzSzK4FdBD3Zi6IqVsGCmirVOESkrETpOf7RyR7c3TcCG8eU3Zw1v5/gElaufS8bp7wduHCyMU21dFJ9OUSkvBwzcZjZHOBK4A3AnJFyd/9YAeOaNdQJUETKTZTbcb8HLALeBfySoIaQTyN5SdOwIyJSbqIkjpPc/X8BPe5+G/Be4I2FDWv2UI1DRMpNlMQx0qv7kJmdBpwANBYsolkmnYxzeHCYnv5jjsIiIlISoiSOW8KBBP8Xwa2wLwJfLGhUs4g6AYpIuYlyV9U/hrO/BFYVNpzZJ7sTYGOqtsjRiIgUXpS7qj6bq9zd/3rqw5l99OxxESk3UToA9mTNzwF+B3ipMOHMPhp2RETKTZRLVV/OXjazLxG0dQhQV1ONmWocIlI+ojSOj1WD2jpGVcYqqK+t5oD6cohImYjSxvEcR4ZDjwFpQO0bWdSXQ0TKSZQ2jt/Jmh8CWtxdnRayBL3HlThEpDxESRxjhxeZZ3bkGU3unuvxrmUlnYiz/UDPsTcUESkBURLHkwQPZDpI8FS/+QTDmUNwCavs2ztSyTgHuvtxd7KTqohIKYrSOH438LvunnL3eoJLV3e6+0p3L/ukAUGNY2AoQ5eGHRGRMhAlcZwVPlcDAHf/OfCbhQtp9kkl1QlQRMpHlMTRZmZ/YWaNZrbCzD5D8ERACaUTwWNK2pQ4RKQMREkclxHcgvsT4F+BhWGZhEZrHLqzSkTKQJSe4x3AdQDhKLmH3N0n3qu8pMMRclXjEJFyMG6Nw8w+a2Zrw/m4mT0ANAMtZvbO6QpwNlhQU02swlTjEJGyMNGlqg8BW8L5K8JtFxI0jH++wHHNKhUVRn1tNW1dGnZERErfRIljIOuS1LuA29192N1fIlr/j7KSSsRV4xCRsjBR4ug3s9PMLA28A7g3a11NYcOafTTsiIiUi4kSx3XAvwCbga+4+w4AM3sP8NQ0xDaraKBDESkX415ycvfHgLU5yjcCG4/eo7yN1Dg07IiIlLrJPI9Dckgn4wwOO68dHix2KCIiBaXEMUX07HERKRdKHFNk5NnjurNKREpdpNtqzexcoDF7e3f/boFimpVGeo+rxiEipS7Ko2O/B6wGngaGw2IHlDiyjNQ42vTscREpcVFqHOuBdRqfamInzK2iKmaqcYhIyYvSxvE8sGgyBzezi8xsi5k1m9mNOdavNbNHzKzfzG6Isq+Zfc7MXjWzp8PXeyYT21QzM1IJdQIUkdIXpcaRAl40s8eB0f+K7v6+iXYysxhwE/BbwB5gk5nd5e4vZm3WAVwLXJLnvl9x9y9FiH1aqROgiJSDKInjc5M89gag2d23A5jZHcDFwGjicPdWoNXM3pvvvjNROhmnpbOv2GGIiBRUlOdx/HKSx14C7M5a3gO8dYr2vcbM/ghoAv7M3Q+OPYCZXQVcBbB8+fI8wp68VKKa5199bVreS0SkWI7ZxmFmZ5vZJjPrNrMBMxs2s84Ix8417kbUBvaJ9v0mwV1epwP7gC/nOoC73+Lu6919fTqdjvi2xyedjNPeM0Amo/sIRKR0RWkc/zrBo2K3AnOBPwnLjmUPsCxreSmwN2Jc4+7r7i3h8O4Z4FsEl7VmhFQiznDGOdirW3JFpHRF6jnu7s1ALPyH/U/A+RF22wSsMbOVZlYNXArcFTGucfc1s8VZ272f4K6vGUF9OUSkHERpHO8N/3k/bWZfJLg8VHusndx9yMyuAe4BYsCt7v6CmV0drr/ZzBYRtFPMAzJmdj1Bn5HOXPuGh/6imZ1OcOlqJ/DxyD9tgWX3Hj9lUbLI0YiIFEaUxHE5Qc3kGuC/ElxC+kCUg+cagt3db86a309wGSrSvmH55VHeuxhSozUO3ZIrIqUryl1Vr5jZXGCxu//VNMQ0a40OdKi+HCJSwqLcVfW7BONU3R0un25mUdsqykoyXkl1ZYVGyBWRkhalcfxzBHcuHQJw96cJRsqVMcyMdCJOm2ocIlLCoiSOIXdXr7aIUsm4ahwiUtIiDXJoZh8GYma2xsy+Bjxc4LhmrbTGqxKREhclcXwKeAPBAIe3A53A9QWMaVZLJ6t1V5WIlLQod1X1Ap8JX3IM6UScjp4BhjNOrCLXyCkiIrPbuInjWHdOHWtY9XKVSsbJOLT39LMwOafY4YiITLmJahznEIxQezvwGLkHHpQxRnqPt3UNKHGISEmaKHEsIniQ0mXAh4F/B27PGvpDchjtBKh2DhEpUeM2jocDGt7t7lcAZwPNwC/M7FPTFt0slBqtcShxiEhpmrBx3MziwHsJah2NwFeBOwsf1uylGoeIlLqJGsdvA04Dfg78lbvPmOHLZ7LaeCVzq2KqcYhIyZqoxnE50AOcDFxrNto2boC7+7wCxzZrpdV7XERK2LiJw90jPeRJjpZKVKv3uIiULCWHAkgn4+o9LiIlS4mjAFIar0pESpgSRwGkk3EO9g4yOJwpdigiIlNOiaMARvpytHcPFDkSEZGpp8RRAGk9e1xESpgSRwGM1DjUziEipUiJowAWqve4iJQwJY4CUI1DREqZEkcBzK2OkYhXqo1DREqSEkeBpJPqyyEipUmJo0BSCT17XERKkxJHgajGISKlSomjQDTsiIiUKiWOAkkn4nT2DdE/NFzsUEREppQSR4GkRnuPa9gRESktBU0cZnaRmW0xs2YzuzHH+rVm9oiZ9ZvZDVH2NbM6M7vPzLaG0wWF/BkmK61nj4tIiSpY4jCzGHAT8G5gHXCZma0bs1kHcC3wpTz2vRG4393XAPeHyzPOSI1D7RwiUmoKWePYADS7+3Z3HwDuAC7O3sDdW919EzCYx74XA7eF87cBlxQo/uOigQ5FpFQVMnEsAXZnLe8Jy4533wZ33wcQThfmOoCZXWVmTWbWdODAgbwCnwr1tdWAahwiUnoKmTgsR5lPw77Bxu63uPt6d1+fTqfz2XVKzKmKMW+Ohh0RkdJTyMSxB1iWtbwU2DsF+7aY2WKAcNp6nHEWTDoZ1wi5IlJyCpk4NgFrzGylmVUDlwJ3TcG+dwFXhPNXAD+dwpinVCoRp61Lt+OKSGmpLNSB3X3IzK4B7gFiwK3u/oKZXR2uv9nMFgFNwDwgY2bXA+vcvTPXvuGhvwD80MyuBHYBHyzUz3C80sk4L+ztLHYYIiJTqmCJA8DdNwIbx5TdnDW/n+AyVKR9w/J24MKpjbQwli6o4e7n9/PEKx2cuaKu2OGIiEwJ9RwvoKvOW8WSBXO56rtPsLujt9jhiIhMCSWOAqqrrebWPz6LweEMH/vOJjr7xnZXERGZfZQ4Cmx1OsHNf3gmO9p6+OT3n2RoOFPskEREjosSxzQ496QUf3PJafzn1jb+6t9exD2vLikiIjNKQRvH5YhLNyxne1sPtzy0nVXpWj76tpXFDklEZFKUOKbRpy9ay462Hv73z15kRX0NF6xtKHZIIiJ506WqaRSrMP7+Q6ezdtE8PvXPT7F5v/p4iMjso8QxzWrjlXz7j9eTmFPJld9porWrr9ghiYjkRYmjCBafMJd//KOz6OgZ4KrvPkHfoB4vKyKzhxJHkbxx6Ql85UOn88yeQ/zZj54hk9GdViIyOyhxFNFFpy3i0xet5d+f3cdX/uPlYocjIhKJ7qoqso+ft4rtB7r52gPNrEzV8ntn5By6S0RkxlCNo8jMjL+55I2cs6qeG3/8HJt2dhQ7JBGRCSlxzADVlRV88w/PYOmCuVz13SZeae8pdkgCZDJOl8YXK4q+wWEe2NzC1+7fyoObW+nuHyp2SJLFymH4i/Xr13tTU1OxwzimHW09vP8bv6a+tpo7//RtnDC3qtghlaW+wWHufPJV/vE/t7OjvYdLTl/CdReuoTFVW+zQStq+1w7zwOZWHniplV9va6Nv8Mi4bpUVxpuXzefc1fWcs7qeM5YvYE5VrIjRlgcze8Ld1x9VrsQxszy6vZ3Lv/0Yb11Zzz999CyqYqoUTpdDvQP8v0df4TsP76Ste4DTlszjLcsW8KMndjM47Pz+GUv51IUnsXRBTbFDjWRXey/P7DnEeWvSnFAz876EDGecZ/Yc4oGXWrl/cysv7Qs6xC6rm8uFaxu4YO1C3rxsPs+/+hoPb2vj183tPLvnEBkPaunrVyzgbSelOGd1PW9acgKV+luZckocsyRxAPywaTd//i/P8uG3Luf/XHIaZlbskEra7o5evv2rHfywaTe9A8P85slpPn7eKs5ZXY+Z0drVxzce3MY/P7YLx7n0rOVcc8FJNMybU+zQjzIwlOHeF/dzx+O7+VVzGwDxygre+8bFXLphOWc1Lijq56mzb5D/fLmNBza38ostrbT3DBCrMM5csYAL1y7kwlMXsjqdGDfGzr5BNu3o4OFt7Ty8rX002STilWxYWTdaIzl10TwqKvR3c7yUOGZR4gD4ws83c/Mvt/GpC07ifW8+kcZUrWofU+z5V1/jHx7azsbn9mHA+04/kavOW8XaRfNybr/30GG+/mAzP9y0m1iFcfnZK7j6/NWkEvHpDTyH7Qe6uWPTbv7liT109AywZP5cPnTWMjasrONnz+7lp0/tpat/iNXpWi7bsJwPnLGUBbXV0xLbjrYe7n+phQc2t/L4jg6GMs4Jc6t4xylp3rF2Ib95cpr5NZOLpaNngEe3t/PwtjYe3tbO9gNB++CCmqrRS1qxCqN/KEP/YIb+oeFgfmg4XM4wMJRd/vp1wxnnxPlzWF5Xy/K6GlbU17C8vobldTXU11aX/Jc6JY5ZljgyGeea259k43P7AaiKGStTtaxpSHLywiQnNyRY05Cksb6mpKro/UPD7GzrZWtrF1tbujnUO8DqhQlOaUhyyqLkpP/BjHB3Htraxj/8chsPb2snEa/kw29dzkff1sjiE+ZGOsau9l6++sBW7nxyD3OqYvzxuY1cdd6q444tX32Dw9z9/H5uf3wXj+3ooLLCeOepDVy6YRlvX5MmlvWNu3dgiJ89u4/bH9/FU7sOUR2r4KLTFnHphmWcs6p+yv4Buju7Ow6zaWcHTa8c5NHt7exoC/6Zn9yQ4IK1DVx46kLesmx+QT63+1/rG00iDze3sfe11w/pUxUz4pUx4pUVwasqa74yRvVoebAM8Oqhw+xq72V/5+uPVVsdY3l9Lcvr5rKivpZldTWsCJPLifPnlsQXPSWOWZY4IEgeL+7rZGtrFy+3dLO1JZjuPtjLyK+tOlbBqnQtJzccSSYnNyRZXlfzun8cM03f4DDbDnTT3NrN1pbuIFG0dvNKey/DYS96M6itrnzdHTUN8+KcsmgepzQkwmmSNQ2JYzaUDg5n+Ldn9nLLQ9vZvL+LhnlxPva2lVz21uXMmzO56//bDnTz9/+xlZ89u5dEdSV/8vZVfOw3GklO8nhRvdzSxe2P7+InT73Kod5BVtTX8KGzlvH7Zy5lYfLYl8827+/kjsd3c+eTe+jsG2JlqnZ0/3xrT0PDGV7c10nTzoM0vdLBpp0HOdDVD0ByTiVnrljAO05ZyAVrF7Ksbnrbhtydjp4BKisqqK4MXsfzN9E3OMyeg7280t7Lro4j05HXwNCRxvxYhXHi/DmsqKvlpIUJTlqY4OSGJGsWJqatpjcVlDhmYeIYT+/AENtae9jS0hUmkyChvHro8Og28coKVqcTLKubS6zCMDMMqDDDLJwS9CMJlsEwKioALFg2qKwIvn3NqYwxtzrGnMoK5lQF8/EcZXMqY8ypDperYgwMZdh2YCQ5dNMcJohdHUeSX6zCaKyvYc3CIAmctDDBmoVJVqVriVdW0NLZz+b9nWzZ38WWli627A+OMfKHWmHQWB8kz1MWJVm7KMnJi5I01tfSOzDEHY/v5tZf72Dfa32c3JDgv7x9FRefvoTqyqn5Rrh5fydfue9l7nmhhfk1VXz8vNVcce4Kaqqnrn/t4YFh/v25oMbwxCsHqYoZ73rDIi7bsJxzVtVP6np+3+AwG5/bxx2P7+bxnR1UxYzfWtfAZRuW87bVqZzH7O4f4qldB9m08yBNOzt4evchegeCsdaWzJ/LWY0LOLOxjrMaF3DywmTZtDNkMk5LVx+72nt5paOXXWFS2dneQ3Nr9+g5Akglqkc/4yc3JDgp/NzPxEtfShwllDjG090/RHNrNy+PJpRu9r12mIwH377cwYFMOD8ydfcx5QBOxoNvlH3hdeDjVRUzVqUSnNSQYE34h7OmIUFjfW3e/8SHhjO80tEbJJPw9XJLFzvbexgZ9isefsPsHRjm7FV1fPy81Zx/Srpgf5zP7XmNv7tvCw9uOUAqUc0nzj+Jj7x1+TFrQ+7BuR7OOBkfeQXLuzt6+cGm3fzr06/S1TfEqlTQRvF7ZyyhfgrbVppbu7jj8d38+Mk9HOwdZFndXC49aznvPLWBl1u6aAovPb20r5OMB8n61MXzOKuxjjNXLGB944LIl/rKjbuz97U+trZ0jdawX27tormlm66s2vSCmqrRv4k1C4OrBytTtdRUx4hVGJUVFeHUpi0hK3GUQeIopEzG6R/KcHhwmL7B4dFp32AmnAbzR8qDl5mxOp1gTUOCFXWFb4/pGxxma0t3WDPppGdgmEvPWsabls4v6Ptme+KVDr5878s8vK2dmurgGvpwJkjKwyOJIXNk/lh/gtUjd0WFjd2F/FbaNzjMPS8Ed2U9sr19tLymOsZbls/nzBVBbeItyxeQiGvEouPh7rR09o+2521t7R69gtDZN3GHx+BqgB2VUEansSPln3//G9mwsm5SMSpxKHHINHt4Wxt3P78fd8LLhRAzG710GKsIli0sqzCoqDBiZlRY8K0yOaeS317XMO0N7xDcDbVpRwenLp7HqYuTJXUTxkzm7hzo7qe5pZsd7T0MhHd3DWU8mA47w5nMkeXRaSZr/ZHyP33Hat5w4gmTikWJQ4lDRCQv4yUOfYUQEZG8KHGIiEhelDhERCQvShwiIpIXJQ4REcmLEoeIiORFiUNERPKixCEiInkpiw6AZnYAeGWSu6eAtikMZ6opvuOj+I6P4jt+MznGFe6eHltYFonjeJhZU66ekzOF4js+iu/4KL7jNxtiHEuXqkREJC9KHCIikhcljmO7pdgBHIPiOz6K7/govuM3G2J8HbVxiIhIXlTjEBGRvChxiIhIXpQ4QmZ2kZltMbNmM7sxx3ozs6+G6581szOmMbZlZvagmb1kZi+Y2XU5tjnfzF4zs6fD12enK77w/Xea2XPhex/11Kwin79Tss7L02bWaWbXj9lmWs+fmd1qZq1m9nxWWZ2Z3WdmW8PpgnH2nfCzWsD4/q+ZbQ5/fz8xs/nj7DvhZ6GA8X3OzF7N+h2+Z5x9i3X+fpAV204ze3qcfQt+/o6bu5f9C4gB24BVQDXwDLBuzDbvAX4OGHA28Ng0xrcYOCOcTwIv54jvfOBnRTyHO4HUBOuLdv5y/K73E3RsKtr5A84DzgCezyr7InBjOH8j8LfjxD/hZ7WA8f02UBnO/22u+KJ8FgoY3+eAGyL8/oty/sas/zLw2WKdv+N9qcYR2AA0u/t2dx8A7gAuHrPNxcB3PfAoMN/MFk9HcO6+z92fDOe7gJeAJdPx3lOoaOdvjAuBbe4+2ZEEpoS7PwR0jCm+GLgtnL8NuCTHrlE+qwWJz93vdfehcPFRYOlUv29U45y/KIp2/kaYmQF/ANw+1e87XZQ4AkuA3VnLezj6H3OUbQrOzBqBtwCP5Vh9jpk9Y2Y/N7M3TG9kOHCvmT1hZlflWD8jzh9wKeP/wRbz/AE0uPs+CL4sAAtzbDNTzuPHCGqQuRzrs1BI14SX0m4d51LfTDh/bwda3H3rOOuLef4iUeIIWI6ysfcpR9mmoMwsAfwYuN7dO8esfpLg8subga8B/zqdsQFvc/czgHcDnzSz88asnwnnrxp4H/CjHKuLff6imgnn8TPAEPD9cTY51mehUL4JrAZOB/YRXA4aq+jnD7iMiWsbxTp/kSlxBPYAy7KWlwJ7J7FNwZhZFUHS+L673zl2vbt3unt3OL8RqDKz1HTF5+57w2kr8BOCSwLZinr+Qu8GnnT3lrErin3+Qi0jl+/CaWuObYr9ObwC+B3gIx5ekB8rwmehINy9xd2H3T0DfGuc9y32+asEfg/4wXjbFOv85UOJI7AJWGNmK8NvpZcCd43Z5i7gj8K7g84GXhu5rFBo4TXRbwMvufvfjbPNonA7zGwDwe+2fZriqzWz5Mg8QSPq82M2K9r5yzLuN71inr8sdwFXhPNXAD/NsU2Uz2pBmNlFwKeB97l77zjbRPksFCq+7Daz94/zvkU7f6F3ApvdfU+ulcU8f3kpduv8THkR3PXzMsEdF58Jy64Grg7nDbgpXP8csH4aY/sNgur0s8DT4es9Y+K7BniB4C6RR4FzpzG+VeH7PhPGMKPOX/j+NQSJ4ISssqKdP4IEtg8YJPgWfCVQD9wPbA2ndeG2JwIbJ/qsTlN8zQTtAyOfwZvHxjfeZ2Ga4vte+Nl6liAZLJ5J5y8s/87IZy5r22k/f8f70pAjIiKSF12qEhGRvChxiIhIXpQ4REQkL0ocIiKSFyUOERHJixKHyBQws2F7/Qi8Uzbqqpk1Zo+yKlJslcUOQKREHHb304sdhMh0UI1DpIDCZyv8rZk9Hr5OCstXmNn94YB895vZ8rC8IXzWxTPh69zwUDEz+5YFz2O518zmFu2HkrKnxCEyNeaOuVT1oax1ne6+Afg68Pdh2dcJhpl/E8FggV8Ny78K/NKDwRbPIOg9DLAGuMnd3wAcAj5Q0J9GZALqOS4yBcys290TOcp3Ahe4+/ZwoMr97l5vZm0EQ2IMhuX73D1lZgeApe7en3WMRuA+d18TLn8aqHL3v5mGH03kKKpxiBSejzM/3ja59GfND6P2SSkiJQ6RwvtQ1vSRcP5hgpFZAT4C/Cqcvx/4BICZxcxs3nQFKRKVvrWITI25ZvZ01vLd7j5yS27czB4j+KJ2WVh2LXCrmf134ADw0bD8OuAWM7uSoGbxCYJRVkVmDLVxiBRQ2Max3t3bih2LyFTRpSoREcmLahwiIpIX1ThERCQvShwiIpIXJQ4REcmLEoeIiORFiUNERPLy/wFN+5jneLqS4gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(len(cost)), cost)\n",
    "plt.ylabel('Mean Squared Error')\n",
    "plt.xlabel('Epoch')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare with analytical solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weights tensor([[-0.0763],\n",
      "        [ 0.4181]])\n",
      "Bias tensor([0.4888])\n"
     ]
    }
   ],
   "source": [
    "print('Weights', model.weights)\n",
    "print('Bias', model.bias)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Analytical weights tensor([[-0.0703],\n",
      "        [ 0.4219]])\n",
      "Analytical bias tensor([0.4857])\n"
     ]
    }
   ],
   "source": [
    "def analytical_solution(x, y):\n",
    "    Xb = torch.cat( (torch.ones((x.size(0), 1)), x), dim=1)\n",
    "    w = torch.zeros(x.size(1))\n",
    "    z = torch.inverse(torch.matmul(Xb.t(), Xb))\n",
    "    params = torch.matmul(z, torch.matmul(Xb.t(), y))\n",
    "    b, w = torch.tensor([params[0]]), params[1:].view(x.size(1), 1)\n",
    "    return w, b\n",
    "\n",
    "w, b = analytical_solution(X_train, y_train.float())\n",
    "print('Analytical weights', w)\n",
    "print('Analytical bias', b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate on Evaluation Metric (Prediction Accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Accuracy: 90.00\n",
      "Test Accuracy: 96.67\n"
     ]
    }
   ],
   "source": [
    "ones = torch.ones(y_train.size())\n",
    "zeros = torch.zeros(y_train.size())\n",
    "train_pred = model.forward(X_train)\n",
    "train_acc = torch.mean(\n",
    "    (torch.where(train_pred > 0.5, \n",
    "                 ones, \n",
    "                 zeros).int() == y_train).float())\n",
    "\n",
    "ones = torch.ones(y_test.size())\n",
    "zeros = torch.zeros(y_test.size())\n",
    "test_pred = model.forward(X_test)\n",
    "test_acc = torch.mean(\n",
    "    (torch.where(test_pred > 0.5, \n",
    "                 ones, \n",
    "                 zeros).int() == y_test).float())\n",
    "\n",
    "print('Training Accuracy: %.2f' % (train_acc*100))\n",
    "print('Test Accuracy: %.2f' % (test_acc*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Boundary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaoAAADCCAYAAAAYX4Z1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAg40lEQVR4nO3de3Bc5X038O9PN0u27hfb0q4uFrZkbK1sRwsyNZeGSw1JCJchTYCWS0o8zbRAkr5OIfZMSAKE1LTv2ynvhDEvaQPxUCYDuMmE4iZjWlomJkhcLAtsjI1l7cqArMtKtiXrsr/3j12tLbTaXe3Zs+fs7vczowHtWZ19JO+zv3N+z+95HlFVEBER2VWW1Q0gIiKKhIGKiIhsjYGKiIhsjYGKiIhsjYGKiIhsjYGKiIhsLceKF62srNSGhgYrXpooITo7O0+qapXV7ZjBPkXpYL5+ZUmgamhoQEdHhxUvTZQQItJjdRvOxz5F6WC+fsXUHxER2RoDFRER2ZolqT8C8KgDmDg19/G8QuB73uS3h4hooZL0OcZAZZVw/7iRHidaoMnJSXg8HoyPj1vdFFvJz8+H0+lEbm6u1U1JfUn6HGOgIkpTHo8HRUVFaGhogIhY3RxbUFUMDAzA4/FgxYoVVjeHYsQxKqI0NT4+joqKCgap84gIKioqeJeZYnhHRZTGGKTmMuVvwjFnU/GOioiS6qGHHsLjjz9uyrk7OzvhcrmwcuVK3HfffUjafnscczYVA5VV8goX9jgRRfXNb34TO3fuxOHDh3H48GG88sorVjcpvSXpc4ypP6swHUA2s/ttL3bsOYS+4THUlBZg6+Zm3LjBYeiczzzzDB5//HGICFpbW/Hss8/OOv7UU09h586dmJiYwMqVK/Hss89i8eLF+OUvf4kf/OAHyM7ORklJCV577TV0d3fj7rvvxsTEBPx+P1544QWsWrUqdK4TJ05gZGQEl1xyCQDgjjvuwO7du3HdddcZ+h0ogiR9jjFQpSrmxCmBdr/txYMvdmFschoA4B0ew4MvdgFA3MGqu7sbjzzyCF5//XVUVlZicHBwznNuvvlmfOMb3wAAbN++HU8//TTuvfde/PCHP8SePXvgcDgwPDwMAHjyySdx//334/bbb8fExASmp6dnncvr9cLpdIa+dzqd8HrZF9IBU3+pijnxtCUitSLyqoi8LyLdInK/2a+5Y8+hUJCaMTY5jR17DsV9zr179+KWW25BZWUlAKC8vHzOcw4cOIDLLrsMLpcLu3btQnd3NwBg06ZNuOuuu/DUU0+FAtIll1yCRx99FD/5yU/Q09ODgoKCWecKNx7FYpL0wEBFZD9TAP5GVS8EsBHAX4nIGjNfsG94bEGPx0JVowaKu+66C0888QS6urrw/e9/P1Q2/uSTT+Lhhx9Gb28v1q9fj4GBAdx222341a9+hYKCAmzevBl79+6ddS6n0wmPxxP63uPxoKamJu72LwjHnE3F1B+RzajqCQAngv8/KiLvA3AAeM+s16wpLYA3TFCqKS0I8+zYXHXVVbjpppvw7W9/GxUVFRgcHJxzVzU6Oorq6mpMTk5i165dcDgCacYjR46gvb0d7e3t+PWvf43e3l74fD40Njbivvvuw9GjR7F//35ceeWVoXNVV1ejqKgI+/btQ3t7O5555hnce++9cbd/QZhuNxXvqIhsTEQaAGwA8EaYY1tEpENEOvr7+w29ztbNzSjIzZ71WEFuNrZubo77nGvXrsW2bdtwxRVXYN26dfjOd74z5zk/+tGP0N7ejmuuuQarV68+156tW+FyudDS0oLLL78c69atw/PPP4+WlhasX78eBw8exB133DHnfD/96U9xzz33YOXKlbjgggtYSJEmJGnzDM7jdruVe+cY9FBJhGO+5LUjQ4lIp6q6TX6NQgD/BeARVX0x0nPD9an3338fF154YcyvZ0bVn10t9G9DyTFfv2LqL1XlFc5f9UcpT0RyAbwAYFe0IJUoN25wpG1gotRmOFCJSC2AZwAsB+AHsFNV/9HoeSkK5sTTlgQqEJ4G8L6q/oPV7SGyWiLGqJJeoUSU5jYB+HMAV4rIO8GvL1jdKCKrGL6jsqJCiSidqer/AOAEIKKghFb9RapQIiIiikfCiimCFUovAPiWqo6EOb4FwBYAqKurS9TLEsWOy04RpaSE3FHFUqGkqjtV1a2q7qqqqkS8LNHCcNkpWzBzm49t27ahtrYWhYWsfk0nhgMVK5SIyC6uv/56/OEPf7C6GZRgiUj9zVQodYnIO8HHvqeqLyfg3ESUDCalRZO5zQcAbNy4Me62kn0louqPFUp2xPEYWggT0qLJ3uaD0hfX+ktXHI8hiyV7mw9KXwxUlDm4FUNSJXubD0pfXOuPMgdTnkmV7G0+DGO63LZ4R0VEprBim4/vfve7cDqdOHPmDJxOJx566KHYG8x0uW1xm490xW1ATJWMbT4WwvA2Hxl2NxH2b8M+Yzlu85FpuA0ILUQaBiNKHwxU6YofPESUJhioMlEqp3lSue1E8eB7nsUUGSmVB41Tue0WsGIM2u7m/ZvYdfoC3/O8oyJKV/n5+RgYGEBFRUXU+UyZQlUxMDCA/Pz8uQcz5O4kFTFQ0cIxFZESnE4nPB4P+vv7rW6KreTn58PpdFrdDFoABipaOKYiUkJubi5WrFhhdTMyg9GLt0g/TxyjIiIyzOjFGy/+ImKgykR2HTSORSq3nSgefM8z9ZeRUnkcKZXbThQPvud5R0VERPbGQEULx1QEESURU3+0cExFECUW1+aMiIHKTJxvRESx4OdBREz9mYklpxQnEfmZiHwqIgesbguR1RioiOzpXwBca3UjiOwgIYGKV39EiaWqrwEYtLodRHaQqDGqfwHwBIBnEnQ+srNIY2+R0prcJZXSFd/bpkrIHRWv/jIMx95sQUS2iEiHiHRw4VlKZxyjMhPnG5GJVHWnqrpV1V1VVWV1c4hMk7TydBHZAmALANTV1SXrZa3FklMiIsOSFqhUdSeAnQDgdru57ajROVYPlUQ45jN2fjPnf3FuWUxE5DkAfwygUkQ8AL6vqk9b2yoia3DCr1XMHucxcn4z28bxrZio6q1Wt4HILhJVnv4cgN8DaBYRj4j8RSLOSzbFsTciSqKE3FHx6i/FREsbRsMUHRElEav+iIjI1hioiIjI1hiorGL2OI9dx5Hs2i4isi1W/VnF6DhPtLGkSOePNEZFRGQzvKOi5GJ5OhEtEAMVERHZGlN/mYgrPYfl9yt6Bs9gv2cYB7w+3H91EwoXsYsQxevMxBTeOT6Mjp4hdPQM4cc3u+AoLVjwedgL7crOSw3ZuW0xUlUcHzyD/R4fDnh9of+Onp0CAOTlZOGG9Q60ODieRxSrT0bG0XFsCB09g+jsGUJ33wim/QoRoGlpEU6OnmWgSit2Hsuxc9vCUFV4hsaw3+PDfm/gbqnL48PIeDAoZWdhdXURvry+Bq3OErQ4StC0rAi52cyME83H71d88OkoOo4NobMnEJx6B8cAAPm5WVhfW4pvXnEB2hrK8Lm6MpQU5Mb9WgxUNNd8GyAmooTczHPjXFA64PVhvzdwl9Tl9WH4zCQAIDdb0Ly8CF9sDQQlVzAo5eUwKBFFMjYxjXd6h9FxbBAdPUN46/gQRoMXe1VFi+CuL8OdlzTA3VCOtTXFCb3QY6CiucxM3yXw3KqKPt84ujw+dHmH0eUdQZdnGEPBoJSTFQhK165dDlcwKDUvL8KinOyEtYEoXX06Mh4YWzo2hM6eQXT3jWDKH9j4omlZIb7UWgN3fRncDWWoK18METGtLQxUmSgFx5hUFR+PzASl4JfHh4HTEwCA7CxB07IiXLNmGVzOUrgcJVi9vAj5uQxKRNH4/YrDn54KjC0dG8Kb56XxFuVkYV1tKbZc3oiLGsoDabzF8afx4sFAlYlSYIzpk2BQmknf7ff4cPLUWQBAlgBNy4rw+dVLQ2NKa6qLGZTIOil28Tc2MY13PcPo7BnCm8cG8VbPUGjMtrIwD+76ctx5SQPa6suwtqbE8tQ4A5VdmTyWY0iC2/bp6Pisyrv9Hh8+HT0XlFYuLcTlTZVodZTA5SzFmupiFOQxKJGN2Pzi79PRcXQeGwqViXd7faE03qqlhfhiazXa6svhri9DfYW5abx4MFDZlQ2vwkIMtO3kqbOhtN1MYPp4ZBwAIAJcUFWIS1dWosVRglZnCdbUFGNxHt+mRLHy+xUf9p+aVSbeM3AGQDCN5yzFNy5vhLu+DG31ZShdnGdxi6PjJ4BdmbnVfJIMnp4IBqXhUFDq850LSisql2BjY3kwKJViTU0xJ9gSLdD45DTe7Q1Mqu0MfvnGAgVFFUvy0FZfhj9rr0dbQxlabJDGiwc/FVKVzVINQzNBKXi31OX1wTs8Fjq+onIJ3A3lcDlK4HKWYG1NMYrykzsgSxSRmRd/0c69gNfuHz2Lzp7B4B3TELr7fJicDqTxLqhagutalqOtvgzuhnI02DCNFw8GqkxkcIzJd2byXFDyDqPL6wtVCAFAfcVibKgrxR2X1MMVLHYoZlAiuzPz4i/auec57j97Gkc+GZ1VJn4smMbLy8nCOmcJ/uLSc2m8siX2T+PFg4HKyqsoq3zPi91ve7FjzyH0DY+hprQAWzc348YNjjlP9Y1NojsYlGYq8Gby3QBQW16AVkcpbm+vh8tRgpaakqSXrhKli3HNxX5tRIe/CZ3+ZnT6V2H4f78GACgPpvFuvbgO7oZytDiKM2ZOIAOVlVdRFtn9thcPvtiFsclpAIB3eAwPvtiF8clp1FcsmbWqw0cnT4d+zllWAJejBF+9qDYUlNL1Co4oGU5qMTq7P0bH5G3o8DfjgK7AZPBjuVH68CfZHXDfdD/c9WVYUbkkLdJ48WCgykA79hwKBakZY5PTeODFrtD3jtICtDiKcUubEy2OwKoO5QxKZDW7ZilioAoc0Rp0+JvQoc3o9DfhI60Gnu1EHjajVY7i69kvw531AdqyDqNcRgM/6H7c2obbQEIClYhcC+AfAWQD+H+q+lgizksRLGCc6fTZKbx3YiRskcNn/fPdF8HlKEFl4aJEtpYoMWyapQhnXHPRpY3o+M8j6Jz4G3T6mzCEIgBAGUbRlnUIX81+Fe57/gkt/9yEfJm0uMX2ZThQiUg2gP8L4BoAHgBvisivVPU9o+fOaHHuGTWmeXivZxD7Pecq8I70n0Jwbh+WFS9Cfk4Wxqf8c37WUVqAzzcvBYCYx7CI0oaRIqO8QgycFXT6m9Dpb8KbwTTeBHKBVw6iMcuBq7M74ZYP4M46hEY5AZHguRvKgUWLgIkwgcoOE/xtIBF3VBcD+FBVjwKAiPwrgBsAMFCZaeIUxjUX72k9uvyN6NIV6PKvwOFxJ/w//T0AoLJwEVqdJfiCqzq0UvjS4vw5Y1QAUJCbja2bmwHMP4YFgMEqSZilsMACUoeqiqMnTwfWxTs2iM68XTg6EhjPzcvOQkttMe5uKEdbsBqvIlqGwuZpS6slIlA5APSe970HQPtnnyQiWwBsAYC6uroEvGyCmLhU0WkUYAnmptkCjy/M+OQ0Dn48ii5PoBx8/9kf47A6MY1A1U8lfHBlHcXmrA64bn8ULkcJlhUvCjv4OhNs5rtjmm8Ma8eeQwxUScAshf2cnZpGl8cXKhN/6/gQBoMLIpcuzoW7vgxfcdfC3VAGl6OE604mWCICVbgyFJ3zgOpOADsBwO12zzluGROvZFrGn577h0DgD/ZRhJ87OzWNQx+Pzlr77oNPRkNrc5UvyYNLhnB11ltwZX0EV9ZRVGMQoZi05umobbtxg2PeoNM3zxjWfI9TwjFLYYYFFGIMnp4IbQbYeWwI+z0+TEwH0uUrKpfgytVLQ1tcNFYWIusxJ/BRahZ5pIJEBCoPgNrzvncC6EvAeVNeTWlB2MKFmvO2Yp6Y8uODT0bPjSl5h3Ho49HQTPPSxblwOUqwpbkxtFK4o7QA8oNb4m9YlA4bS7s5hmWqmLIUGclIBmSeggs9ewofnbc2XkfPEI72B9J4udmCFkcJ7vyjerTVB1J5VUVh0ngpVOSRihIRqN4EsEpEVgDwAvgagNsScN6Ut3Vz85yxoEU5Wbh0ZSW2vdSFLq8PB0+Mhq7UivNz0OosxT2XNQaWGnKUwFlWkPi5E1E6Vbh2cwwrqWLKUtg2nW6mBNydnNUcHNAV6PA3o8PfhLf8qzDw9/8FACgpCKTxbmlzwl1fjlYn03h2YDhQqeqUiPw1gD0IDPz+TFW7DbcsxU1N+9G8vAg3rK/Br9/tw+mJwIf62Sk/nu/oRVF+DlyOEtx9aQNcjhK0OkpRW25CUIoDx7AsF1OWwrbpdJsZmknjTX4Nnf4mvKuNmEBgTmCDfIw/znoH7hv/Gu76MlxQVYisLOv7IM2WkHlUqvoygJcTca5UNDXtx5H+09gfLHTo8vrwXt8IzgZLwIsW5WBjYzlanaWBlcIdJagrX2zrDsExLEsxSxEnVcWxgTPoODYY2hTwyEwaD1/AWvkId2T/Njip9hCqZCTwgxf/HQCmtO2KK1Ms0LRfcaT/1Kwt0bv7fBifDASlJXnZWOsowZ9vDCzI6nKUoKFiia2DUjiROmy0MSx2dmOYpYjdxJQfB/p86Dg2GKrGO3kqUI1XnJ+Dtvoy3Pw5J9z1ZVj38+aIk2qZ0rYvBqoIpv2Kj06eCpSDByvwuvtGcCaYxlucl421NcW47eJ6uJzFcDlK0ViZekHps6J12M+vrsIv9h2f83OfX13Fzp4gmZ6lmM/wmZlqvCF0HhvCu57hUOairnwxLm+qgru+HO6GMqz8bBovyqRaQyltO+/InQYYqIL8fsVHA6dD5eBdXh+6vb7Q2FJ+bhbW1pTgT921ocmzjVWFyLYoKBmaoxWlU0XrsK8e7A972lcP9uPVg/0cv6KEUFX0DJwJbggYuGM6/GngfZuTJVjrKMGfbawPbHHRUIalRfmRTxilEMNQSpsl6KbKyEDl9yt6Bs/M2n22u28Ep85OAQhU5q2pCSzI6nKWwuUowQVVS5CTbZ+dMeOdowXAcIeNp0Nz/IqimZjyo7vPFxpb6uwZxslTZwEARcE03g3ra9BWX471taUoyEtsNR6nZdhX2gcqVcXxUFAKpvD6fBgdDwSlvJwsrKkuxk0bHKExpVVLC20VlMIxs1NFO3e049HaRQQENuDsPH5up9p3e8+l8WrLC3DZqkq4G8rgri/HqqXmV+NxWoZ9pVWgUlV4hsZmTZ494B2BbyyQl87LzsKF1UX48rqaYPquFKuWFSLX5kEpnEjjRICxThWtw0Y7HukYZaaZC8aZoNTZM4gPPgmk8bKzBC01xbi9vT4YmMqwtDhKGs8EnJZhXykbqFQV3uGxWdV3XV4fhs8EglJutmD18uJZC7I2LStCXk7qBaVwIo0TAcY6VbQOG+14tGOU/ian/ejuGzmvTHxoVhrvc3VluL61Bm0NZVhfW4rFefb4KOK0DHuyx7sjClXFCd/4ubXvgrvPziwKmZMlaF5ehGvXLofLGZg827S8MOZtmk3LO5u4yZvRcaTtu7vw3Bu9mFZFtghuba/Fwze6Qs+L1GGjifazzPOnH9/YJN46PhQqE3/XMxyasuEsK8ClKyvQ1lCOixrKsGppkWVFSEbEkm4nc9guUKkqPhk5i/2e4VBQ6vL4MBAMStlZgqZlRbjmwmVocQYmzzYvL4p7mRNT884mrv9VUpCL4bG5pbYlBblRj2/f3TUrbTitGvr+/GA1HyN/M+b5U5+qondwLLQuXuexIXzw6ShUA/1zTXUxvnZRHS5qCJSJL7MgjWeGWMaweAFmDtsFqjt+9gf89+GTAAJv+lVLC3Hl6qWhQocLq4sTuvZWquad51tpaebxSMefe6M37LHn3uiNKVAZ+Zul6t87k01O+/Fe30hobOnNY0PoHw2k8QoX5eBz9WX4Ymt1YFJtbSmWLLLdx0pCREp58wLMXLZ7R315XQ2uWr0ULmcp1lQXJ7wE9bNSNe88MxY33+ORjs+3KNy0xrZcnJG/War+vTPJyPgk3uoZCpWJv9vrC30AO0oL8EcXVATmLtWXo3l5aqbx4jVfWpsXYOayXaD6irs2+pMSyMrlgIyc20gJ+ce+8bBBKTvGBXGN5OqZ57eXmUrZjuCE2s6eIRz6JJDGyxJgTU0xvnpRLdqCey9Vl/DfKRxegJnLdoEq2SLlnc28nTd6biMl5L/sOI7XjwzOOefGxrKY2h7ttc36WTJuatqP906MhIJSR88gPhk5l8bbUFeK61qq4Q5W46VrGi/ReAFmrox/F0bKO296bK+x2/kISxUZTRUYKSHfsedQ2HMeG4jt6i+W8nQzfpbi88bRAbz+4Ul09Azh7ePDs9J47Ssq4G4oQ1t9GVYvL86oNF4i8QLMXBkfqID5886Gb+cjlKD3PfAbY+dG9DJw036vGF7brJ+lhfv574/hlQMf48LqYvyp24m2hnK468t4tZ9AvAAzFwNVBGbezicjVTDfXCmmKTLL9i+uwd/dsg6FTOOZihdg5kmPZRpMsnVzMwo+UwqfqNt5M88NIDRXaqZoYmau1PbdXaa/NtlLTWkBg5QN7H7bi02P7cWKB36DTY/txe63ueJ6rPjujcDM23mzUwWxzJVimoIoOTjPyhgGqijMvJ0389zzzYmaeZxpCqLk4TwrYxio0lS2SMS5UlzuhSh5OM/KGENjVCLyFRHpFhG/iLgT1Sgybr45URsby0JpCO/wGBTn0hDMmROZY75CJRYwxcZoMcUBADcDeC0BbaEEmm9O1LGBsYhpCCJKPBYwGWMo9aeq7wOAxLj0DiUPt4snsg/OszKGY1Q2Fm0cKdJxbhdPZC8sYIpf1NSfiPxORA6E+bphIS8kIltEpENEOvr7w+9OS+dEG0eKdjxSqmFmu/rPmu9xIiIrRb2jUtWrE/FCqroTwE4AcLvdse0nkcGilbNGOx7PWn/zbW9PlC7suhsCRcbUHyK/wYy++eI9t9Gt5gFz1/ojSjV23g2BIjMUqETkJgD/BKAKwG9E5B1V3ZyQliVJpDcYAENvPiPnNrLfVDRc68++ROQrAB4CcCGAi1W1w9oWpQ8zJ91yQq+5DJWnq+pLqupU1UWquizVghQQ+Q1mtIzbyLmjlbMaKXdlqaytccqHSczMJDBLYa6MT/2ZWcZt5NxG9puKhqWy9sUpH+ZJ9d0QMlnGByozy7iNnjve/aZiwVJZyjRmbm7IjRPNlfHbfERKgxlNkZl5bkpdnPJhjRs3OPDjm11wlBZAENjh+Mc3uxK2G4JZ5ybeUcWUBos3RWbmuSl1ccqHdVJ1N4RMJzrPdhBmcrvd2tHBYiZKXSLSqaqmLcQsIv8J4H/FWvXHPkXpYL5+lfF3VER2kg5TPlIZJ+3aEwMVkY2o6ksAXrK6HZmIk3btK+OLKYiIgMjzHslaDFREROCkXTtj6s8g5rSJkstIn9u+uwvPvdGLaVVki+DW9lo8fKMLACft2hnvqAzglu5EyWWkz23f3YVf7DuO6WCl87QqfrHvOLbvDoxDcW6jfTFQGcCcNlFyGelzz73RG/FxTtq1L6b+DGBOmyi5jPS56XnmjJ7/OCft2hPvqAyYL3fNnDaROYz0uex5Fvqd73GyDwYqA5jTJkq83W97semxvVjxwG+w6bG9s8afjPS5W9trF/Q42QdTfwZwuwyixIo26dZIn5up7puv6o/sKy3W+mOJOCWb2Wv9LVS6rPW36bG9YUvEHaUFeP2BKy1oESVT2q71x2VPiNIHC5QonJQfo2KJOFH6YIEShZPygYpXYETpgwVKFE7KBypegRGlD066pXAMjVGJyA4A1wOYAHAEwN2qOpyAdsVs6+bmWWNUAK/AiFIZJ93SZxm9o/otgBZVbQXwAYAHjTdpYXgFRkTJEmmOF5nH0B2Vqv7Hed/uA3CLsebEh1dgRGQ2VhhbJ5FjVF8H8O8JPF9K4BUWUWZghbF1ot5RicjvACwPc2ibqv5b8DnbAEwB2BXhPFsAbAGAurq6uBprN7zCIsocrDC2TtRApapXRzouIncC+BKAqzTCMhequhPATiAwi36B7bSlSFdYDFRE6YUbK1rHUOpPRK4F8LcAvqyqZxLTpNTBKywi+zErHc85XtYxuoTSEwAWAfitBJbK36eqf2m4VSmCV1hE9mJmOp6LUFvHaNXfykQ1JBVxDheRvZidjmeFsTVSflFaK/EKi8hemI5PTwxUBvEKi8g+mI5PTym/1h9ROhGRHSJyUET2i8hLIlJqdZtSCQse0hMDFZG9WL4sWSrjkmrpiak/Ihuxy7JkqYzp+PTDQGWy3W97WWxB8fo6gOetbkSisU/QQjFQmYhLLFE4mbwsGfsExYNjVCbiIpYUjqperaotYb5mgtTMsmS3R1uWTFXdququqqpKVvMNYZ+gePCOykSc00ELdd6yZFek47Jk7BMUD95RmWi+uRuc00ERPAGgCIFlyd4RkSetblAisU9QPBioTMQ5HbRQqrpSVWtVdX3wK63WzmSfoHgw9WciLrFENBv7BMWDgcpknNNBNBv7BC0UU39ERGRrDFRERGRrDFRERGRrEmE+oXkvKtIPoCfpLxxeJYCTVjfCBPy9zFWvqraZZcs+lRT8vcwXtl9ZEqjsREQ6VNVtdTsSjb8XWSVd/434e1mHqT8iIrI1BioiIrI1Bipgp9UNMAl/L7JKuv4b8feySMaPURERkb3xjoqIiGyNgQqAiOwQkYMisl9EXhKRUqvbFC8RuVZEDonIhyLygNXtSQQRqRWRV0XkfRHpFpH7rW4TRcY+ZW+p1qeY+gMgIn8CYK+qTonITwBAVf/W4mYtmIhkA/gAwDUAPADeBHCrqr5nacMMEpFqANWq+paIFAHoBHBjqv9e6Yx9yt5SrU/xjgqAqv6Hqk4Fv90HwGllewy4GMCHqnpUVScA/CuAGyxuk2GqekJV3wr+/yiA9wFwVVMbY5+yt1TrUwxUc30dwL9b3Yg4OQD0nve9BzZ+88VDRBoAbADwhsVNodixT9lYKvSpjNnmQ0R+B2B5mEPbVPXfgs/ZBmAKwK5kti2BJMxjaZPbFZFCAC8A+JaqjljdnkzHPpX6UqVPZUygUtWrIx0XkTsBfAnAVZq6A3ceALXnfe8E0GdRWxJKRHIR6FC7VPVFq9tD7FOpLpX6FIspEKjqAfAPAK5Q1X6r2xMvEclBYOD3KgBeBAZ+b1PVbksbZpCICICfAxhU1W9Z3ByKAfuUvaVan2KgAiAiHwJYBGAg+NA+Vf1LC5sUNxH5AoD/AyAbwM9U9RFrW2SciFwK4L8BdAHwBx/+nqq+bF2rKBL2KXtLtT7FQEVERLbGqj8iIrI1BioiIrI1BioiIrI1BioiIrI1BioiIrI1BioiIrI1BioiIrI1BioiIrK1/w9xlDXJXocVlAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 504x216 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "##########################\n",
    "### 2D Decision Boundary\n",
    "##########################\n",
    "\n",
    "w, b = model.weights, model.bias - 0.5\n",
    "\n",
    "x_min = -3\n",
    "y_min = ( (-(w[0] * x_min) - b[0]) \n",
    "          / w[1] )\n",
    "\n",
    "x_max = 3\n",
    "y_max = ( (-(w[0] * x_max) - b[0]) \n",
    "          / w[1] )\n",
    "\n",
    "\n",
    "fig, ax = plt.subplots(1, 2, sharex=True, figsize=(7, 3))\n",
    "\n",
    "ax[0].plot([x_min, x_max], [y_min, y_max])\n",
    "ax[1].plot([x_min, x_max], [y_min, y_max])\n",
    "\n",
    "ax[0].scatter(X_train[y_train==0, 0], X_train[y_train==0, 1], label='class 0', marker='o')\n",
    "ax[0].scatter(X_train[y_train==1, 0], X_train[y_train==1, 1], label='class 1', marker='s')\n",
    "\n",
    "ax[1].scatter(X_test[y_test==0, 0], X_test[y_test==0, 1], label='class 0', marker='o')\n",
    "ax[1].scatter(X_test[y_test==1, 0], X_test[y_test==1, 1], label='class 1', marker='s')\n",
    "\n",
    "ax[1].legend(loc='upper left')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}