PyTorch/neural_network_from_scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building a Neural Network from scratch\n",
    "\n",
    "---\n",
    "Article: [A Simple Neural Network from Scratch with PyTorch and Google Colab](https://medium.com/dair-ai/a-simple-neural-network-from-scratch-with-pytorch-and-google-colab-c7f3830618e0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `torch` module provides all the necessary **Tensor** operators we will need to implement our first neural network from scratch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data\n",
    "\n",
    "Create some data using `torch.tensor` command.\n",
    "\n",
    "---\n",
    "In the data below, `X` represents the amount of hours studied and how much time students spent sleeping, whereas `y` represents grades.\n",
    "\n",
    "The variable `x_pred` is a single input for which we want to predict a grade using the parameters learned by the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.tensor(([2, 9], [1, 5], [3, 6]), dtype=torch.float)\n",
    "y = torch.tensor(([92], [100], [89]), dtype=torch.float)\n",
    "x_pred = torch.tensor(([4, 8]), dtype=torch.float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the size of the tensors with the `.size()`. This is equivalent to the `.shape` used in **TensorFlow**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of X: torch.Size([3, 2])\n",
      "Size of y: torch.Size([3, 1])\n"
     ]
    }
   ],
   "source": [
    "print('Size of X: {}'.format(X.size()))\n",
    "print('Size of y: {}'.format(y.size()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scaling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_max, _ = torch.max(X, 0)\n",
    "x_pred_max, _ = torch.max(x_pred, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.div(X, X_max)\n",
    "x_pred = torch.div(x_pred, x_pred_max)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = y / 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model\n",
    "\n",
    "---\n",
    "We will build the following neural network.\n",
    "<img src=\"https://miro.medium.com/max/1754/1*hAg3w3tSNjvoUE2HCro9Ww.png\" alt=\"NN_Architecture\" style=\"width: 750px;\"/>\n",
    "<center>Image from: <a href=\"https://medium.com/dair-ai/a-simple-neural-network-from-scratch-with-pytorch-and-google-colab-c7f3830618e0\">A Simple Neural Network from Scratch with PyTorch and Google Colab (Aug 14, 2018)</a></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NeuralNetwork(nn.Module):\n",
    "    def __init__(self, ):\n",
    "        super(NeuralNetwork, self).__init__()\n",
    "        \n",
    "        # parameters\n",
    "        self.input_size = 2\n",
    "        self.output_size = 1\n",
    "        self.hidden_size = 3\n",
    "        \n",
    "        # weights\n",
    "        self.W1 = torch.randn(self.input_size, self.hidden_size)\n",
    "        self.W2 = torch.randn(self.hidden_size, self.output_size)\n",
    "        \n",
    "    def sigmoid(self, s):\n",
    "        return 1 / (1 + torch.exp(-s))\n",
    "    \n",
    "    def sigmoidPrime(self, s):\n",
    "        return s * (1 - s)\n",
    "        \n",
    "    def forward(self, X):\n",
    "        self.z = torch.matmul(X, self.W1)\n",
    "        self.z2 = self.sigmoid(self.z)\n",
    "        self.z3 = torch.matmul(self.z2, self.W2)\n",
    "        o = self.sigmoid(self.z3)\n",
    "        return o\n",
    "    \n",
    "    def backword(self, X, y, o):\n",
    "        self.o_error = y - o\n",
    "        self.o_delta = self.o_error * self.sigmoidPrime(o)\n",
    "        self.z2_error = torch.matmul(self.o_delta, torch.t(self.W2))\n",
    "        self.z2_delta = self.z2_error * self.sigmoidPrime(self.z2)\n",
    "        self.W1 += torch.matmul(torch.t(X), self.z2_delta)\n",
    "        self.W2 += torch.matmul(torch.t(self.z2), self.o_delta)\n",
    "        \n",
    "    def train(self, X, y):\n",
    "        o = self.forward(X)\n",
    "        self.backword(X, y, o)\n",
    "        \n",
    "    def save_weights(self, model):\n",
    "        torch.save(model, 'NN')\n",
    "        \n",
    "        # we can reload the model with all the weights using:\n",
    "        # torch.load('NN')\n",
    "        \n",
    "    def predict(self):\n",
    "        print('Predicted data based on trained weights:')\n",
    "        print('Input (scaled):\\n{}'.format(x_pred))\n",
    "        print('Output:\\n{}'.format(self.forward(x_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = NeuralNetwork()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will train the model for $1000$ epochs.\n",
    "\n",
    "In PyTorch, `model(X)` automatically calls the `forward` function so we do not have to explicitly call `model.forward(x)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0, Loss: 0.020104894414544106\n",
      "Epoch 1, Loss: 0.016600605100393295\n",
      "Epoch 2, Loss: 0.014065555296838284\n",
      "Epoch 3, Loss: 0.012167830020189285\n",
      "Epoch 4, Loss: 0.010707352310419083\n",
      "Epoch 5, Loss: 0.009557505138218403\n",
      "Epoch 6, Loss: 0.008634835481643677\n",
      "Epoch 7, Loss: 0.007882439531385899\n",
      "Epoch 8, Loss: 0.007260380778461695\n",
      "Epoch 9, Loss: 0.00673987390473485\n",
      "Epoch 10, Loss: 0.006299765780568123\n",
      "Epoch 11, Loss: 0.005924209952354431\n",
      "Epoch 12, Loss: 0.005601090844720602\n",
      "Epoch 13, Loss: 0.00532105565071106\n",
      "Epoch 14, Loss: 0.005076756235212088\n",
      "Epoch 15, Loss: 0.004862372297793627\n",
      "Epoch 16, Loss: 0.0046732123009860516\n",
      "Epoch 17, Loss: 0.004505508579313755\n",
      "Epoch 18, Loss: 0.004356133285909891\n",
      "Epoch 19, Loss: 0.0042225392535328865\n",
      "Epoch 20, Loss: 0.004102608654648066\n",
      "Epoch 21, Loss: 0.0039945608004927635\n",
      "Epoch 22, Loss: 0.00389690138399601\n",
      "Epoch 23, Loss: 0.003808359382674098\n",
      "Epoch 24, Loss: 0.0037278523668646812\n",
      "Epoch 25, Loss: 0.0036544662434607744\n",
      "Epoch 26, Loss: 0.0035873905289918184\n",
      "Epoch 27, Loss: 0.003525947453454137\n",
      "Epoch 28, Loss: 0.0034695470239967108\n",
      "Epoch 29, Loss: 0.0034176481422036886\n",
      "Epoch 30, Loss: 0.003369809128344059\n",
      "Epoch 31, Loss: 0.003325631143525243\n",
      "Epoch 32, Loss: 0.0032847595866769552\n",
      "Epoch 33, Loss: 0.003246883163228631\n",
      "Epoch 34, Loss: 0.0032117266673594713\n",
      "Epoch 35, Loss: 0.0031790502835065126\n",
      "Epoch 36, Loss: 0.003148624673485756\n",
      "Epoch 37, Loss: 0.003120261477306485\n",
      "Epoch 38, Loss: 0.003093780716881156\n",
      "Epoch 39, Loss: 0.003069029888138175\n",
      "Epoch 40, Loss: 0.0030458681285381317\n",
      "Epoch 41, Loss: 0.003024171106517315\n",
      "Epoch 42, Loss: 0.003003812627866864\n",
      "Epoch 43, Loss: 0.002984696766361594\n",
      "Epoch 44, Loss: 0.002966724568977952\n",
      "Epoch 45, Loss: 0.002949813148006797\n",
      "Epoch 46, Loss: 0.0029338824097067118\n",
      "Epoch 47, Loss: 0.0029188666958361864\n",
      "Epoch 48, Loss: 0.002904687775298953\n",
      "Epoch 49, Loss: 0.002891303738579154\n",
      "Epoch 50, Loss: 0.002878637285903096\n",
      "Epoch 51, Loss: 0.0028666642028838396\n",
      "Epoch 52, Loss: 0.0028553169686347246\n",
      "Epoch 53, Loss: 0.002844572067260742\n",
      "Epoch 54, Loss: 0.0028343673329800367\n",
      "Epoch 55, Loss: 0.0028246857691556215\n",
      "Epoch 56, Loss: 0.002815485931932926\n",
      "Epoch 57, Loss: 0.002806738018989563\n",
      "Epoch 58, Loss: 0.0027984168846160173\n",
      "Epoch 59, Loss: 0.0027904901653528214\n",
      "Epoch 60, Loss: 0.0027829406317323446\n",
      "Epoch 61, Loss: 0.0027757405769079924\n",
      "Epoch 62, Loss: 0.002768877660855651\n",
      "Epoch 63, Loss: 0.0027623160276561975\n",
      "Epoch 64, Loss: 0.0027560472954064608\n",
      "Epoch 65, Loss: 0.002750054933130741\n",
      "Epoch 66, Loss: 0.0027443254366517067\n",
      "Epoch 67, Loss: 0.0027388364542275667\n",
      "Epoch 68, Loss: 0.0027335805352777243\n",
      "Epoch 69, Loss: 0.002728542312979698\n",
      "Epoch 70, Loss: 0.0027237078174948692\n",
      "Epoch 71, Loss: 0.0027190668042749166\n",
      "Epoch 72, Loss: 0.0027146146167069674\n",
      "Epoch 73, Loss: 0.0027103337924927473\n",
      "Epoch 74, Loss: 0.0027062164153903723\n",
      "Epoch 75, Loss: 0.0027022557333111763\n",
      "Epoch 76, Loss: 0.002698438474908471\n",
      "Epoch 77, Loss: 0.002694767666980624\n",
      "Epoch 78, Loss: 0.0026912291068583727\n",
      "Epoch 79, Loss: 0.002687817672267556\n",
      "Epoch 80, Loss: 0.0026845193933695555\n",
      "Epoch 81, Loss: 0.002681340090930462\n",
      "Epoch 82, Loss: 0.0026782650966197252\n",
      "Epoch 83, Loss: 0.002675293944776058\n",
      "Epoch 84, Loss: 0.002672419650480151\n",
      "Epoch 85, Loss: 0.0026696387212723494\n",
      "Epoch 86, Loss: 0.0026669411454349756\n",
      "Epoch 87, Loss: 0.002664335770532489\n",
      "Epoch 88, Loss: 0.002661803038790822\n",
      "Epoch 89, Loss: 0.0026593513321131468\n",
      "Epoch 90, Loss: 0.00265696388669312\n",
      "Epoch 91, Loss: 0.002654649317264557\n",
      "Epoch 92, Loss: 0.002652403200045228\n",
      "Epoch 93, Loss: 0.0026502106338739395\n",
      "Epoch 94, Loss: 0.0026480855885893106\n",
      "Epoch 95, Loss: 0.002646013395860791\n",
      "Epoch 96, Loss: 0.002643997548148036\n",
      "Epoch 97, Loss: 0.0026420382782816887\n",
      "Epoch 98, Loss: 0.002640119520947337\n",
      "Epoch 99, Loss: 0.002638252219185233\n",
      "Epoch 100, Loss: 0.0026364277582615614\n",
      "Epoch 101, Loss: 0.002634642645716667\n",
      "Epoch 102, Loss: 0.002632906660437584\n",
      "Epoch 103, Loss: 0.002631206763908267\n",
      "Epoch 104, Loss: 0.0026295382995158434\n",
      "Epoch 105, Loss: 0.0026279164012521505\n",
      "Epoch 106, Loss: 0.0026263147592544556\n",
      "Epoch 107, Loss: 0.002624761313199997\n",
      "Epoch 108, Loss: 0.0026232264935970306\n",
      "Epoch 109, Loss: 0.0026217305567115545\n",
      "Epoch 110, Loss: 0.002620255807414651\n",
      "Epoch 111, Loss: 0.0026188110932707787\n",
      "Epoch 112, Loss: 0.002617388730868697\n",
      "Epoch 113, Loss: 0.002615997800603509\n",
      "Epoch 114, Loss: 0.002614628756418824\n",
      "Epoch 115, Loss: 0.0026132792700082064\n",
      "Epoch 116, Loss: 0.0026119526010006666\n",
      "Epoch 117, Loss: 0.002610651543363929\n",
      "Epoch 118, Loss: 0.0026093628257513046\n",
      "Epoch 119, Loss: 0.002608098089694977\n",
      "Epoch 120, Loss: 0.0026068531442433596\n",
      "Epoch 121, Loss: 0.0026056261267513037\n",
      "Epoch 122, Loss: 0.002604410983622074\n",
      "Epoch 123, Loss: 0.002603213069960475\n",
      "Epoch 124, Loss: 0.0026020342484116554\n",
      "Epoch 125, Loss: 0.0026008670683950186\n",
      "Epoch 126, Loss: 0.002599711762741208\n",
      "Epoch 127, Loss: 0.002598576480522752\n",
      "Epoch 128, Loss: 0.002597454469650984\n",
      "Epoch 129, Loss: 0.0025963408406823874\n",
      "Epoch 130, Loss: 0.0025952348951250315\n",
      "Epoch 131, Loss: 0.002594148740172386\n",
      "Epoch 132, Loss: 0.0025930728297680616\n",
      "Epoch 133, Loss: 0.002592000411823392\n",
      "Epoch 134, Loss: 0.0025909394025802612\n",
      "Epoch 135, Loss: 0.0025898946914821863\n",
      "Epoch 136, Loss: 0.00258885626681149\n",
      "Epoch 137, Loss: 0.002587827853858471\n",
      "Epoch 138, Loss: 0.0025868036318570375\n",
      "Epoch 139, Loss: 0.0025857898872345686\n",
      "Epoch 140, Loss: 0.00258478126488626\n",
      "Epoch 141, Loss: 0.002583785681053996\n",
      "Epoch 142, Loss: 0.0025827952194958925\n",
      "Epoch 143, Loss: 0.002581812674179673\n",
      "Epoch 144, Loss: 0.0025808345526456833\n",
      "Epoch 145, Loss: 0.0025798624847084284\n",
      "Epoch 146, Loss: 0.0025788983330130577\n",
      "Epoch 147, Loss: 0.002577937673777342\n",
      "Epoch 148, Loss: 0.002576980274170637\n",
      "Epoch 149, Loss: 0.0025760314892977476\n",
      "Epoch 150, Loss: 0.0025750913191586733\n",
      "Epoch 151, Loss: 0.002574147656559944\n",
      "Epoch 152, Loss: 0.002573215402662754\n",
      "Epoch 153, Loss: 0.0025722896680235863\n",
      "Epoch 154, Loss: 0.0025713620707392693\n",
      "Epoch 155, Loss: 0.0025704400613904\n",
      "Epoch 156, Loss: 0.0025695236399769783\n",
      "Epoch 157, Loss: 0.002568609779700637\n",
      "Epoch 158, Loss: 0.002567694988101721\n",
      "Epoch 159, Loss: 0.0025667885784059763\n",
      "Epoch 160, Loss: 0.002565887290984392\n",
      "Epoch 161, Loss: 0.0025649862363934517\n",
      "Epoch 162, Loss: 0.002564087277278304\n",
      "Epoch 163, Loss: 0.0025631918106228113\n",
      "Epoch 164, Loss: 0.0025623010005801916\n",
      "Epoch 165, Loss: 0.002561410190537572\n",
      "Epoch 166, Loss: 0.002560526365414262\n",
      "Epoch 167, Loss: 0.002559640444815159\n",
      "Epoch 168, Loss: 0.0025587568525224924\n",
      "Epoch 169, Loss: 0.002557872561737895\n",
      "Epoch 170, Loss: 0.002557000145316124\n",
      "Epoch 171, Loss: 0.0025561179500073195\n",
      "Epoch 172, Loss: 0.0025552441366016865\n",
      "Epoch 173, Loss: 0.0025543738156557083\n",
      "Epoch 174, Loss: 0.002553504891693592\n",
      "Epoch 175, Loss: 0.002552633872255683\n",
      "Epoch 176, Loss: 0.0025517656467854977\n",
      "Epoch 177, Loss: 0.0025508941616863012\n",
      "Epoch 178, Loss: 0.002550032688304782\n",
      "Epoch 179, Loss: 0.0025491686537861824\n",
      "Epoch 180, Loss: 0.002548301825299859\n",
      "Epoch 181, Loss: 0.0025474431458860636\n",
      "Epoch 182, Loss: 0.002546582603827119\n",
      "Epoch 183, Loss: 0.002545720897614956\n",
      "Epoch 184, Loss: 0.0025448633823543787\n",
      "Epoch 185, Loss: 0.0025440030731260777\n",
      "Epoch 186, Loss: 0.002543150447309017\n",
      "Epoch 187, Loss: 0.0025422906037420034\n",
      "Epoch 188, Loss: 0.002541439374908805\n",
      "Epoch 189, Loss: 0.0025405813939869404\n",
      "Epoch 190, Loss: 0.002539725275710225\n",
      "Epoch 191, Loss: 0.0025388773065060377\n",
      "Epoch 192, Loss: 0.0025380204897373915\n",
      "Epoch 193, Loss: 0.002537164371460676\n",
      "Epoch 194, Loss: 0.0025363119784742594\n",
      "Epoch 195, Loss: 0.0025354621466249228\n",
      "Epoch 196, Loss: 0.0025346064940094948\n",
      "Epoch 197, Loss: 0.0025337610859423876\n",
      "Epoch 198, Loss: 0.0025329089257866144\n",
      "Epoch 199, Loss: 0.0025320600252598524\n",
      "Epoch 200, Loss: 0.002531204605475068\n",
      "Epoch 201, Loss: 0.0025303559377789497\n",
      "Epoch 202, Loss: 0.0025295077357441187\n",
      "Epoch 203, Loss: 0.002528654644265771\n",
      "Epoch 204, Loss: 0.0025278094690293074\n",
      "Epoch 205, Loss: 0.0025269577745348215\n",
      "Epoch 206, Loss: 0.0025261074770241976\n",
      "Epoch 207, Loss: 0.002525256248190999\n",
      "Epoch 208, Loss: 0.0025244064163416624\n",
      "Epoch 209, Loss: 0.0025235535576939583\n",
      "Epoch 210, Loss: 0.002522706985473633\n",
      "Epoch 211, Loss: 0.00252185738645494\n",
      "Epoch 212, Loss: 0.00252100289799273\n",
      "Epoch 213, Loss: 0.0025201609823852777\n",
      "Epoch 214, Loss: 0.0025193076580762863\n",
      "Epoch 215, Loss: 0.002518454799428582\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 216, Loss: 0.00251760333776474\n",
      "Epoch 217, Loss: 0.0025167574640363455\n",
      "Epoch 218, Loss: 0.002515906235203147\n",
      "Epoch 219, Loss: 0.002515050582587719\n",
      "Epoch 220, Loss: 0.0025141998194158077\n",
      "Epoch 221, Loss: 0.002513347426429391\n",
      "Epoch 222, Loss: 0.00251249922439456\n",
      "Epoch 223, Loss: 0.0025116426404565573\n",
      "Epoch 224, Loss: 0.0025107937399297953\n",
      "Epoch 225, Loss: 0.0025099418126046658\n",
      "Epoch 226, Loss: 0.002509085461497307\n",
      "Epoch 227, Loss: 0.0025082339998334646\n",
      "Epoch 228, Loss: 0.0025073823053389788\n",
      "Epoch 229, Loss: 0.002506527816876769\n",
      "Epoch 230, Loss: 0.0025056747253984213\n",
      "Epoch 231, Loss: 0.0025048204697668552\n",
      "Epoch 232, Loss: 0.002503960859030485\n",
      "Epoch 233, Loss: 0.0025031028781086206\n",
      "Epoch 234, Loss: 0.002502255840227008\n",
      "Epoch 235, Loss: 0.0025013932026922703\n",
      "Epoch 236, Loss: 0.002500542439520359\n",
      "Epoch 237, Loss: 0.0024996830616146326\n",
      "Epoch 238, Loss: 0.0024988267105072737\n",
      "Epoch 239, Loss: 0.0024979703593999147\n",
      "Epoch 240, Loss: 0.0024971088860183954\n",
      "Epoch 241, Loss: 0.0024962525349110365\n",
      "Epoch 242, Loss: 0.002495389198884368\n",
      "Epoch 243, Loss: 0.0024945291224867105\n",
      "Epoch 244, Loss: 0.0024936674162745476\n",
      "Epoch 245, Loss: 0.002492808038368821\n",
      "Epoch 246, Loss: 0.0024919493589550257\n",
      "Epoch 247, Loss: 0.0024910883512347937\n",
      "Epoch 248, Loss: 0.0024902226869016886\n",
      "Epoch 249, Loss: 0.002489360747858882\n",
      "Epoch 250, Loss: 0.002488498343154788\n",
      "Epoch 251, Loss: 0.002487634541466832\n",
      "Epoch 252, Loss: 0.002486767014488578\n",
      "Epoch 253, Loss: 0.0024859074037522078\n",
      "Epoch 254, Loss: 0.002485044300556183\n",
      "Epoch 255, Loss: 0.002484177006408572\n",
      "Epoch 256, Loss: 0.002483309945091605\n",
      "Epoch 257, Loss: 0.0024824426509439945\n",
      "Epoch 258, Loss: 0.002481575822457671\n",
      "Epoch 259, Loss: 0.002480707596987486\n",
      "Epoch 260, Loss: 0.0024798379745334387\n",
      "Epoch 261, Loss: 0.002478974172845483\n",
      "Epoch 262, Loss: 0.002478100825101137\n",
      "Epoch 263, Loss: 0.0024772288743406534\n",
      "Epoch 264, Loss: 0.002476361347362399\n",
      "Epoch 265, Loss: 0.002475488232448697\n",
      "Epoch 266, Loss: 0.0024746181443333626\n",
      "Epoch 267, Loss: 0.002473746659234166\n",
      "Epoch 268, Loss: 0.0024728740099817514\n",
      "Epoch 269, Loss: 0.002471998566761613\n",
      "Epoch 270, Loss: 0.0024711249861866236\n",
      "Epoch 271, Loss: 0.0024702514056116343\n",
      "Epoch 272, Loss: 0.002469372469931841\n",
      "Epoch 273, Loss: 0.0024684963282197714\n",
      "Epoch 274, Loss: 0.0024676204193383455\n",
      "Epoch 275, Loss: 0.002466747537255287\n",
      "Epoch 276, Loss: 0.002465866506099701\n",
      "Epoch 277, Loss: 0.0024649924598634243\n",
      "Epoch 278, Loss: 0.0024641158524900675\n",
      "Epoch 279, Loss: 0.002463236218318343\n",
      "Epoch 280, Loss: 0.0024623561184853315\n",
      "Epoch 281, Loss: 0.002461471362039447\n",
      "Epoch 282, Loss: 0.002460595453158021\n",
      "Epoch 283, Loss: 0.002459710231050849\n",
      "Epoch 284, Loss: 0.002458830364048481\n",
      "Epoch 285, Loss: 0.002457947935909033\n",
      "Epoch 286, Loss: 0.0024570629466325045\n",
      "Epoch 287, Loss: 0.0024561784230172634\n",
      "Epoch 288, Loss: 0.0024552950635552406\n",
      "Epoch 289, Loss: 0.0024544105399399996\n",
      "Epoch 290, Loss: 0.0024535281118005514\n",
      "Epoch 291, Loss: 0.0024526396300643682\n",
      "Epoch 292, Loss: 0.0024517576675862074\n",
      "Epoch 293, Loss: 0.002450864063575864\n",
      "Epoch 294, Loss: 0.00244997744448483\n",
      "Epoch 295, Loss: 0.0024490903597325087\n",
      "Epoch 296, Loss: 0.0024482032749801874\n",
      "Epoch 297, Loss: 0.002447312232106924\n",
      "Epoch 298, Loss: 0.002446416998282075\n",
      "Epoch 299, Loss: 0.0024455292150378227\n",
      "Epoch 300, Loss: 0.0024446374736726284\n",
      "Epoch 301, Loss: 0.00244374992325902\n",
      "Epoch 302, Loss: 0.0024428556207567453\n",
      "Epoch 303, Loss: 0.002441962016746402\n",
      "Epoch 304, Loss: 0.002441067947074771\n",
      "Epoch 305, Loss: 0.002440173178911209\n",
      "Epoch 306, Loss: 0.0024392788764089346\n",
      "Epoch 307, Loss: 0.0024383841082453728\n",
      "Epoch 308, Loss: 0.0024374884087592363\n",
      "Epoch 309, Loss: 0.002436590613797307\n",
      "Epoch 310, Loss: 0.0024356937501579523\n",
      "Epoch 311, Loss: 0.002434793394058943\n",
      "Epoch 312, Loss: 0.0024338962975889444\n",
      "Epoch 313, Loss: 0.002433000598102808\n",
      "Epoch 314, Loss: 0.0024320967495441437\n",
      "Epoch 315, Loss: 0.0024312015157192945\n",
      "Epoch 316, Loss: 0.0024302955716848373\n",
      "Epoch 317, Loss: 0.0024293966125696898\n",
      "Epoch 318, Loss: 0.002428490901365876\n",
      "Epoch 319, Loss: 0.0024275954347103834\n",
      "Epoch 320, Loss: 0.0024266892578452826\n",
      "Epoch 321, Loss: 0.0024257842451334\n",
      "Epoch 322, Loss: 0.0024248810950666666\n",
      "Epoch 323, Loss: 0.002423976780846715\n",
      "Epoch 324, Loss: 0.002423068741336465\n",
      "Epoch 325, Loss: 0.0024221634957939386\n",
      "Epoch 326, Loss: 0.002421259880065918\n",
      "Epoch 327, Loss: 0.002420354401692748\n",
      "Epoch 328, Loss: 0.0024194444995373487\n",
      "Epoch 329, Loss: 0.0024185320362448692\n",
      "Epoch 330, Loss: 0.002417623996734619\n",
      "Epoch 331, Loss: 0.0024167166557163\n",
      "Epoch 332, Loss: 0.0024158067535609007\n",
      "Epoch 333, Loss: 0.0024148980155587196\n",
      "Epoch 334, Loss: 0.0024139857850968838\n",
      "Epoch 335, Loss: 0.0024130737874656916\n",
      "Epoch 336, Loss: 0.002412159461528063\n",
      "Epoch 337, Loss: 0.0024112467654049397\n",
      "Epoch 338, Loss: 0.002410337096080184\n",
      "Epoch 339, Loss: 0.0024094178806990385\n",
      "Epoch 340, Loss: 0.002408503321930766\n",
      "Epoch 341, Loss: 0.0024075889959931374\n",
      "Epoch 342, Loss: 0.0024066760670393705\n",
      "Epoch 343, Loss: 0.002405759645625949\n",
      "Epoch 344, Loss: 0.002404840663075447\n",
      "Epoch 345, Loss: 0.002403923077508807\n",
      "Epoch 346, Loss: 0.002403006423264742\n",
      "Epoch 347, Loss: 0.002402086742222309\n",
      "Epoch 348, Loss: 0.002401170553639531\n",
      "Epoch 349, Loss: 0.002400249009951949\n",
      "Epoch 350, Loss: 0.0023993290960788727\n",
      "Epoch 351, Loss: 0.002398406621068716\n",
      "Epoch 352, Loss: 0.002397485775873065\n",
      "Epoch 353, Loss: 0.002396564232185483\n",
      "Epoch 354, Loss: 0.002395644085481763\n",
      "Epoch 355, Loss: 0.0023947192821651697\n",
      "Epoch 356, Loss: 0.002393799601122737\n",
      "Epoch 357, Loss: 0.0023928722366690636\n",
      "Epoch 358, Loss: 0.0023919495288282633\n",
      "Epoch 359, Loss: 0.002391022862866521\n",
      "Epoch 360, Loss: 0.002390096662566066\n",
      "Epoch 361, Loss: 0.002389170229434967\n",
      "Epoch 362, Loss: 0.002388244727626443\n",
      "Epoch 363, Loss: 0.0023873206228017807\n",
      "Epoch 364, Loss: 0.002386384643614292\n",
      "Epoch 365, Loss: 0.002385454485192895\n",
      "Epoch 366, Loss: 0.0023845285177230835\n",
      "Epoch 367, Loss: 0.002383600454777479\n",
      "Epoch 368, Loss: 0.002382667036727071\n",
      "Epoch 369, Loss: 0.0023817368783056736\n",
      "Epoch 370, Loss: 0.002380811143666506\n",
      "Epoch 371, Loss: 0.002379878656938672\n",
      "Epoch 372, Loss: 0.0023789452388882637\n",
      "Epoch 373, Loss: 0.0023780122864991426\n",
      "Epoch 374, Loss: 0.0023770753759890795\n",
      "Epoch 375, Loss: 0.0023761403281241655\n",
      "Epoch 376, Loss: 0.0023752113338559866\n",
      "Epoch 377, Loss: 0.0023742772173136473\n",
      "Epoch 378, Loss: 0.002373339841142297\n",
      "Epoch 379, Loss: 0.0023724022321403027\n",
      "Epoch 380, Loss: 0.0023714702110737562\n",
      "Epoch 381, Loss: 0.002370531903579831\n",
      "Epoch 382, Loss: 0.002369592199102044\n",
      "Epoch 383, Loss: 0.0023686529602855444\n",
      "Epoch 384, Loss: 0.0023677183780819178\n",
      "Epoch 385, Loss: 0.002366777276620269\n",
      "Epoch 386, Loss: 0.0023658378049731255\n",
      "Epoch 387, Loss: 0.0023648999631404877\n",
      "Epoch 388, Loss: 0.002363958628848195\n",
      "Epoch 389, Loss: 0.002363016363233328\n",
      "Epoch 390, Loss: 0.0023620727006345987\n",
      "Epoch 391, Loss: 0.002361129503697157\n",
      "Epoch 392, Loss: 0.002360190963372588\n",
      "Epoch 393, Loss: 0.0023592470679432154\n",
      "Epoch 394, Loss: 0.002358305035158992\n",
      "Epoch 395, Loss: 0.002357356483116746\n",
      "Epoch 396, Loss: 0.0023564111907035112\n",
      "Epoch 397, Loss: 0.0023554647341370583\n",
      "Epoch 398, Loss: 0.0023545201402157545\n",
      "Epoch 399, Loss: 0.0023535771761089563\n",
      "Epoch 400, Loss: 0.0023526279255747795\n",
      "Epoch 401, Loss: 0.0023516800720244646\n",
      "Epoch 402, Loss: 0.0023507324513047934\n",
      "Epoch 403, Loss: 0.0023497871588915586\n",
      "Epoch 404, Loss: 0.0023488390725106\n",
      "Epoch 405, Loss: 0.0023478891234844923\n",
      "Epoch 406, Loss: 0.0023469419684261084\n",
      "Epoch 407, Loss: 0.0023459880612790585\n",
      "Epoch 408, Loss: 0.0023450362496078014\n",
      "Epoch 409, Loss: 0.0023440865334123373\n",
      "Epoch 410, Loss: 0.0023431365843862295\n",
      "Epoch 411, Loss: 0.0023421859368681908\n",
      "Epoch 412, Loss: 0.0023412343580275774\n",
      "Epoch 413, Loss: 0.0023402797523885965\n",
      "Epoch 414, Loss: 0.0023393237497657537\n",
      "Epoch 415, Loss: 0.0023383733350783587\n",
      "Epoch 416, Loss: 0.002337415935471654\n",
      "Epoch 417, Loss: 0.002336459932848811\n",
      "Epoch 418, Loss: 0.002335506258532405\n",
      "Epoch 419, Loss: 0.002334553049877286\n",
      "Epoch 420, Loss: 0.0023335947189480066\n",
      "Epoch 421, Loss: 0.002332638716325164\n",
      "Epoch 422, Loss: 0.002331681316718459\n",
      "Epoch 423, Loss: 0.0023307267110794783\n",
      "Epoch 424, Loss: 0.0023297606967389584\n",
      "Epoch 425, Loss: 0.002328804461285472\n",
      "Epoch 426, Loss: 0.0023278475273400545\n",
      "Epoch 427, Loss: 0.002326889196410775\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 428, Loss: 0.0023259262088686228\n",
      "Epoch 429, Loss: 0.0023249660152941942\n",
      "Epoch 430, Loss: 0.0023240051232278347\n",
      "Epoch 431, Loss: 0.0023230446968227625\n",
      "Epoch 432, Loss: 0.0023220842704176903\n",
      "Epoch 433, Loss: 0.002321117790415883\n",
      "Epoch 434, Loss: 0.0023201582953333855\n",
      "Epoch 435, Loss: 0.002319195307791233\n",
      "Epoch 436, Loss: 0.0023182325530797243\n",
      "Epoch 437, Loss: 0.0023172690998762846\n",
      "Epoch 438, Loss: 0.0023163044825196266\n",
      "Epoch 439, Loss: 0.002315340330824256\n",
      "Epoch 440, Loss: 0.002314369659870863\n",
      "Epoch 441, Loss: 0.002313406439498067\n",
      "Epoch 442, Loss: 0.002312441822141409\n",
      "Epoch 443, Loss: 0.00231147394515574\n",
      "Epoch 444, Loss: 0.002310506533831358\n",
      "Epoch 445, Loss: 0.0023095381911844015\n",
      "Epoch 446, Loss: 0.002308569848537445\n",
      "Epoch 447, Loss: 0.002307603368535638\n",
      "Epoch 448, Loss: 0.0023066347930580378\n",
      "Epoch 449, Loss: 0.002305666683241725\n",
      "Epoch 450, Loss: 0.0023046948481351137\n",
      "Epoch 451, Loss: 0.0023037276696413755\n",
      "Epoch 452, Loss: 0.002302752574905753\n",
      "Epoch 453, Loss: 0.002301784697920084\n",
      "Epoch 454, Loss: 0.0023008089046925306\n",
      "Epoch 455, Loss: 0.0022998368367552757\n",
      "Epoch 456, Loss: 0.0022988684941083193\n",
      "Epoch 457, Loss: 0.0022978943306952715\n",
      "Epoch 458, Loss: 0.0022969257552176714\n",
      "Epoch 459, Loss: 0.0022959497291594744\n",
      "Epoch 460, Loss: 0.002294976497069001\n",
      "Epoch 461, Loss: 0.002294001169502735\n",
      "Epoch 462, Loss: 0.0022930263075977564\n",
      "Epoch 463, Loss: 0.002292054472491145\n",
      "Epoch 464, Loss: 0.002291079144924879\n",
      "Epoch 465, Loss: 0.0022901000920683146\n",
      "Epoch 466, Loss: 0.0022891254629939795\n",
      "Epoch 467, Loss: 0.0022881447803229094\n",
      "Epoch 468, Loss: 0.002287171082571149\n",
      "Epoch 469, Loss: 0.002286190865561366\n",
      "Epoch 470, Loss: 0.0022852178663015366\n",
      "Epoch 471, Loss: 0.00228423741646111\n",
      "Epoch 472, Loss: 0.0022832639515399933\n",
      "Epoch 473, Loss: 0.0022822872269898653\n",
      "Epoch 474, Loss: 0.002281300025060773\n",
      "Epoch 475, Loss: 0.002280321205034852\n",
      "Epoch 476, Loss: 0.0022793454118072987\n",
      "Epoch 477, Loss: 0.0022783607710152864\n",
      "Epoch 478, Loss: 0.0022773833479732275\n",
      "Epoch 479, Loss: 0.002276396844536066\n",
      "Epoch 480, Loss: 0.002275417558848858\n",
      "Epoch 481, Loss: 0.0022744331508874893\n",
      "Epoch 482, Loss: 0.0022734561935067177\n",
      "Epoch 483, Loss: 0.0022724729496985674\n",
      "Epoch 484, Loss: 0.0022714859806001186\n",
      "Epoch 485, Loss: 0.0022705045994371176\n",
      "Epoch 486, Loss: 0.002269522752612829\n",
      "Epoch 487, Loss: 0.0022685343865305185\n",
      "Epoch 488, Loss: 0.002267551841214299\n",
      "Epoch 489, Loss: 0.0022665683645755053\n",
      "Epoch 490, Loss: 0.0022655853535979986\n",
      "Epoch 491, Loss: 0.0022646014112979174\n",
      "Epoch 492, Loss: 0.002263615606352687\n",
      "Epoch 493, Loss: 0.0022626270074397326\n",
      "Epoch 494, Loss: 0.0022616416681557894\n",
      "Epoch 495, Loss: 0.002260653069242835\n",
      "Epoch 496, Loss: 0.0022596672642976046\n",
      "Epoch 497, Loss: 0.0022586809936910868\n",
      "Epoch 498, Loss: 0.002257690764963627\n",
      "Epoch 499, Loss: 0.00225670519284904\n",
      "Epoch 500, Loss: 0.002255715662613511\n",
      "Epoch 501, Loss: 0.002254724269732833\n",
      "Epoch 502, Loss: 0.002253737533465028\n",
      "Epoch 503, Loss: 0.002252749400213361\n",
      "Epoch 504, Loss: 0.0022517619654536247\n",
      "Epoch 505, Loss: 0.002250772202387452\n",
      "Epoch 506, Loss: 0.0022497803438454866\n",
      "Epoch 507, Loss: 0.002248789183795452\n",
      "Epoch 508, Loss: 0.0022477980237454176\n",
      "Epoch 509, Loss: 0.00224680476821959\n",
      "Epoch 510, Loss: 0.002245816169306636\n",
      "Epoch 511, Loss: 0.0022448196541517973\n",
      "Epoch 512, Loss: 0.0022438305895775557\n",
      "Epoch 513, Loss: 0.002242836868390441\n",
      "Epoch 514, Loss: 0.0022418389562517405\n",
      "Epoch 515, Loss: 0.002240848494693637\n",
      "Epoch 516, Loss: 0.002239854773506522\n",
      "Epoch 517, Loss: 0.0022388638462871313\n",
      "Epoch 518, Loss: 0.0022378680296242237\n",
      "Epoch 519, Loss: 0.0022368766367435455\n",
      "Epoch 520, Loss: 0.0022358817514032125\n",
      "Epoch 521, Loss: 0.0022348847705870867\n",
      "Epoch 522, Loss: 0.00223388965241611\n",
      "Epoch 523, Loss: 0.0022328924387693405\n",
      "Epoch 524, Loss: 0.0022318954579532146\n",
      "Epoch 525, Loss: 0.0022309026680886745\n",
      "Epoch 526, Loss: 0.002229904057458043\n",
      "Epoch 527, Loss: 0.0022289121989160776\n",
      "Epoch 528, Loss: 0.0022279133554548025\n",
      "Epoch 529, Loss: 0.002226913580670953\n",
      "Epoch 530, Loss: 0.0022259175311774015\n",
      "Epoch 531, Loss: 0.002224922878667712\n",
      "Epoch 532, Loss: 0.0022239217069000006\n",
      "Epoch 533, Loss: 0.0022229221649467945\n",
      "Epoch 534, Loss: 0.0022219258826225996\n",
      "Epoch 535, Loss: 0.0022209289018064737\n",
      "Epoch 536, Loss: 0.002219926565885544\n",
      "Epoch 537, Loss: 0.0022189277224242687\n",
      "Epoch 538, Loss: 0.002217929344624281\n",
      "Epoch 539, Loss: 0.0022169288713485003\n",
      "Epoch 540, Loss: 0.002215931424871087\n",
      "Epoch 541, Loss: 0.002214929321780801\n",
      "Epoch 542, Loss: 0.002213924890384078\n",
      "Epoch 543, Loss: 0.002212926046922803\n",
      "Epoch 544, Loss: 0.002211925107985735\n",
      "Epoch 545, Loss: 0.002210924867540598\n",
      "Epoch 546, Loss: 0.0022099195048213005\n",
      "Epoch 547, Loss: 0.002208916237577796\n",
      "Epoch 548, Loss: 0.0022079171612858772\n",
      "Epoch 549, Loss: 0.0022069134283810854\n",
      "Epoch 550, Loss: 0.002205914119258523\n",
      "Epoch 551, Loss: 0.0022049115505069494\n",
      "Epoch 552, Loss: 0.002203907584771514\n",
      "Epoch 553, Loss: 0.0022029043175280094\n",
      "Epoch 554, Loss: 0.002201898954808712\n",
      "Epoch 555, Loss: 0.0022008928935974836\n",
      "Epoch 556, Loss: 0.0021998926531523466\n",
      "Epoch 557, Loss: 0.0021988898515701294\n",
      "Epoch 558, Loss: 0.002197886351495981\n",
      "Epoch 559, Loss: 0.00219688075594604\n",
      "Epoch 560, Loss: 0.0021958753932267427\n",
      "Epoch 561, Loss: 0.00219487096183002\n",
      "Epoch 562, Loss: 0.0021938628051429987\n",
      "Epoch 563, Loss: 0.0021928607020527124\n",
      "Epoch 564, Loss: 0.002191854640841484\n",
      "Epoch 565, Loss: 0.0021908481139689684\n",
      "Epoch 566, Loss: 0.00218983949162066\n",
      "Epoch 567, Loss: 0.0021888362243771553\n",
      "Epoch 568, Loss: 0.0021878296975046396\n",
      "Epoch 569, Loss: 0.0021868275944143534\n",
      "Epoch 570, Loss: 0.00218581547960639\n",
      "Epoch 571, Loss: 0.0021848103497177362\n",
      "Epoch 572, Loss: 0.002183800796046853\n",
      "Epoch 573, Loss: 0.002182792639359832\n",
      "Epoch 574, Loss: 0.002181790303438902\n",
      "Epoch 575, Loss: 0.0021807807497680187\n",
      "Epoch 576, Loss: 0.002179772825911641\n",
      "Epoch 577, Loss: 0.0021787614095956087\n",
      "Epoch 578, Loss: 0.0021777560468763113\n",
      "Epoch 579, Loss: 0.0021767509169876575\n",
      "Epoch 580, Loss: 0.0021757385693490505\n",
      "Epoch 581, Loss: 0.0021747287828475237\n",
      "Epoch 582, Loss: 0.0021737227216362953\n",
      "Epoch 583, Loss: 0.0021727129351347685\n",
      "Epoch 584, Loss: 0.0021717038471251726\n",
      "Epoch 585, Loss: 0.002170695224776864\n",
      "Epoch 586, Loss: 0.002169682877138257\n",
      "Epoch 587, Loss: 0.002168673789128661\n",
      "Epoch 588, Loss: 0.0021676665637642145\n",
      "Epoch 589, Loss: 0.0021666509564965963\n",
      "Epoch 590, Loss: 0.0021656472235918045\n",
      "Epoch 591, Loss: 0.0021646376699209213\n",
      "Epoch 592, Loss: 0.0021636243909597397\n",
      "Epoch 593, Loss: 0.002162614604458213\n",
      "Epoch 594, Loss: 0.0021616036538034678\n",
      "Epoch 595, Loss: 0.0021605922374874353\n",
      "Epoch 596, Loss: 0.002159580821171403\n",
      "Epoch 597, Loss: 0.002158569172024727\n",
      "Epoch 598, Loss: 0.0021575591526925564\n",
      "Epoch 599, Loss: 0.0021565479692071676\n",
      "Epoch 600, Loss: 0.0021555351559072733\n",
      "Epoch 601, Loss: 0.0021545211784541607\n",
      "Epoch 602, Loss: 0.002153511857613921\n",
      "Epoch 603, Loss: 0.002152500208467245\n",
      "Epoch 604, Loss: 0.00215148669667542\n",
      "Epoch 605, Loss: 0.002150477608665824\n",
      "Epoch 606, Loss: 0.0021494647953659296\n",
      "Epoch 607, Loss: 0.002148447325453162\n",
      "Epoch 608, Loss: 0.0021474435925483704\n",
      "Epoch 609, Loss: 0.0021464258898049593\n",
      "Epoch 610, Loss: 0.002145411679521203\n",
      "Epoch 611, Loss: 0.0021443988662213087\n",
      "Epoch 612, Loss: 0.0021433879155665636\n",
      "Epoch 613, Loss: 0.002142376033589244\n",
      "Epoch 614, Loss: 0.002141363453119993\n",
      "Epoch 615, Loss: 0.0021403476130217314\n",
      "Epoch 616, Loss: 0.002139334799721837\n",
      "Epoch 617, Loss: 0.002138321055099368\n",
      "Epoch 618, Loss: 0.0021373119670897722\n",
      "Epoch 619, Loss: 0.002136294497177005\n",
      "Epoch 620, Loss: 0.0021352798212319613\n",
      "Epoch 621, Loss: 0.0021342693362385035\n",
      "Epoch 622, Loss: 0.0021332555916160345\n",
      "Epoch 623, Loss: 0.0021322397515177727\n",
      "Epoch 624, Loss: 0.002131227171048522\n",
      "Epoch 625, Loss: 0.002130210166797042\n",
      "Epoch 626, Loss: 0.002129193162545562\n",
      "Epoch 627, Loss: 0.0021281845401972532\n",
      "Epoch 628, Loss: 0.0021271687000989914\n",
      "Epoch 629, Loss: 0.0021261561196297407\n",
      "Epoch 630, Loss: 0.0021251372527331114\n",
      "Epoch 631, Loss: 0.0021241242066025734\n",
      "Epoch 632, Loss: 0.00212311209179461\n",
      "Epoch 633, Loss: 0.0021220960188657045\n",
      "Epoch 634, Loss: 0.00212108320556581\n",
      "Epoch 635, Loss: 0.002120066899806261\n",
      "Epoch 636, Loss: 0.0021190540865063667\n",
      "Epoch 637, Loss: 0.002118042903020978\n",
      "Epoch 638, Loss: 0.002117025898769498\n",
      "Epoch 639, Loss: 0.0021160051692277193\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 640, Loss: 0.0021149918902665377\n",
      "Epoch 641, Loss: 0.0021139811724424362\n",
      "Epoch 642, Loss: 0.002112963469699025\n",
      "Epoch 643, Loss: 0.0021119501907378435\n",
      "Epoch 644, Loss: 0.0021109336521476507\n",
      "Epoch 645, Loss: 0.0021099199075251818\n",
      "Epoch 646, Loss: 0.0021089043002575636\n",
      "Epoch 647, Loss: 0.002107888925820589\n",
      "Epoch 648, Loss: 0.002106871921569109\n",
      "Epoch 649, Loss: 0.0021058591082692146\n",
      "Epoch 650, Loss: 0.0021048467606306076\n",
      "Epoch 651, Loss: 0.0021038271952420473\n",
      "Epoch 652, Loss: 0.002102817175909877\n",
      "Epoch 653, Loss: 0.0021018001716583967\n",
      "Epoch 654, Loss: 0.0021007820032536983\n",
      "Epoch 655, Loss: 0.002099765930324793\n",
      "Epoch 656, Loss: 0.002098756842315197\n",
      "Epoch 657, Loss: 0.002097737742587924\n",
      "Epoch 658, Loss: 0.002096721902489662\n",
      "Epoch 659, Loss: 0.0020957079250365496\n",
      "Epoch 660, Loss: 0.0020946902222931385\n",
      "Epoch 661, Loss: 0.0020936785731464624\n",
      "Epoch 662, Loss: 0.0020926634315401316\n",
      "Epoch 663, Loss: 0.002091647358611226\n",
      "Epoch 664, Loss: 0.0020906333811581135\n",
      "Epoch 665, Loss: 0.002089619869366288\n",
      "Epoch 666, Loss: 0.0020886030979454517\n",
      "Epoch 667, Loss: 0.002087589353322983\n",
      "Epoch 668, Loss: 0.002086570719256997\n",
      "Epoch 669, Loss: 0.0020855579059571028\n",
      "Epoch 670, Loss: 0.0020845418330281973\n",
      "Epoch 671, Loss: 0.0020835264585912228\n",
      "Epoch 672, Loss: 0.002082515275105834\n",
      "Epoch 673, Loss: 0.0020814980380237103\n",
      "Epoch 674, Loss: 0.0020804814994335175\n",
      "Epoch 675, Loss: 0.0020794656593352556\n",
      "Epoch 676, Loss: 0.0020784500520676374\n",
      "Epoch 677, Loss: 0.002077440032735467\n",
      "Epoch 678, Loss: 0.0020764207001775503\n",
      "Epoch 679, Loss: 0.002075409283861518\n",
      "Epoch 680, Loss: 0.0020743929781019688\n",
      "Epoch 681, Loss: 0.002073380397632718\n",
      "Epoch 682, Loss: 0.002072365256026387\n",
      "Epoch 683, Loss: 0.002071348251774907\n",
      "Epoch 684, Loss: 0.0020703368354588747\n",
      "Epoch 685, Loss: 0.0020693184342235327\n",
      "Epoch 686, Loss: 0.0020683081820607185\n",
      "Epoch 687, Loss: 0.0020672969985753298\n",
      "Epoch 688, Loss: 0.0020662788301706314\n",
      "Epoch 689, Loss: 0.002065267413854599\n",
      "Epoch 690, Loss: 0.002064251573756337\n",
      "Epoch 691, Loss: 0.0020632361993193626\n",
      "Epoch 692, Loss: 0.002062220359221101\n",
      "Epoch 693, Loss: 0.0020612108055502176\n",
      "Epoch 694, Loss: 0.0020601984579116106\n",
      "Epoch 695, Loss: 0.0020591828506439924\n",
      "Epoch 696, Loss: 0.0020581672433763742\n",
      "Epoch 697, Loss: 0.002057156525552273\n",
      "Epoch 698, Loss: 0.0020561409182846546\n",
      "Epoch 699, Loss: 0.0020551285706460476\n",
      "Epoch 700, Loss: 0.0020541141275316477\n",
      "Epoch 701, Loss: 0.0020531064365059137\n",
      "Epoch 702, Loss: 0.0020520903635770082\n",
      "Epoch 703, Loss: 0.0020510766189545393\n",
      "Epoch 704, Loss: 0.0020500640384852886\n",
      "Epoch 705, Loss: 0.002049049362540245\n",
      "Epoch 706, Loss: 0.0020480381790548563\n",
      "Epoch 707, Loss: 0.0020470276940613985\n",
      "Epoch 708, Loss: 0.002046009758487344\n",
      "Epoch 709, Loss: 0.0020450029987841845\n",
      "Epoch 710, Loss: 0.0020439885556697845\n",
      "Epoch 711, Loss: 0.0020429797004908323\n",
      "Epoch 712, Loss: 0.0020419652573764324\n",
      "Epoch 713, Loss: 0.0020409550052136183\n",
      "Epoch 714, Loss: 0.0020399417262524366\n",
      "Epoch 715, Loss: 0.002038935199379921\n",
      "Epoch 716, Loss: 0.0020379177294671535\n",
      "Epoch 717, Loss: 0.0020369095727801323\n",
      "Epoch 718, Loss: 0.002035893267020583\n",
      "Epoch 719, Loss: 0.002034886507317424\n",
      "Epoch 720, Loss: 0.0020338722970336676\n",
      "Epoch 721, Loss: 0.002032863674685359\n",
      "Epoch 722, Loss: 0.0020318522583693266\n",
      "Epoch 723, Loss: 0.0020308394450694323\n",
      "Epoch 724, Loss: 0.002029829891398549\n",
      "Epoch 725, Loss: 0.0020288198720663786\n",
      "Epoch 726, Loss: 0.002027814043685794\n",
      "Epoch 727, Loss: 0.0020268040243536234\n",
      "Epoch 728, Loss: 0.0020257923752069473\n",
      "Epoch 729, Loss: 0.0020247860811650753\n",
      "Epoch 730, Loss: 0.0020237755961716175\n",
      "Epoch 731, Loss: 0.0020227625500410795\n",
      "Epoch 732, Loss: 0.0020217534620314837\n",
      "Epoch 733, Loss: 0.002020743675529957\n",
      "Epoch 734, Loss: 0.0020197383128106594\n",
      "Epoch 735, Loss: 0.0020187273621559143\n",
      "Epoch 736, Loss: 0.0020177217666059732\n",
      "Epoch 737, Loss: 0.0020167126785963774\n",
      "Epoch 738, Loss: 0.002015706617385149\n",
      "Epoch 739, Loss: 0.002014697762206197\n",
      "Epoch 740, Loss: 0.00201368797570467\n",
      "Epoch 741, Loss: 0.0020126786548644304\n",
      "Epoch 742, Loss: 0.0020116723608225584\n",
      "Epoch 743, Loss: 0.0020106667652726173\n",
      "Epoch 744, Loss: 0.002009656513109803\n",
      "Epoch 745, Loss: 0.002008653013035655\n",
      "Epoch 746, Loss: 0.0020076490473002195\n",
      "Epoch 747, Loss: 0.0020066401921212673\n",
      "Epoch 748, Loss: 0.0020056322682648897\n",
      "Epoch 749, Loss: 0.00200462993234396\n",
      "Epoch 750, Loss: 0.002003621542826295\n",
      "Epoch 751, Loss: 0.0020026173442602158\n",
      "Epoch 752, Loss: 0.002001611515879631\n",
      "Epoch 753, Loss: 0.0020006061531603336\n",
      "Epoch 754, Loss: 0.0019996033515781164\n",
      "Epoch 755, Loss: 0.001998597988858819\n",
      "Epoch 756, Loss: 0.0019975926261395216\n",
      "Epoch 757, Loss: 0.0019965895917266607\n",
      "Epoch 758, Loss: 0.0019955814350396395\n",
      "Epoch 759, Loss: 0.001994579564779997\n",
      "Epoch 760, Loss: 0.0019935728050768375\n",
      "Epoch 761, Loss: 0.001992573728784919\n",
      "Epoch 762, Loss: 0.001991569297388196\n",
      "Epoch 763, Loss: 0.001990565564483404\n",
      "Epoch 764, Loss: 0.0019895609002560377\n",
      "Epoch 765, Loss: 0.0019885601941496134\n",
      "Epoch 766, Loss: 0.0019875578582286835\n",
      "Epoch 767, Loss: 0.0019865562207996845\n",
      "Epoch 768, Loss: 0.001985556213185191\n",
      "Epoch 769, Loss: 0.0019845543429255486\n",
      "Epoch 770, Loss: 0.0019835515413433313\n",
      "Epoch 771, Loss: 0.0019825529307127\n",
      "Epoch 772, Loss: 0.001981552457436919\n",
      "Epoch 773, Loss: 0.001980547560378909\n",
      "Epoch 774, Loss: 0.0019795470871031284\n",
      "Epoch 775, Loss: 0.0019785494077950716\n",
      "Epoch 776, Loss: 0.0019775473047047853\n",
      "Epoch 777, Loss: 0.0019765482284128666\n",
      "Epoch 778, Loss: 0.001975548453629017\n",
      "Epoch 779, Loss: 0.0019745505414903164\n",
      "Epoch 780, Loss: 0.001973553327843547\n",
      "Epoch 781, Loss: 0.0019725512247532606\n",
      "Epoch 782, Loss: 0.0019715495873242617\n",
      "Epoch 783, Loss: 0.0019705574959516525\n",
      "Epoch 784, Loss: 0.0019695572555065155\n",
      "Epoch 785, Loss: 0.0019685637671500444\n",
      "Epoch 786, Loss: 0.0019675656221807003\n",
      "Epoch 787, Loss: 0.001966568175703287\n",
      "Epoch 788, Loss: 0.001965567469596863\n",
      "Epoch 789, Loss: 0.001964575843885541\n",
      "Epoch 790, Loss: 0.001963576301932335\n",
      "Epoch 791, Loss: 0.001962583279237151\n",
      "Epoch 792, Loss: 0.0019615872297436\n",
      "Epoch 793, Loss: 0.0019605914130806923\n",
      "Epoch 794, Loss: 0.0019595976918935776\n",
      "Epoch 795, Loss: 0.0019586016424000263\n",
      "Epoch 796, Loss: 0.00195760908536613\n",
      "Epoch 797, Loss: 0.001956611406058073\n",
      "Epoch 798, Loss: 0.0019556162878870964\n",
      "Epoch 799, Loss: 0.001954625127837062\n",
      "Epoch 800, Loss: 0.001953630708158016\n",
      "Epoch 801, Loss: 0.001952637918293476\n",
      "Epoch 802, Loss: 0.0019516459433361888\n",
      "Epoch 803, Loss: 0.0019506566459313035\n",
      "Epoch 804, Loss: 0.0019496629247441888\n",
      "Epoch 805, Loss: 0.001948671299032867\n",
      "Epoch 806, Loss: 0.0019476775778457522\n",
      "Epoch 807, Loss: 0.0019466918893158436\n",
      "Epoch 808, Loss: 0.0019457003800198436\n",
      "Epoch 809, Loss: 0.0019447071244940162\n",
      "Epoch 810, Loss: 0.0019437167793512344\n",
      "Epoch 811, Loss: 0.0019427278311923146\n",
      "Epoch 812, Loss: 0.0019417396979406476\n",
      "Epoch 813, Loss: 0.0019407509826123714\n",
      "Epoch 814, Loss: 0.001939761103130877\n",
      "Epoch 815, Loss: 0.0019387720385566354\n",
      "Epoch 816, Loss: 0.0019377860007807612\n",
      "Epoch 817, Loss: 0.0019367941422387958\n",
      "Epoch 818, Loss: 0.0019358129939064384\n",
      "Epoch 819, Loss: 0.0019348225323483348\n",
      "Epoch 820, Loss: 0.0019338334677740932\n",
      "Epoch 821, Loss: 0.0019328502239659429\n",
      "Epoch 822, Loss: 0.0019318647682666779\n",
      "Epoch 823, Loss: 0.0019308788469061255\n",
      "Epoch 824, Loss: 0.0019298932747915387\n",
      "Epoch 825, Loss: 0.001928910263814032\n",
      "Epoch 826, Loss: 0.0019279196858406067\n",
      "Epoch 827, Loss: 0.0019269371405243874\n",
      "Epoch 828, Loss: 0.0019259570399299264\n",
      "Epoch 829, Loss: 0.0019249730976298451\n",
      "Epoch 830, Loss: 0.0019239868270233274\n",
      "Epoch 831, Loss: 0.0019230021862313151\n",
      "Epoch 832, Loss: 0.001922022202052176\n",
      "Epoch 833, Loss: 0.0019210410537198186\n",
      "Epoch 834, Loss: 0.001920055947266519\n",
      "Epoch 835, Loss: 0.0019190721213817596\n",
      "Epoch 836, Loss: 0.0019180962117388844\n",
      "Epoch 837, Loss: 0.0019171129679307342\n",
      "Epoch 838, Loss: 0.0019161347299814224\n",
      "Epoch 839, Loss: 0.0019151534652337432\n",
      "Epoch 840, Loss: 0.0019141747616231441\n",
      "Epoch 841, Loss: 0.001913194078952074\n",
      "Epoch 842, Loss: 0.0019122165394946933\n",
      "Epoch 843, Loss: 0.001911235973238945\n",
      "Epoch 844, Loss: 0.0019102605292573571\n",
      "Epoch 845, Loss: 0.0019092797301709652\n",
      "Epoch 846, Loss: 0.0019083041697740555\n",
      "Epoch 847, Loss: 0.0019073294242843986\n",
      "Epoch 848, Loss: 0.001906352466903627\n",
      "Epoch 849, Loss: 0.0019053762080147862\n",
      "Epoch 850, Loss: 0.0019044014625251293\n",
      "Epoch 851, Loss: 0.001903423690237105\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 852, Loss: 0.00190245162229985\n",
      "Epoch 853, Loss: 0.0019014743156731129\n",
      "Epoch 854, Loss: 0.001900499570183456\n",
      "Epoch 855, Loss: 0.0018995291320607066\n",
      "Epoch 856, Loss: 0.001898553571663797\n",
      "Epoch 857, Loss: 0.0018975860439240932\n",
      "Epoch 858, Loss: 0.001896609552204609\n",
      "Epoch 859, Loss: 0.0018956380663439631\n",
      "Epoch 860, Loss: 0.001894668792374432\n",
      "Epoch 861, Loss: 0.0018936949782073498\n",
      "Epoch 862, Loss: 0.0018927253549918532\n",
      "Epoch 863, Loss: 0.0018917525885626674\n",
      "Epoch 864, Loss: 0.0018907837802544236\n",
      "Epoch 865, Loss: 0.0018898132257163525\n",
      "Epoch 866, Loss: 0.001888845581561327\n",
      "Epoch 867, Loss: 0.0018878733972087502\n",
      "Epoch 868, Loss: 0.0018869066843762994\n",
      "Epoch 869, Loss: 0.001885935664176941\n",
      "Epoch 870, Loss: 0.00188496767077595\n",
      "Epoch 871, Loss: 0.0018840072443708777\n",
      "Epoch 872, Loss: 0.0018830354092642665\n",
      "Epoch 873, Loss: 0.001882069162093103\n",
      "Epoch 874, Loss: 0.001881105243228376\n",
      "Epoch 875, Loss: 0.0018801400437951088\n",
      "Epoch 876, Loss: 0.0018791748443618417\n",
      "Epoch 877, Loss: 0.00187820999417454\n",
      "Epoch 878, Loss: 0.0018772451439872384\n",
      "Epoch 879, Loss: 0.0018762815743684769\n",
      "Epoch 880, Loss: 0.0018753185868263245\n",
      "Epoch 881, Loss: 0.0018743574619293213\n",
      "Epoch 882, Loss: 0.0018733980832621455\n",
      "Epoch 883, Loss: 0.0018724313704296947\n",
      "Epoch 884, Loss: 0.0018714704783633351\n",
      "Epoch 885, Loss: 0.0018705114489421248\n",
      "Epoch 886, Loss: 0.0018695509061217308\n",
      "Epoch 887, Loss: 0.0018685891991481185\n",
      "Epoch 888, Loss: 0.001867632963694632\n",
      "Epoch 889, Loss: 0.0018666709074750543\n",
      "Epoch 890, Loss: 0.001865712576545775\n",
      "Epoch 891, Loss: 0.0018647565739229321\n",
      "Epoch 892, Loss: 0.0018637942848727107\n",
      "Epoch 893, Loss: 0.0018628408433869481\n",
      "Epoch 894, Loss: 0.0018618860049173236\n",
      "Epoch 895, Loss: 0.001860927790403366\n",
      "Epoch 896, Loss: 0.0018599660834297538\n",
      "Epoch 897, Loss: 0.001859014737419784\n",
      "Epoch 898, Loss: 0.001858059666119516\n",
      "Epoch 899, Loss: 0.0018571087857708335\n",
      "Epoch 900, Loss: 0.001856153248809278\n",
      "Epoch 901, Loss: 0.0018551998073235154\n",
      "Epoch 902, Loss: 0.0018542474135756493\n",
      "Epoch 903, Loss: 0.001853293739259243\n",
      "Epoch 904, Loss: 0.0018523400649428368\n",
      "Epoch 905, Loss: 0.0018513924442231655\n",
      "Epoch 906, Loss: 0.001850439584814012\n",
      "Epoch 907, Loss: 0.0018494868418201804\n",
      "Epoch 908, Loss: 0.0018485375912860036\n",
      "Epoch 909, Loss: 0.001847588922828436\n",
      "Epoch 910, Loss: 0.0018466388573870063\n",
      "Epoch 911, Loss: 0.0018456910038366914\n",
      "Epoch 912, Loss: 0.001844744081608951\n",
      "Epoch 913, Loss: 0.0018437964608892798\n",
      "Epoch 914, Loss: 0.0018428504699841142\n",
      "Epoch 915, Loss: 0.0018419037805870175\n",
      "Epoch 916, Loss: 0.001840955694206059\n",
      "Epoch 917, Loss: 0.0018400136614218354\n",
      "Epoch 918, Loss: 0.001839064876548946\n",
      "Epoch 919, Loss: 0.0018381215631961823\n",
      "Epoch 920, Loss: 0.0018371768528595567\n",
      "Epoch 921, Loss: 0.0018362348200753331\n",
      "Epoch 922, Loss: 0.0018352889455854893\n",
      "Epoch 923, Loss: 0.0018343469128012657\n",
      "Epoch 924, Loss: 0.0018334080232307315\n",
      "Epoch 925, Loss: 0.0018324655247852206\n",
      "Epoch 926, Loss: 0.001831523492000997\n",
      "Epoch 927, Loss: 0.0018305819248780608\n",
      "Epoch 928, Loss: 0.0018296422204002738\n",
      "Epoch 929, Loss: 0.0018287025159224868\n",
      "Epoch 930, Loss: 0.0018277630442753434\n",
      "Epoch 931, Loss: 0.0018268265994265676\n",
      "Epoch 932, Loss: 0.0018258901545777917\n",
      "Epoch 933, Loss: 0.001824948820285499\n",
      "Epoch 934, Loss: 0.0018240137724205852\n",
      "Epoch 935, Loss: 0.001823080237954855\n",
      "Epoch 936, Loss: 0.0018221442587673664\n",
      "Epoch 937, Loss: 0.001821206766180694\n",
      "Epoch 938, Loss: 0.0018202685751020908\n",
      "Epoch 939, Loss: 0.0018193359719589353\n",
      "Epoch 940, Loss: 0.0018184026703238487\n",
      "Epoch 941, Loss: 0.0018174746073782444\n",
      "Epoch 942, Loss: 0.001816540607251227\n",
      "Epoch 943, Loss: 0.0018156060250476003\n",
      "Epoch 944, Loss: 0.00181467121001333\n",
      "Epoch 945, Loss: 0.0018137450097128749\n",
      "Epoch 946, Loss: 0.0018128176452592015\n",
      "Epoch 947, Loss: 0.001811884343624115\n",
      "Epoch 948, Loss: 0.0018109572120010853\n",
      "Epoch 949, Loss: 0.0018100268207490444\n",
      "Epoch 950, Loss: 0.0018090995727106929\n",
      "Epoch 951, Loss: 0.0018081687157973647\n",
      "Epoch 952, Loss: 0.001807242282666266\n",
      "Epoch 953, Loss: 0.0018063141033053398\n",
      "Epoch 954, Loss: 0.001805391744710505\n",
      "Epoch 955, Loss: 0.0018044644966721535\n",
      "Epoch 956, Loss: 0.0018035402754321694\n",
      "Epoch 957, Loss: 0.0018026120960712433\n",
      "Epoch 958, Loss: 0.0018016905523836613\n",
      "Epoch 959, Loss: 0.0018007656326517463\n",
      "Epoch 960, Loss: 0.0017998452531173825\n",
      "Epoch 961, Loss: 0.001798922661691904\n",
      "Epoch 962, Loss: 0.0017980023985728621\n",
      "Epoch 963, Loss: 0.0017970820190384984\n",
      "Epoch 964, Loss: 0.0017961588455364108\n",
      "Epoch 965, Loss: 0.0017952361376956105\n",
      "Epoch 966, Loss: 0.0017943219281733036\n",
      "Epoch 967, Loss: 0.001793405506759882\n",
      "Epoch 968, Loss: 0.0017924861749634147\n",
      "Epoch 969, Loss: 0.0017915653297677636\n",
      "Epoch 970, Loss: 0.0017906497232615948\n",
      "Epoch 971, Loss: 0.001789734815247357\n",
      "Epoch 972, Loss: 0.001788812573067844\n",
      "Epoch 973, Loss: 0.0017878984799608588\n",
      "Epoch 974, Loss: 0.001786984852515161\n",
      "Epoch 975, Loss: 0.0017860709922388196\n",
      "Epoch 976, Loss: 0.0017851596930995584\n",
      "Epoch 977, Loss: 0.0017842432716861367\n",
      "Epoch 978, Loss: 0.0017833318561315536\n",
      "Epoch 979, Loss: 0.0017824248643592\n",
      "Epoch 980, Loss: 0.0017815078608691692\n",
      "Epoch 981, Loss: 0.001780597842298448\n",
      "Epoch 982, Loss: 0.0017796875908970833\n",
      "Epoch 983, Loss: 0.0017787812976166606\n",
      "Epoch 984, Loss: 0.0017778672045096755\n",
      "Epoch 985, Loss: 0.001776958699338138\n",
      "Epoch 986, Loss: 0.0017760529881343246\n",
      "Epoch 987, Loss: 0.0017751437844708562\n",
      "Epoch 988, Loss: 0.0017742384225130081\n",
      "Epoch 989, Loss: 0.0017733342247083783\n",
      "Epoch 990, Loss: 0.0017724279314279556\n",
      "Epoch 991, Loss: 0.0017715241992846131\n",
      "Epoch 992, Loss: 0.001770618837326765\n",
      "Epoch 993, Loss: 0.0017697160365059972\n",
      "Epoch 994, Loss: 0.0017688125371932983\n",
      "Epoch 995, Loss: 0.0017679092707112432\n",
      "Epoch 996, Loss: 0.001767009380273521\n",
      "Epoch 997, Loss: 0.001766105298884213\n",
      "Epoch 998, Loss: 0.0017652082024142146\n",
      "Epoch 999, Loss: 0.0017643081955611706\n"
     ]
    }
   ],
   "source": [
    "for i in range(1000):\n",
    "    loss = torch.mean((y - model(X)) ** 2).detach().item()\n",
    "    print('Epoch {}, Loss: {}'.format(str(i), loss))\n",
    "    model.train(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "model.save_weights(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted data based on trained weights:\n",
      "Input (scaled):\n",
      "tensor([0.5000, 1.0000])\n",
      "Output:\n",
      "tensor([0.9563])\n"
     ]
    }
   ],
   "source": [
    "model.predict()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}