exercise_690.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi-Layer Neural Network using Sigmoid & Bias on Toy Problem 3\n",
    "\n",
    "In exercise 670, we built a multi-layer classifier for Toy Problem 3 and used **the ReLU as the activation function**.\n",
    "\n",
    "We saw that the classifier could get to 97% accuracy on Toy Problem 3.\n",
    "\n",
    "In exercise 680, we saw that if we **used a sigmoid** instead of the ReLU, the classifier would not learn anything.\n",
    "\n",
    "Now, we use a sigmoid again, but also use a bias in addition to the weights.\n",
    "\n",
    "We see that the bias gives the classifier more power (it learns the XOR function or Toy Problem 3).\n",
    "\n",
    "We've provided a utility class 'Data' (in data_reader.py) to load the training data (it works for all the toy problems)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Labels:\n",
      "[0, 0, 0, 1, 1, 0, 1, 1, 1, 1]\n",
      "Features:\n",
      "[[-30, -49], [57, 97], [16, 5], [80, -1], [32, -49], [-73, -82], [28, -85], [89, -29], [44, -46], [83, -49]]\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "from data_reader import Data\n",
    "\n",
    "data = Data(\"data/toy_problem_3_train.txt\")\n",
    "\n",
    "labels, features = data.get_sample()\n",
    "\n",
    "print(\"Labels:\\n\"+str(labels))\n",
    "\n",
    "print(\"Features:\\n\"+str(features))\n",
    "    \n",
    "target = torch.autograd.Variable(torch.LongTensor(labels))\n",
    "#print(\"Labels Tensor:\\n\"+str(target))\n",
    "\n",
    "features = torch.autograd.Variable(torch.Tensor(features))\n",
    "#print(\"Features Tensor:\\n\"+str(features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We initialize the weights and biases (one set of weights and biases per layer) randomly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weights1 => Parameter containing:\n",
      "tensor([[ 0.7704,  0.6615,  0.6316,  0.8493],\n",
      "        [ 0.6410,  0.6116,  0.6457,  0.7928]])\n",
      "Bias1 => Parameter containing:\n",
      "tensor([[ 0.9962,  0.6279,  0.1134,  0.6034]])\n",
      "Weights2 => Parameter containing:\n",
      "tensor([[ 0.7390,  0.6564],\n",
      "        [ 0.2585,  0.0130],\n",
      "        [ 0.3158,  0.5311],\n",
      "        [ 0.8083,  0.1721]])\n",
      "Bias2 => Parameter containing:\n",
      "tensor([[ 0.4459,  0.6267]])\n"
     ]
    }
   ],
   "source": [
    "middle = 4\n",
    "\n",
    "weights1 = torch.nn.Parameter(torch.rand(2, middle))\n",
    "print(\"Weights1 => \"+str(weights1))\n",
    "\n",
    "bias1 = torch.nn.Parameter(torch.rand(1, middle))\n",
    "print(\"Bias1 => \"+str(bias1))\n",
    "\n",
    "weights2 = torch.nn.Parameter(torch.rand(middle, 2))\n",
    "print(\"Weights2 => \"+str(weights2))\n",
    "\n",
    "bias2 = torch.nn.Parameter(torch.rand(1,2))\n",
    "print(\"Bias2 => \"+str(bias2))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now perform 1000 learning iterations below as many times as we want.\n",
    "\n",
    "Notice that the code for the learning iterations is almost identical to that of exercise 630 but that we've used the Adam optimizer class in Pytorch to nudge the weights in the direction they must go."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The loss is now 0.722926914691925\n",
      "The loss is now 0.6985739469528198\n",
      "The loss is now 0.6916922330856323\n",
      "The loss is now 0.6909358501434326\n",
      "The loss is now 0.6848089694976807\n",
      "The loss is now 0.68853360414505\n",
      "The loss is now 0.688930094242096\n",
      "The loss is now 0.6906450986862183\n",
      "The loss is now 0.6858261823654175\n",
      "The loss is now 0.6870893836021423\n",
      "The loss is now 0.6844713687896729\n",
      "The loss is now 0.686631977558136\n",
      "The loss is now 0.6817602515220642\n",
      "The loss is now 0.6831540465354919\n",
      "The loss is now 0.6796778440475464\n",
      "The loss is now 0.6721691489219666\n",
      "The loss is now 0.6748241782188416\n",
      "The loss is now 0.671863853931427\n",
      "The loss is now 0.6672892570495605\n",
      "The loss is now 0.6181325316429138\n",
      "The loss is now 0.611628532409668\n",
      "The loss is now 0.5935599207878113\n",
      "The loss is now 0.5560938715934753\n",
      "The loss is now 0.5563618540763855\n",
      "The loss is now 0.5367292165756226\n",
      "The loss is now 0.5396347641944885\n",
      "The loss is now 0.5033348798751831\n",
      "The loss is now 0.5050330758094788\n",
      "The loss is now 0.4595966041088104\n",
      "The loss is now 0.4339649975299835\n",
      "The loss is now 0.41711705923080444\n",
      "The loss is now 0.42426782846450806\n",
      "The loss is now 0.4179445505142212\n",
      "The loss is now 0.41568639874458313\n",
      "The loss is now 0.3851049840450287\n",
      "The loss is now 0.3701460063457489\n",
      "The loss is now 0.36989614367485046\n",
      "The loss is now 0.37163078784942627\n",
      "The loss is now 0.39095717668533325\n",
      "The loss is now 0.3497507870197296\n",
      "The loss is now 0.3818574547767639\n",
      "The loss is now 0.4014522433280945\n",
      "The loss is now 0.3695647120475769\n",
      "The loss is now 0.3947582542896271\n",
      "The loss is now 0.35953158140182495\n",
      "The loss is now 0.38059160113334656\n",
      "The loss is now 0.3930833041667938\n",
      "The loss is now 0.3720582127571106\n",
      "The loss is now 0.38750025629997253\n",
      "The loss is now 0.376242071390152\n",
      "The loss is now 0.3627578318119049\n",
      "The loss is now 0.3835773766040802\n",
      "The loss is now 0.38608214259147644\n",
      "The loss is now 0.3688598573207855\n",
      "The loss is now 0.38793662190437317\n",
      "The loss is now 0.34678754210472107\n",
      "The loss is now 0.35592278838157654\n",
      "The loss is now 0.36950254440307617\n",
      "The loss is now 0.35388681292533875\n",
      "The loss is now 0.34963470697402954\n",
      "The loss is now 0.35528966784477234\n",
      "The loss is now 0.3566282391548157\n",
      "The loss is now 0.3409440815448761\n",
      "The loss is now 0.35655948519706726\n",
      "The loss is now 0.37629398703575134\n",
      "The loss is now 0.3598496913909912\n",
      "The loss is now 0.3501574993133545\n",
      "The loss is now 0.34937459230422974\n",
      "The loss is now 0.4028155207633972\n",
      "The loss is now 0.35994064807891846\n",
      "The loss is now 0.35114824771881104\n",
      "The loss is now 0.3286530673503876\n",
      "The loss is now 0.3272094130516052\n",
      "The loss is now 0.385477215051651\n",
      "The loss is now 0.37465521693229675\n",
      "The loss is now 0.35068655014038086\n",
      "The loss is now 0.34184110164642334\n",
      "The loss is now 0.3608826696872711\n",
      "The loss is now 0.37883371114730835\n",
      "The loss is now 0.33747008442878723\n",
      "The loss is now 0.3785701096057892\n",
      "The loss is now 0.3841695785522461\n",
      "The loss is now 0.3459905982017517\n",
      "The loss is now 0.34304115176200867\n",
      "The loss is now 0.3229943811893463\n",
      "The loss is now 0.36598002910614014\n",
      "The loss is now 0.3406476080417633\n",
      "The loss is now 0.38722339272499084\n",
      "The loss is now 0.3625413179397583\n",
      "The loss is now 0.35707059502601624\n",
      "The loss is now 0.3799459636211395\n",
      "The loss is now 0.34609249234199524\n",
      "The loss is now 0.3272170126438141\n",
      "The loss is now 0.3429931700229645\n",
      "The loss is now 0.3719436824321747\n",
      "The loss is now 0.3405810594558716\n",
      "The loss is now 0.34076422452926636\n",
      "The loss is now 0.3626115024089813\n",
      "The loss is now 0.3339625298976898\n",
      "The loss is now 0.3658903241157532\n",
      "The loss is now 0.37277689576148987\n"
     ]
    }
   ],
   "source": [
    "optimizer = torch.optim.Adam([weights1, weights2, bias1, bias2], lr=0.01)\n",
    "\n",
    "for i in range(1001):\n",
    "    optimizer.zero_grad()   # zero the gradient buffers\n",
    "    \n",
    "    labels, features = data.get_sample(1000)\n",
    "    \n",
    "    features = torch.autograd.Variable(torch.Tensor(features))\n",
    "    #print(\"Features: \"+str(features))\n",
    "    \n",
    "    target = torch.autograd.Variable(torch.LongTensor(labels))\n",
    "    #print(\"Target: \"+str(target))\n",
    "    \n",
    "    result = features.mm(weights1) + bias1\n",
    "    result1 = F.sigmoid(result)\n",
    "    result2 = result1.mm(weights2) + bias2\n",
    "    \n",
    "    loss = F.cross_entropy(result2, target)\n",
    "    #print(\"Cross entropy loss: \"+str(loss))\n",
    "\n",
    "    loss.backward()\n",
    "    \n",
    "    optimizer.step()\n",
    "        \n",
    "    if i % 10 == 0:\n",
    "        print(\"The loss is now \"+str(loss.data.item()))\n",
    "\n",
    "torch.save(weights1, \"models/toy_problem_3_trained_deep_model_weights1.bin\")\n",
    "torch.save(weights2, \"models/toy_problem_3_trained_deep_model_weights2.bin\")\n",
    "torch.save(bias1, \"models/toy_problem_3_trained_deep_model_bias1.bin\")\n",
    "torch.save(bias2, \"models/toy_problem_3_trained_deep_model_bias2.bin\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## The Loss\n",
    "\n",
    "Observe the loss that is printed at the end of every 10 iterations.\n",
    "\n",
    "The loss does decrease this time.\n",
    "\n",
    "This tells us that the machine learning algorithm is learning.\n",
    "\n",
    "## Parameters\n",
    "\n",
    "We can now print the weights and the biases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The first layer weights are now tensor([[ 0.0815,  0.0523,  0.0835,  0.2602],\n",
      "        [ 0.0701,  0.0548,  0.4841,  0.7988]])\n",
      "and the second layer's weights are now tensor([[-1.9810,  3.3764],\n",
      "        [ 3.3219, -3.0504],\n",
      "        [-1.2035,  2.0503],\n",
      "        [ 2.1574, -1.1770]])\n",
      "The first layer bias is now tensor([[ 4.7902, -4.3313,  5.1664, -5.1487]])\n",
      "and the second layer's bias is now tensor([[ 2.1386, -1.0660]])\n"
     ]
    }
   ],
   "source": [
    "print(\"The first layer weights are now \"+str(weights1.data))\n",
    "print(\"and the second layer's weights are now \"+str(weights2.data))\n",
    "print(\"The first layer bias is now \"+str(bias1.data))\n",
    "print(\"and the second layer's bias is now \"+str(bias2.data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifier Test - Toy Problem 3\n",
    "\n",
    "We have just trained a multilayer classifier for Toy Problem 3.\n",
    "\n",
    "It seems to be learning (the loss on the training data decreases).\n",
    "\n",
    "Let us evaluate the performance of the classifier on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.0815,  0.0523,  0.0835,  0.2602],\n",
      "        [ 0.0701,  0.0548,  0.4841,  0.7988]])\n",
      "tensor([[-1.9810,  3.3764],\n",
      "        [ 3.3219, -3.0504],\n",
      "        [-1.2035,  2.0503],\n",
      "        [ 2.1574, -1.1770]])\n",
      "tensor([[ 4.7902, -4.3313,  5.1664, -5.1487]])\n",
      "tensor([[ 2.1386, -1.0660]])\n",
      "Accuracy: 0.843\n"
     ]
    }
   ],
   "source": [
    "data = Data(\"data/toy_problem_3_test.txt\")\n",
    "\n",
    "weights1 = torch.load(\"models/toy_problem_3_trained_deep_model_weights1.bin\")\n",
    "print(weights1)\n",
    "weights2 = torch.load(\"models/toy_problem_3_trained_deep_model_weights2.bin\")\n",
    "print(weights2)\n",
    "bias1 = torch.load(\"models/toy_problem_3_trained_deep_model_bias1.bin\")\n",
    "print(bias1)\n",
    "bias2 = torch.load(\"models/toy_problem_3_trained_deep_model_bias2.bin\")\n",
    "print(bias2)\n",
    "\n",
    "labels, features = data.get_all()\n",
    "\n",
    "features = torch.autograd.Variable(torch.Tensor(features))\n",
    "#print(features)\n",
    "\n",
    "target = torch.autograd.Variable(torch.LongTensor(labels))\n",
    "#print(target)\n",
    "\n",
    "result = torch.mm(features, weights1) + bias1\n",
    "result1 = F.sigmoid(result)\n",
    "result2 = torch.mm(result1, weights2) + bias2\n",
    "#print(result2)\n",
    "\n",
    "maxv, observed = torch.max(result2, 1)\n",
    "\n",
    "total = 0\n",
    "correct = 0\n",
    "for i in range(len(labels)):\n",
    "    total += 1\n",
    "    #print(str(target.data[i]) + \" \" + str(observed.data[i]))\n",
    "    if target.data[i] == observed.data[i]:\n",
    "        correct += 1\n",
    "accuracy = correct / total\n",
    "print(\"Accuracy: \"+str(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "As you can see, the accuracy is better than 50%.\n",
    "\n",
    "The classifier has fared better than before when using bias terms.\n",
    "\n",
    "It tells us that the multi-layer neural network (with a bias term) **is able to learn the non-linear XOR function using the sigmoid activation function**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}