lab-05_logistic_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 5: Logistic Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: Seungjae Lee (이승재)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    We use elemental PyTorch to implement linear regression here. However, in most actual applications, abstractions such as <code>nn.Module</code> or <code>nn.Linear</code> are used. You can see those implementations near the end of this notebook.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reminder: Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hypothesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ H(X) = \\frac{1}{1+e^{-W^T X}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ cost(W) = -\\frac{1}{m} \\sum y \\log\\left(H(x)\\right) + (1-y) \\left( \\log(1-H(x) \\right) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - If $y \\simeq H(x)$, cost is near 0.\n",
    " - If $y \\neq H(x)$, cost is high."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weight Update via Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ W := W - \\alpha \\frac{\\partial}{\\partial W} cost(W) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - $\\alpha$: Learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x106951ed0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For reproducibility\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_data = [[1, 2], [2, 3], [3, 1], [4, 3], [5, 3], [6, 2]]\n",
    "y_data = [[0], [0], [0], [1], [1], [1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the following classification problem: given the number of hours each student spent watching the lecture and working in the code lab, predict whether the student passed or failed a course. For example, the first (index 0) student watched the lecture for 1 hour and spent 2 hours in the lab session ([1, 2]), and ended up failing the course ([0])."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = torch.FloatTensor(x_data)\n",
    "y_train = torch.FloatTensor(y_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As always, we need these data to be in `torch.Tensor` format, so we convert them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 2])\n",
      "torch.Size([6, 1])\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape)\n",
    "print(y_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing the Hypothesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ H(X) = \\frac{1}{1+e^{-W^T X}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch has a `torch.exp()` function that resembles the exponential function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "e^1 equals:  tensor([2.7183])\n"
     ]
    }
   ],
   "source": [
    "print('e^1 equals: ', torch.exp(torch.FloatTensor([1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use it to compute the hypothesis function conveniently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "W = torch.zeros((2, 1), requires_grad=True)\n",
    "b = torch.zeros(1, requires_grad=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "hypothesis = 1 / (1 + torch.exp(-(x_train.matmul(W) + b)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000]], grad_fn=<MulBackward>)\n",
      "torch.Size([6, 1])\n"
     ]
    }
   ],
   "source": [
    "print(hypothesis)\n",
    "print(hypothesis.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, we could use `torch.sigmoid()` function! This resembles the sigmoid function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/(1+e^{-1}) equals:  tensor([0.7311])\n"
     ]
    }
   ],
   "source": [
    "print('1/(1+e^{-1}) equals: ', torch.sigmoid(torch.FloatTensor([1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the code for hypothesis function is cleaner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "hypothesis = torch.sigmoid(x_train.matmul(W) + b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000]], grad_fn=<SigmoidBackward>)\n",
      "torch.Size([6, 1])\n"
     ]
    }
   ],
   "source": [
    "print(hypothesis)\n",
    "print(hypothesis.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing the Cost Function (Low-level)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ cost(W) = -\\frac{1}{m} \\sum y \\log\\left(H(x)\\right) + (1-y) \\left( \\log(1-H(x) \\right) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to measure the difference between `hypothesis` and `y_train`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000],\n",
      "        [0.5000]], grad_fn=<SigmoidBackward>)\n",
      "tensor([[0.],\n",
      "        [0.],\n",
      "        [0.],\n",
      "        [1.],\n",
      "        [1.],\n",
      "        [1.]])\n"
     ]
    }
   ],
   "source": [
    "print(hypothesis)\n",
    "print(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For one element, the loss can be computed as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.6931], grad_fn=<NegBackward>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "-(y_train[0] * torch.log(hypothesis[0]) + \n",
    "  (1 - y_train[0]) * torch.log(1 - hypothesis[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute the losses for the entire batch, we can simply input the entire vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.6931],\n",
      "        [0.6931],\n",
      "        [0.6931],\n",
      "        [0.6931],\n",
      "        [0.6931],\n",
      "        [0.6931]], grad_fn=<NegBackward>)\n"
     ]
    }
   ],
   "source": [
    "losses = -(y_train * torch.log(hypothesis) + \n",
    "           (1 - y_train) * torch.log(1 - hypothesis))\n",
    "print(losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we just `.mean()` to take the mean of these individual losses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.6931, grad_fn=<MeanBackward1>)\n"
     ]
    }
   ],
   "source": [
    "cost = losses.mean()\n",
    "print(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing the Cost Function with `F.binary_cross_entropy`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In reality, binary classification is used so often that PyTorch has a simple function called `F.binary_cross_entropy` implemented to lighten the burden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.6931, grad_fn=<BinaryCrossEntropyBackward>)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "F.binary_cross_entropy(hypothesis, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with Low-level Binary Cross Entropy Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_data = [[1, 2], [2, 3], [3, 1], [4, 3], [5, 3], [6, 2]]\n",
    "y_data = [[0], [0], [0], [1], [1], [1]]\n",
    "x_train = torch.FloatTensor(x_data)\n",
    "y_train = torch.FloatTensor(y_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 Cost: 0.693147\n",
      "Epoch  100/1000 Cost: 0.134722\n",
      "Epoch  200/1000 Cost: 0.080643\n",
      "Epoch  300/1000 Cost: 0.057900\n",
      "Epoch  400/1000 Cost: 0.045300\n",
      "Epoch  500/1000 Cost: 0.037261\n",
      "Epoch  600/1000 Cost: 0.031673\n",
      "Epoch  700/1000 Cost: 0.027556\n",
      "Epoch  800/1000 Cost: 0.024394\n",
      "Epoch  900/1000 Cost: 0.021888\n",
      "Epoch 1000/1000 Cost: 0.019852\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((2, 1), requires_grad=True)\n",
    "b = torch.zeros(1, requires_grad=True)\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W, b], lr=1)\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # Cost 계산\n",
    "    hypothesis = torch.sigmoid(x_train.matmul(W) + b) # or .mm or @\n",
    "    cost = -(y_train * torch.log(hypothesis) + \n",
    "             (1 - y_train) * torch.log(1 - hypothesis)).mean()\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 100번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with `F.binary_cross_entropy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 Cost: 0.693147\n",
      "Epoch  100/1000 Cost: 0.134722\n",
      "Epoch  200/1000 Cost: 0.080643\n",
      "Epoch  300/1000 Cost: 0.057900\n",
      "Epoch  400/1000 Cost: 0.045300\n",
      "Epoch  500/1000 Cost: 0.037261\n",
      "Epoch  600/1000 Cost: 0.031672\n",
      "Epoch  700/1000 Cost: 0.027556\n",
      "Epoch  800/1000 Cost: 0.024394\n",
      "Epoch  900/1000 Cost: 0.021888\n",
      "Epoch 1000/1000 Cost: 0.019852\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((2, 1), requires_grad=True)\n",
    "b = torch.zeros(1, requires_grad=True)\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W, b], lr=1)\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # Cost 계산\n",
    "    hypothesis = torch.sigmoid(x_train.matmul(W) + b) # or .mm or @\n",
    "    cost = F.binary_cross_entropy(hypothesis, y_train)\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 100번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Real Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "xy = np.loadtxt('data-03-diabetes.csv', delimiter=',', dtype=np.float32)\n",
    "x_data = xy[:, 0:-1]\n",
    "y_data = xy[:, [-1]]\n",
    "x_train = torch.FloatTensor(x_data)\n",
    "y_train = torch.FloatTensor(y_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.2941,  0.4874,  0.1803, -0.2929,  0.0000,  0.0015, -0.5312, -0.0333],\n",
      "        [-0.8824, -0.1457,  0.0820, -0.4141,  0.0000, -0.2072, -0.7669, -0.6667],\n",
      "        [-0.0588,  0.8392,  0.0492,  0.0000,  0.0000, -0.3055, -0.4927, -0.6333],\n",
      "        [-0.8824, -0.1055,  0.0820, -0.5354, -0.7778, -0.1624, -0.9240,  0.0000],\n",
      "        [ 0.0000,  0.3769, -0.3443, -0.2929, -0.6028,  0.2846,  0.8873, -0.6000]])\n",
      "tensor([[0.],\n",
      "        [1.],\n",
      "        [0.],\n",
      "        [1.],\n",
      "        [0.]])\n"
     ]
    }
   ],
   "source": [
    "print(x_train[0:5])\n",
    "print(y_train[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with Real Data using low-level Binary Cross Entropy Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/100 Cost: 0.693148\n",
      "Epoch   10/100 Cost: 0.572727\n",
      "Epoch   20/100 Cost: 0.539493\n",
      "Epoch   30/100 Cost: 0.519708\n",
      "Epoch   40/100 Cost: 0.507066\n",
      "Epoch   50/100 Cost: 0.498539\n",
      "Epoch   60/100 Cost: 0.492549\n",
      "Epoch   70/100 Cost: 0.488209\n",
      "Epoch   80/100 Cost: 0.484985\n",
      "Epoch   90/100 Cost: 0.482543\n",
      "Epoch  100/100 Cost: 0.480661\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((8, 1), requires_grad=True)\n",
    "b = torch.zeros(1, requires_grad=True)\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W, b], lr=1)\n",
    "\n",
    "nb_epochs = 100\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # Cost 계산\n",
    "    hypothesis = torch.sigmoid(x_train.matmul(W) + b) # or .mm or @\n",
    "    cost = -(y_train * torch.log(hypothesis) + (1 - y_train) * torch.log(1 - hypothesis)).mean()\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 10번마다 로그 출력\n",
    "    if epoch % 10 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with Real Data using `F.binary_cross_entropy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/100 Cost: 0.693147\n",
      "Epoch   10/100 Cost: 0.572727\n",
      "Epoch   20/100 Cost: 0.539494\n",
      "Epoch   30/100 Cost: 0.519708\n",
      "Epoch   40/100 Cost: 0.507065\n",
      "Epoch   50/100 Cost: 0.498539\n",
      "Epoch   60/100 Cost: 0.492549\n",
      "Epoch   70/100 Cost: 0.488208\n",
      "Epoch   80/100 Cost: 0.484985\n",
      "Epoch   90/100 Cost: 0.482543\n",
      "Epoch  100/100 Cost: 0.480661\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((8, 1), requires_grad=True)\n",
    "b = torch.zeros(1, requires_grad=True)\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W, b], lr=1)\n",
    "\n",
    "nb_epochs = 100\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # Cost 계산\n",
    "    hypothesis = torch.sigmoid(x_train.matmul(W) + b) # or .mm or @\n",
    "    cost = F.binary_cross_entropy(hypothesis, y_train)\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 10번마다 로그 출력\n",
    "    if epoch % 10 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking the Accuracy our our Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we finish training the model, we want to check how well our model fits the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.4103],\n",
      "        [0.9242],\n",
      "        [0.2300],\n",
      "        [0.9411],\n",
      "        [0.1772]], grad_fn=<SliceBackward>)\n"
     ]
    }
   ],
   "source": [
    "hypothesis = torch.sigmoid(x_train.matmul(W) + b)\n",
    "print(hypothesis[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can change **hypothesis** (real number from 0 to 1) to **binary predictions** (either 0 or 1) by comparing them to 0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0]], dtype=torch.uint8)\n"
     ]
    }
   ],
   "source": [
    "prediction = hypothesis >= torch.FloatTensor([0.5])\n",
    "print(prediction[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we compare it with the correct labels `y_train`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0],\n",
      "        [1],\n",
      "        [0],\n",
      "        [1],\n",
      "        [0]], dtype=torch.uint8)\n",
      "tensor([[0.],\n",
      "        [1.],\n",
      "        [0.],\n",
      "        [1.],\n",
      "        [0.]])\n"
     ]
    }
   ],
   "source": [
    "print(prediction[:5])\n",
    "print(y_train[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1],\n",
      "        [1]], dtype=torch.uint8)\n"
     ]
    }
   ],
   "source": [
    "correct_prediction = prediction.float() == y_train\n",
    "print(correct_prediction[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can calculate the accuracy by counting the number of correct predictions and dividng by total number of predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has an accuracy of 76.68% for the training set.\n"
     ]
    }
   ],
   "source": [
    "accuracy = correct_prediction.sum().item() / len(correct_prediction)\n",
    "print('The model has an accuracy of {:2.2f}% for the training set.'.format(accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional: High-level Implementation with `nn.Module`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BinaryClassifier(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.linear = nn.Linear(8, 1)\n",
    "        self.sigmoid = nn.Sigmoid()\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.sigmoid(self.linear(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = BinaryClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/100 Cost: 0.704829 Accuracy 45.72%\n",
      "Epoch   10/100 Cost: 0.572391 Accuracy 67.59%\n",
      "Epoch   20/100 Cost: 0.539563 Accuracy 73.25%\n",
      "Epoch   30/100 Cost: 0.520042 Accuracy 75.89%\n",
      "Epoch   40/100 Cost: 0.507561 Accuracy 76.15%\n",
      "Epoch   50/100 Cost: 0.499125 Accuracy 76.42%\n",
      "Epoch   60/100 Cost: 0.493177 Accuracy 77.21%\n",
      "Epoch   70/100 Cost: 0.488846 Accuracy 76.81%\n",
      "Epoch   80/100 Cost: 0.485612 Accuracy 76.28%\n",
      "Epoch   90/100 Cost: 0.483146 Accuracy 76.55%\n",
      "Epoch  100/100 Cost: 0.481234 Accuracy 76.81%\n"
     ]
    }
   ],
   "source": [
    "# optimizer 설정\n",
    "optimizer = optim.SGD(model.parameters(), lr=1)\n",
    "\n",
    "nb_epochs = 100\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # H(x) 계산\n",
    "    hypothesis = model(x_train)\n",
    "\n",
    "    # cost 계산\n",
    "    cost = F.binary_cross_entropy(hypothesis, y_train)\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    # 20번마다 로그 출력\n",
    "    if epoch % 10 == 0:\n",
    "        prediction = hypothesis >= torch.FloatTensor([0.5])\n",
    "        correct_prediction = prediction.float() == y_train\n",
    "        accuracy = correct_prediction.sum().item() / len(correct_prediction)\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f} Accuracy {:2.2f}%'.format(\n",
    "            epoch, nb_epochs, cost.item(), accuracy * 100,\n",
    "        ))\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}