lab-04_1_multivariable_linear_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 4-1: Multivariate Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: Seungjae Lee (이승재)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    We use elemental PyTorch to implement linear regression here. However, in most actual applications, abstractions such as <code>nn.Module</code> or <code>nn.Linear</code> are used.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Theoretical Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ H(x_1, x_2, x_3) = x_1w_1 + x_2w_2 + x_3w_3 + b $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ cost(W, b) = \\frac{1}{m} \\sum^m_{i=1} \\left( H(x^{(i)}) - y^{(i)} \\right)^2 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - $H(x)$: 주어진 $x$ 값에 대해 예측을 어떻게 할 것인가\n",
    " - $cost(W, b)$: $H(x)$ 가 $y$ 를 얼마나 잘 예측했는가"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7fd26ff8afb0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For reproducibility\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Data Representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use fake data for this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 데이터\n",
    "x1_train = torch.FloatTensor([[73], [93], [89], [96], [73]])\n",
    "x2_train = torch.FloatTensor([[80], [88], [91], [98], [66]])\n",
    "x3_train = torch.FloatTensor([[75], [93], [90], [100], [70]])\n",
    "y_train = torch.FloatTensor([[152], [185], [180], [196], [142]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 w1: 0.294 w2: 0.297 w3: 0.297 b: 0.003 Cost: 29661.800781\n",
      "Epoch  100/1000 w1: 0.674 w2: 0.676 w3: 0.676 b: 0.008 Cost: 1.563634\n",
      "Epoch  200/1000 w1: 0.679 w2: 0.677 w3: 0.677 b: 0.008 Cost: 1.497603\n",
      "Epoch  300/1000 w1: 0.684 w2: 0.677 w3: 0.677 b: 0.008 Cost: 1.435026\n",
      "Epoch  400/1000 w1: 0.689 w2: 0.678 w3: 0.678 b: 0.008 Cost: 1.375730\n",
      "Epoch  500/1000 w1: 0.694 w2: 0.678 w3: 0.678 b: 0.009 Cost: 1.319503\n",
      "Epoch  600/1000 w1: 0.699 w2: 0.679 w3: 0.679 b: 0.009 Cost: 1.266215\n",
      "Epoch  700/1000 w1: 0.704 w2: 0.679 w3: 0.679 b: 0.009 Cost: 1.215693\n",
      "Epoch  800/1000 w1: 0.709 w2: 0.679 w3: 0.679 b: 0.009 Cost: 1.167821\n",
      "Epoch  900/1000 w1: 0.713 w2: 0.680 w3: 0.680 b: 0.009 Cost: 1.122419\n",
      "Epoch 1000/1000 w1: 0.718 w2: 0.680 w3: 0.680 b: 0.009 Cost: 1.079375\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "w1 = torch.zeros(1, requires_grad=True)\n",
    "w2 = torch.zeros(1, requires_grad=True)\n",
    "w3 = torch.zeros(1, requires_grad=True)\n",
    "b = torch.zeros(1, requires_grad=True)\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([w1, w2, w3, b], lr=1e-5)\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "    \n",
    "    # H(x) 계산\n",
    "    hypothesis = x1_train * w1 + x2_train * w2 + x3_train * w3 + b\n",
    "\n",
    "    # cost 계산\n",
    "    cost = torch.mean((hypothesis - y_train) ** 2)\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 100번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} w1: {:.3f} w2: {:.3f} w3: {:.3f} b: {:.3f} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, w1.item(), w3.item(), w3.item(), b.item(), cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix Data Representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\begin{pmatrix}\n",
    "x_1 & x_2 & x_3\n",
    "\\end{pmatrix}\n",
    "\\cdot\n",
    "\\begin{pmatrix}\n",
    "w_1 \\\\\n",
    "w_2 \\\\\n",
    "w_3 \\\\\n",
    "\\end{pmatrix}\n",
    "=\n",
    "\\begin{pmatrix}\n",
    "x_1w_1 + x_2w_2 + x_3w_3\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ H(X) = XW $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = torch.FloatTensor([[73, 80, 75],\n",
    "                             [93, 88, 93],\n",
    "                             [89, 91, 90],\n",
    "                             [96, 98, 100],\n",
    "                             [73, 66, 70]])\n",
    "y_train = torch.FloatTensor([[152], [185], [180], [196], [142]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([5, 3])\n",
      "torch.Size([5, 1])\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape)\n",
    "print(y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/20 hypothesis: tensor([0., 0., 0., 0., 0.]) Cost: 29661.800781\n",
      "Epoch    1/20 hypothesis: tensor([67.2578, 80.8397, 79.6523, 86.7394, 61.6605]) Cost: 9298.520508\n",
      "Epoch    2/20 hypothesis: tensor([104.9128, 126.0990, 124.2466, 135.3015,  96.1821]) Cost: 2915.713135\n",
      "Epoch    3/20 hypothesis: tensor([125.9942, 151.4381, 149.2133, 162.4896, 115.5097]) Cost: 915.040527\n",
      "Epoch    4/20 hypothesis: tensor([137.7968, 165.6247, 163.1911, 177.7112, 126.3307]) Cost: 287.936005\n",
      "Epoch    5/20 hypothesis: tensor([144.4044, 173.5674, 171.0168, 186.2332, 132.3891]) Cost: 91.371017\n",
      "Epoch    6/20 hypothesis: tensor([148.1035, 178.0144, 175.3980, 191.0042, 135.7812]) Cost: 29.758139\n",
      "Epoch    7/20 hypothesis: tensor([150.1744, 180.5042, 177.8508, 193.6753, 137.6805]) Cost: 10.445305\n",
      "Epoch    8/20 hypothesis: tensor([151.3336, 181.8983, 179.2240, 195.1707, 138.7440]) Cost: 4.391228\n",
      "Epoch    9/20 hypothesis: tensor([151.9824, 182.6789, 179.9928, 196.0079, 139.3396]) Cost: 2.493135\n",
      "Epoch   10/20 hypothesis: tensor([152.3454, 183.1161, 180.4231, 196.4765, 139.6732]) Cost: 1.897688\n",
      "Epoch   11/20 hypothesis: tensor([152.5485, 183.3610, 180.6640, 196.7389, 139.8602]) Cost: 1.710541\n",
      "Epoch   12/20 hypothesis: tensor([152.6620, 183.4982, 180.7988, 196.8857, 139.9651]) Cost: 1.651413\n",
      "Epoch   13/20 hypothesis: tensor([152.7253, 183.5752, 180.8742, 196.9678, 140.0240]) Cost: 1.632387\n",
      "Epoch   14/20 hypothesis: tensor([152.7606, 183.6184, 180.9164, 197.0138, 140.0571]) Cost: 1.625923\n",
      "Epoch   15/20 hypothesis: tensor([152.7802, 183.6427, 180.9399, 197.0395, 140.0759]) Cost: 1.623412\n",
      "Epoch   16/20 hypothesis: tensor([152.7909, 183.6565, 180.9530, 197.0538, 140.0865]) Cost: 1.622141\n",
      "Epoch   17/20 hypothesis: tensor([152.7968, 183.6643, 180.9603, 197.0618, 140.0927]) Cost: 1.621253\n",
      "Epoch   18/20 hypothesis: tensor([152.7999, 183.6688, 180.9644, 197.0662, 140.0963]) Cost: 1.620500\n",
      "Epoch   19/20 hypothesis: tensor([152.8014, 183.6715, 180.9666, 197.0686, 140.0985]) Cost: 1.619770\n",
      "Epoch   20/20 hypothesis: tensor([152.8020, 183.6731, 180.9677, 197.0699, 140.1000]) Cost: 1.619033\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((3, 1), requires_grad=True)\n",
    "b = torch.zeros(1, requires_grad=True)\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W, b], lr=1e-5)\n",
    "\n",
    "nb_epochs = 20\n",
    "for epoch in range(nb_epochs + 1):\n",
    "    \n",
    "    # H(x) 계산\n",
    "    hypothesis = x_train.matmul(W) + b # or .mm or @\n",
    "\n",
    "    # cost 계산\n",
    "    cost = torch.mean((hypothesis - y_train) ** 2)\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 100번마다 로그 출력\n",
    "    print('Epoch {:4d}/{} hypothesis: {} Cost: {:.6f}'.format(\n",
    "        epoch, nb_epochs, hypothesis.squeeze().detach(), cost.item()\n",
    "    ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## High-level Implementation with `nn.Module`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you remember this model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LinearRegressionModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.linear = nn.Linear(1, 1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.linear(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We just need to change the input dimension from 1 to 3!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultivariateLinearRegressionModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.linear = nn.Linear(3, 1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.linear(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/20 Cost: 31667.597656\n",
      "Epoch    1/20 Cost: 9926.266602\n",
      "Epoch    2/20 Cost: 3111.513672\n",
      "Epoch    3/20 Cost: 975.451355\n",
      "Epoch    4/20 Cost: 305.908539\n",
      "Epoch    5/20 Cost: 96.042488\n",
      "Epoch    6/20 Cost: 30.260750\n",
      "Epoch    7/20 Cost: 9.641701\n",
      "Epoch    8/20 Cost: 3.178671\n",
      "Epoch    9/20 Cost: 1.152871\n",
      "Epoch   10/20 Cost: 0.517863\n",
      "Epoch   11/20 Cost: 0.318801\n",
      "Epoch   12/20 Cost: 0.256388\n",
      "Epoch   13/20 Cost: 0.236821\n",
      "Epoch   14/20 Cost: 0.230660\n",
      "Epoch   15/20 Cost: 0.228719\n",
      "Epoch   16/20 Cost: 0.228095\n",
      "Epoch   17/20 Cost: 0.227880\n",
      "Epoch   18/20 Cost: 0.227799\n",
      "Epoch   19/20 Cost: 0.227762\n",
      "Epoch   20/20 Cost: 0.227732\n"
     ]
    }
   ],
   "source": [
    "# 데이터\n",
    "x_train = torch.FloatTensor([[73, 80, 75],\n",
    "                             [93, 88, 93],\n",
    "                             [89, 91, 90],\n",
    "                             [96, 98, 100],\n",
    "                             [73, 66, 70]])\n",
    "y_train = torch.FloatTensor([[152], [185], [180], [196], [142]])\n",
    "# 모델 초기화\n",
    "model = MultivariateLinearRegressionModel()\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD(model.parameters(), lr=1e-5)\n",
    "\n",
    "nb_epochs = 20\n",
    "for epoch in range(nb_epochs+1):\n",
    "    \n",
    "    # H(x) 계산\n",
    "    prediction = model(x_train)\n",
    "    \n",
    "    # cost 계산\n",
    "    cost = F.mse_loss(prediction, y_train)\n",
    "    \n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    # 20번마다 로그 출력\n",
    "    print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "        epoch, nb_epochs, cost.item()\n",
    "    ))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}