lab-02_linear_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 2: Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: Seungjae Lee (이승재)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    We use elemental PyTorch to implement linear regression here. However, in most actual applications, abstractions such as <code>nn.Module</code> or <code>nn.Linear</code> are used.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Theoretical Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ H(x) = Wx + b $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ cost(W, b) = \\frac{1}{m} \\sum^m_{i=1} \\left( H(x^{(i)}) - y^{(i)} \\right)^2 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - $H(x)$: 주어진 $x$ 값에 대해 예측을 어떻게 할 것인가\n",
    " - $cost(W, b)$: $H(x)$ 가 $y$ 를 얼마나 잘 예측했는가"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7fa29837ef90>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For reproducibility\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use fake data for this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = torch.FloatTensor([[1], [2], [3]])\n",
    "y_train = torch.FloatTensor([[1], [2], [3]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1.],\n",
      "        [2.],\n",
      "        [3.]])\n",
      "torch.Size([3, 1])\n"
     ]
    }
   ],
   "source": [
    "print(x_train)\n",
    "print(x_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1.],\n",
      "        [2.],\n",
      "        [3.]])\n",
      "torch.Size([3, 1])\n"
     ]
    }
   ],
   "source": [
    "print(y_train)\n",
    "print(y_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "기본적으로 PyTorch는 NCHW 형태이다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Weight Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "W = torch.zeros(1, requires_grad=True)\n",
    "print(W)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "b = torch.zeros(1, requires_grad=True)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hypothesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ H(x) = Wx + b $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.],\n",
      "        [0.],\n",
      "        [0.]], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "hypothesis = x_train * W + b\n",
    "print(hypothesis)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ cost(W, b) = \\frac{1}{m} \\sum^m_{i=1} \\left( H(x^{(i)}) - y^{(i)} \\right)^2 $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.],\n",
      "        [0.],\n",
      "        [0.]], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "print(hypothesis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1.],\n",
      "        [2.],\n",
      "        [3.]])\n"
     ]
    }
   ],
   "source": [
    "print(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-1.],\n",
      "        [-2.],\n",
      "        [-3.]], grad_fn=<SubBackward0>)\n"
     ]
    }
   ],
   "source": [
    "print(hypothesis - y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1.],\n",
      "        [4.],\n",
      "        [9.]], grad_fn=<PowBackward0>)\n"
     ]
    }
   ],
   "source": [
    "print((hypothesis - y_train) ** 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(4.6667, grad_fn=<MeanBackward1>)\n"
     ]
    }
   ],
   "source": [
    "cost = torch.mean((hypothesis - y_train) ** 2)\n",
    "print(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = optim.SGD([W, b], lr=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer.zero_grad()\n",
    "cost.backward()\n",
    "optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([0.0933], requires_grad=True)\n",
      "tensor([0.0400], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "print(W)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check if the hypothesis is now better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.1333],\n",
      "        [0.2267],\n",
      "        [0.3200]], grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "hypothesis = x_train * W + b\n",
    "print(hypothesis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(3.6927, grad_fn=<MeanBackward1>)\n"
     ]
    }
   ],
   "source": [
    "cost = torch.mean((hypothesis - y_train) ** 2)\n",
    "print(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with Full Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In reality, we will be training on the dataset for multiple epochs. This can be done simply with loops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 W: 0.093, b: 0.040 Cost: 4.666667\n",
      "Epoch  100/1000 W: 0.873, b: 0.289 Cost: 0.012043\n",
      "Epoch  200/1000 W: 0.900, b: 0.227 Cost: 0.007442\n",
      "Epoch  300/1000 W: 0.921, b: 0.179 Cost: 0.004598\n",
      "Epoch  400/1000 W: 0.938, b: 0.140 Cost: 0.002842\n",
      "Epoch  500/1000 W: 0.951, b: 0.110 Cost: 0.001756\n",
      "Epoch  600/1000 W: 0.962, b: 0.087 Cost: 0.001085\n",
      "Epoch  700/1000 W: 0.970, b: 0.068 Cost: 0.000670\n",
      "Epoch  800/1000 W: 0.976, b: 0.054 Cost: 0.000414\n",
      "Epoch  900/1000 W: 0.981, b: 0.042 Cost: 0.000256\n",
      "Epoch 1000/1000 W: 0.985, b: 0.033 Cost: 0.000158\n"
     ]
    }
   ],
   "source": [
    "# 데이터\n",
    "x_train = torch.FloatTensor([[1], [2], [3]])\n",
    "y_train = torch.FloatTensor([[1], [2], [3]])\n",
    "# 모델 초기화\n",
    "W = torch.zeros(1, requires_grad=True)\n",
    "b = torch.zeros(1, requires_grad=True)\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W, b], lr=0.01)\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "    \n",
    "    # H(x) 계산\n",
    "    hypothesis = x_train * W + b\n",
    "    \n",
    "    # cost 계산\n",
    "    cost = torch.mean((hypothesis - y_train) ** 2)\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 100번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} W: {:.3f}, b: {:.3f} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, W.item(), b.item(), cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## High-level Implementation with `nn.Module`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that we had this fake data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = torch.FloatTensor([[1], [2], [3]])\n",
    "y_train = torch.FloatTensor([[1], [2], [3]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 linear regression 모델을 만들면 되는데, 기본적으로 PyTorch의 모든 모델은 제공되는 `nn.Module`을 inherit 해서 만들게 됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LinearRegressionModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.linear = nn.Linear(1, 1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.linear(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "모델의 `__init__`에서는 사용할 레이어들을 정의하게 됩니다. 여기서 우리는 linear regression 모델을 만들기 때문에, `nn.Linear` 를 이용할 것입니다. 그리고 `forward`에서는 이 모델이 어떻게 입력값에서 출력값을 계산하는지 알려줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LinearRegressionModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hypothesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 모델을 생성해서 예측값 $H(x)$를 구해보자"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "hypothesis = model(x_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.0739],\n",
      "        [0.5891],\n",
      "        [1.1044]], grad_fn=<ThAddmmBackward>)\n"
     ]
    }
   ],
   "source": [
    "print(hypothesis)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 mean squared error (MSE) 로 cost를 구해보자. MSE 역시 PyTorch에서 기본적으로 제공한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.0739],\n",
      "        [0.5891],\n",
      "        [1.1044]], grad_fn=<ThAddmmBackward>)\n",
      "tensor([[1.],\n",
      "        [2.],\n",
      "        [3.]])\n"
     ]
    }
   ],
   "source": [
    "print(hypothesis)\n",
    "print(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "cost = F.mse_loss(hypothesis, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(2.1471, grad_fn=<MseLossBackward>)\n"
     ]
    }
   ],
   "source": [
    "print(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "마지막 주어진 cost를 이용해 $H(x)$ 의 $W, b$ 를 바꾸어서 cost를 줄여봅니다. 이때 PyTorch의 `torch.optim` 에 있는 `optimizer` 들 중 하나를 사용할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = optim.SGD(model.parameters(), lr=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer.zero_grad()\n",
    "cost.backward()\n",
    "optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training with Full Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 Linear Regression 코드를 이해했으니, 실제로 코드를 돌려 피팅시켜보겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 W: -0.101, b: 0.508 Cost: 4.630286\n",
      "Epoch  100/1000 W: 0.713, b: 0.653 Cost: 0.061555\n",
      "Epoch  200/1000 W: 0.774, b: 0.514 Cost: 0.038037\n",
      "Epoch  300/1000 W: 0.822, b: 0.404 Cost: 0.023505\n",
      "Epoch  400/1000 W: 0.860, b: 0.317 Cost: 0.014525\n",
      "Epoch  500/1000 W: 0.890, b: 0.250 Cost: 0.008975\n",
      "Epoch  600/1000 W: 0.914, b: 0.196 Cost: 0.005546\n",
      "Epoch  700/1000 W: 0.932, b: 0.154 Cost: 0.003427\n",
      "Epoch  800/1000 W: 0.947, b: 0.121 Cost: 0.002118\n",
      "Epoch  900/1000 W: 0.958, b: 0.095 Cost: 0.001309\n",
      "Epoch 1000/1000 W: 0.967, b: 0.075 Cost: 0.000809\n"
     ]
    }
   ],
   "source": [
    "# 데이터\n",
    "x_train = torch.FloatTensor([[1], [2], [3]])\n",
    "y_train = torch.FloatTensor([[1], [2], [3]])\n",
    "# 모델 초기화\n",
    "model = LinearRegressionModel()\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01)\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "    \n",
    "    # H(x) 계산\n",
    "    prediction = model(x_train)\n",
    "    \n",
    "    # cost 계산\n",
    "    cost = F.mse_loss(prediction, y_train)\n",
    "    \n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad()\n",
    "    cost.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    # 100번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        params = list(model.parameters())\n",
    "        W = params[0].item()\n",
    "        b = params[1].item()\n",
    "        print('Epoch {:4d}/{} W: {:.3f}, b: {:.3f} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, W, b, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "점점 $H(x)$ 의 $W$ 와 $b$ 를 조정해서 cost가 줄어드는 것을 볼 수 있습니다."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}