CNN_Baseline_(1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wiTQibprRnlj"
   },
   "source": [
    "# **Cifar10 image classification**\n",
    "******************************************\n",
    "**점수관련**  \n",
    "본 프로젝트는 Accuracy 점수와 보고서를 성적에 반영할 예정입니다. 평가 항목은 아래와 같습니다. \n",
    "\n",
    "A. 결과 (40%)\n",
    "- Metric 성능 :(이미지 분류 - Accuracy)\n",
    "\n",
    "B. 신규성 (30%)\n",
    "- Network 변경 내용 - **필수**\n",
    "- 성능 개선 및 overfitting 방지 시도 (ex. train validation split)\n",
    "\n",
    "C. 이론적 근거 (20%)\n",
    "- 기존 baseline과의 차별점 \n",
    "- 성능 개선을 위한 시도와 이유\n",
    "\n",
    "D. 보고서 완성도 (10%)\n",
    "\n",
    "******************************************\n",
    "**보고서**\n",
    "1. 연구 목적\n",
    "2. 모델 구조\n",
    "3. 실험 내용\n",
    "4. 실험 결과\n",
    "5. 고찰 및 결론\n",
    "6. Colab 파일 (동작 가능여부)\n",
    "\n",
    "제출하실 파일은 **주피터 노트북 파일**('.ipynb',파일>다운로드)과 **결과 보고서**(pdf)입니다.\n",
    "******************************************\n",
    "**코드 검증**  \n",
    "- 코드 성능 평가(metric)는 torchvision.datasets에 있는 cifar10의 testset을 사용해 Accuracy로 평가합니다.\n",
    "- pretrain 모델 사용 불가능합니다.\n",
    "\n",
    "또한 Random 라이브러리 사용시에 seed 고정하는 등 재현을 고려하시고 코딩부탁드립니다. \n",
    "******************************************\n",
    "**GPU 사용 법**  \n",
    "런타임 > 런타임 유형 변경 > 하드웨어 가속기에서 GPU를 선택하면 GPU를 사용 할 수 있습니다.  \n",
    "******************************************\n",
    "**Colab 사용시 유의사항**  \n",
    "12시간 단위로 가상머신을 사용할 수 있고 12시간이 지나면 모든 파일과 작업로그들이 초기화 됩니다. 또한, 12시간 넘게 GPU를 사용하기는 어려우니 일찍 시작하셔서 틈틈히 실험해보시는 것을 추천드립니다.\n",
    "\n",
    "\n",
    "******************************************\n",
    "**Q?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CTVqQzTrIwTx"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "NtweS09sMWoh"
   },
   "outputs": [],
   "source": [
    "# Training\n",
    "import torch\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "from torch.utils.data import DataLoader\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from tqdm import tqdm as tqdm\n",
    "import time\n",
    "import os\n",
    "os.environ[\"CUDA_DEVICE_ORDER\"]=\"PCI_BUS_ID\"\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NV_iVhYhJmMB",
    "outputId": "ec064561-af7d-4fac-ccc2-c8e35457eac5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thu Oct 28 23:10:41 2021       \r\n",
      "+-----------------------------------------------------------------------------+\r\n",
      "| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |\r\n",
      "|-------------------------------+----------------------+----------------------+\r\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n",
      "|                               |                      |               MIG M. |\r\n",
      "|===============================+======================+======================|\r\n",
      "|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |\r\n",
      "| 64%   67C    P0    88W / 260W |     21MiB / 11016MiB |     68%      Default |\r\n",
      "|                               |                      |                  N/A |\r\n",
      "+-------------------------------+----------------------+----------------------+\r\n",
      "|   1  NVIDIA GeForce ...  Off  | 00000000:03:00.0 Off |                  N/A |\r\n",
      "| 69%   73C    P2   299W / 340W |  10006MiB / 10018MiB |     90%      Default |\r\n",
      "|                               |                      |                  N/A |\r\n",
      "+-------------------------------+----------------------+----------------------+\r\n",
      "                                                                               \r\n",
      "+-----------------------------------------------------------------------------+\r\n",
      "| Processes:                                                                  |\r\n",
      "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\r\n",
      "|        ID   ID                                                   Usage      |\r\n",
      "|=============================================================================|\r\n",
      "+-----------------------------------------------------------------------------+\r\n"
     ]
    }
   ],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 120,
     "referenced_widgets": [
      "01d13bc99df7471dab7e0d451f4ca306",
      "0e1cdd7636d14f4e99fe337a60def6d2",
      "ab9a872a12be43a38c78fd5ad08466d5",
      "348af209c82b43939c37268d7f3a8f7e",
      "ddac62df1b4d4601b956158c4121060b",
      "70e762fedc8b4cdeacd5061008af36f9",
      "4787f83577b84d3e90b3c5796dcf9b37",
      "c9628333ac2845928bc5eaebf9b1192f",
      "d69f95f0416749d497733f019dfffc56",
      "63bb95b28d404b259eb3cdbbfa036e2c",
      "f863cac857d64166b6db9ae7dffca5aa"
     ]
    },
    "id": "pmO0w741Ilao",
    "outputId": "1ad9fa20-fd6a-482e-9fdc-d41206991023",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda:0 is available\n",
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "# 하이퍼 파라미터\n",
    "EPOCH = 100\n",
    "batch_size = 64\n",
    "learning_rate = 1e-3\n",
    "momentum = 0.9\n",
    "weight_decay = 1e-4\n",
    "\n",
    "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
    "print(f'{device} is available')\n",
    "\n",
    "# 분류 Class list\n",
    "classes = ('plane', 'car', 'bird', 'cat',\n",
    "           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')\n",
    "\n",
    "\n",
    "# 이미지 전처리\n",
    "transform = transforms.Compose(\n",
    "    [transforms.ToTensor(),\n",
    "    transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))]\n",
    "    )\n",
    "\n",
    "# Dataset. 변경 불가\n",
    "trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)\n",
    "testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True)\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "MTzCM8fKIVZy"
   },
   "outputs": [],
   "source": [
    "# ResNet\n",
    "\n",
    "\n",
    "class ResNet(nn.Module):\n",
    "    def __init__(self, block, num_blocks, num_classes=10):\n",
    "        super(ResNet, self).__init__()\n",
    "        self.in_planes = 64\n",
    "        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False)\n",
    "        self.bn1 = nn.BatchNorm2d(64)\n",
    "        self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1)\n",
    "        self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)\n",
    "        self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)\n",
    "        self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)\n",
    "        self.linear = nn.Linear(block.expansion*512, num_classes)\n",
    "\n",
    "    def _make_layer(self, block, planes, num_blocks, stride):\n",
    "        strides = [stride] + [1]*(num_blocks-1)\n",
    "        layers = []\n",
    "        for stride in strides:\n",
    "            layers.append(block(self.in_planes, planes, stride))\n",
    "            self.in_planes = planes * block.expansion\n",
    "        return nn.Sequential(*layers)\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = F.relu(self.bn1(self.conv1(x)))\n",
    "        out = self.layer1(out)\n",
    "        out = self.layer2(out)\n",
    "        out = self.layer3(out)\n",
    "        out = self.layer4(out)\n",
    "        out = F.avg_pool2d(out, 4)\n",
    "        out = out.view(out.size(0), -1)\n",
    "        out = self.linear(out)\n",
    "        return out\n",
    "\n",
    "class Bottleneck(nn.Module):\n",
    "    expansion = 4\n",
    "    def __init__(self, in_planes, planes, stride=1):\n",
    "        super(Bottleneck, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=1, bias=False)\n",
    "        self.bn1 = nn.BatchNorm2d(planes)\n",
    "        self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)\n",
    "        # self.bn2 = nn.BatchNorm2d(self.expansion*planes)\n",
    "        self.bn2 = nn.BatchNorm2d(planes)\n",
    "        self.conv3 = nn.Conv2d(planes, self.expansion*planes, kernel_size=1, bias=False)\n",
    "        self.bn3 = nn.BatchNorm2d(self.expansion*planes)\n",
    "\n",
    "        self.shortcut = nn.Sequential()\n",
    "        if stride != 1 or in_planes != self.expansion*planes:\n",
    "            self.shortcut = nn.Sequential(\n",
    "                nn.Conv2d(in_planes, self.expansion*planes,\n",
    "                          kernel_size=1, stride=stride, bias=False),\n",
    "                nn.BatchNorm2d(self.expansion*planes)\n",
    "            )\n",
    "\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = F.relu(self.bn1(self.conv1(x)))\n",
    "        out = F.relu(self.bn2(self.conv2(out)))\n",
    "        out = self.bn3(self.conv3(out))\n",
    "        out += self.shortcut(x)\n",
    "        out = F.relu(out)\n",
    "        return out\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "aQ85UXuTKEs5"
   },
   "outputs": [],
   "source": [
    "def ResNet50():\n",
    "    return ResNet(BasicBlock, [3, 4, 6, 3])\n",
    "\n",
    "def ResNet152():\n",
    "    return ResNet(Bottleneck, [3, 8, 36, 3])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "tDRKsgkOMnqO"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of parameters :  58156618\n"
     ]
    }
   ],
   "source": [
    "model = ResNet152().to(device)\n",
    "print(\"Number of parameters : \", sum(p.numel() for p in model.parameters() if p.requires_grad)) # 학습 대상 parameter들 출력"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RFG0KEDpIWRJ",
    "outputId": "c12600b9-989d-46bf-e74f-75aec0a235e7",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.74it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] loss: 1.725474\n",
      "epoch time : 209.35248851776123\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2] loss: 1.377476\n",
      "epoch time : 209.69253873825073\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3] loss: 1.166762\n",
      "epoch time : 209.63772749900818\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4] loss: 0.987735\n",
      "epoch time : 209.73563718795776\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5] loss: 0.846889\n",
      "epoch time : 209.73428606987\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6] loss: 0.713293\n",
      "epoch time : 209.80800080299377\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7] loss: 0.596326\n",
      "epoch time : 209.70469427108765\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8] loss: 0.487395\n",
      "epoch time : 209.79184436798096\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[9] loss: 0.373195\n",
      "epoch time : 209.84421730041504\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.72it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10] loss: 0.287266\n",
      "epoch time : 210.61166071891785\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11] loss: 0.217300\n",
      "epoch time : 209.5712788105011\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12] loss: 0.147856\n",
      "epoch time : 209.67087244987488\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13] loss: 0.110858\n",
      "epoch time : 209.85894346237183\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[14] loss: 0.082501\n",
      "epoch time : 209.71693801879883\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[15] loss: 0.063868\n",
      "epoch time : 209.825012922287\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[16] loss: 0.044177\n",
      "epoch time : 209.7421178817749\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[17] loss: 0.030763\n",
      "epoch time : 209.91302680969238\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[18] loss: 0.020200\n",
      "epoch time : 209.737886428833\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[19] loss: 0.009200\n",
      "epoch time : 209.64550757408142\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[20] loss: 0.005961\n",
      "epoch time : 210.42163395881653\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[21] loss: 0.003973\n",
      "epoch time : 209.53904938697815\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[22] loss: 0.002656\n",
      "epoch time : 209.70979118347168\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[23] loss: 0.002265\n",
      "epoch time : 209.61687231063843\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[24] loss: 0.001502\n",
      "epoch time : 209.86093544960022\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[25] loss: 0.001401\n",
      "epoch time : 209.72739338874817\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[26] loss: 0.001462\n",
      "epoch time : 209.8526463508606\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[27] loss: 0.001669\n",
      "epoch time : 209.92116045951843\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[28] loss: 0.001011\n",
      "epoch time : 209.63541221618652\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[29] loss: 0.000822\n",
      "epoch time : 209.69795680046082\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[30] loss: 0.000958\n",
      "epoch time : 210.3352587223053\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[31] loss: 0.001415\n",
      "epoch time : 209.71751165390015\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:29<00:00,  3.73it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[32] loss: 0.001383\n",
      "epoch time : 209.38048362731934\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:27<00:00,  3.76it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[33] loss: 0.000900\n",
      "epoch time : 207.70666480064392\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.78it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[34] loss: 0.000620\n",
      "epoch time : 206.76807570457458\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.78it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[35] loss: 0.000773\n",
      "epoch time : 206.79552698135376\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.78it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[36] loss: 0.000601\n",
      "epoch time : 206.8012661933899\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.78it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[37] loss: 0.000506\n",
      "epoch time : 206.6314971446991\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.78it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[38] loss: 0.000480\n",
      "epoch time : 206.8392996788025\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.78it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[39] loss: 0.000419\n",
      "epoch time : 206.78884172439575\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.78it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[40] loss: 0.000421\n",
      "epoch time : 207.45803308486938\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[41] loss: 0.000387\n",
      "epoch time : 206.40349125862122\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[42] loss: 0.000600\n",
      "epoch time : 206.35929465293884\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[43] loss: 0.000442\n",
      "epoch time : 206.57192730903625\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[44] loss: 0.000397\n",
      "epoch time : 206.32819819450378\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[45] loss: 0.000430\n",
      "epoch time : 206.444260597229\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[46] loss: 0.000429\n",
      "epoch time : 206.32177019119263\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[47] loss: 0.000544\n",
      "epoch time : 206.41449689865112\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[48] loss: 0.000304\n",
      "epoch time : 206.44459199905396\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[49] loss: 0.000342\n",
      "epoch time : 206.3400092124939\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50] loss: 0.000446\n",
      "epoch time : 207.20471143722534\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[51] loss: 0.000318\n",
      "epoch time : 206.14576315879822\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[52] loss: 0.000632\n",
      "epoch time : 206.0895550251007\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[53] loss: 0.000353\n",
      "epoch time : 206.13336730003357\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[54] loss: 0.000432\n",
      "epoch time : 206.05885410308838\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[55] loss: 0.000360\n",
      "epoch time : 206.07303047180176\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[56] loss: 0.000390\n",
      "epoch time : 206.12228679656982\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[57] loss: 0.000489\n",
      "epoch time : 206.16974902153015\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[58] loss: 0.000358\n",
      "epoch time : 206.07979369163513\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:25<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[59] loss: 0.000482\n",
      "epoch time : 205.97139716148376\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:25<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[60] loss: 0.000352\n",
      "epoch time : 206.6659767627716\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:25<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[61] loss: 0.000479\n",
      "epoch time : 205.97324895858765\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[62] loss: 0.000325\n",
      "epoch time : 206.03611159324646\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[63] loss: 0.000389\n",
      "epoch time : 206.15802145004272\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[64] loss: 0.000340\n",
      "epoch time : 206.12713885307312\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[65] loss: 0.000427\n",
      "epoch time : 206.04201364517212\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[66] loss: 0.000334\n",
      "epoch time : 206.23554158210754\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[67] loss: 0.000391\n",
      "epoch time : 206.15537905693054\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[68] loss: 0.000419\n",
      "epoch time : 206.0435311794281\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[69] loss: 0.000539\n",
      "epoch time : 206.05206990242004\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[70] loss: 0.000361\n",
      "epoch time : 206.81023907661438\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[71] loss: 0.000623\n",
      "epoch time : 206.0924060344696\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[72] loss: 0.000798\n",
      "epoch time : 206.02648496627808\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[73] loss: 0.000575\n",
      "epoch time : 206.05544686317444\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[74] loss: 0.000466\n",
      "epoch time : 206.10153698921204\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[75] loss: 0.000369\n",
      "epoch time : 206.12305903434753\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[76] loss: 0.000430\n",
      "epoch time : 206.08635783195496\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[77] loss: 0.000316\n",
      "epoch time : 206.08795404434204\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[78] loss: 0.000340\n",
      "epoch time : 206.03649735450745\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[79] loss: 0.000439\n",
      "epoch time : 206.13076663017273\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[80] loss: 0.000665\n",
      "epoch time : 206.89646697044373\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[81] loss: 0.000424\n",
      "epoch time : 206.16292667388916\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[82] loss: 0.000317\n",
      "epoch time : 206.10427951812744\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[83] loss: 0.000999\n",
      "epoch time : 206.05263996124268\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[84] loss: 0.002930\n",
      "epoch time : 206.0734293460846\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:25<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[85] loss: 0.001521\n",
      "epoch time : 205.86198139190674\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:25<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[86] loss: 0.000934\n",
      "epoch time : 205.9396653175354\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:25<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[87] loss: 0.002298\n",
      "epoch time : 205.98123145103455\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:25<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[88] loss: 0.002133\n",
      "epoch time : 205.97865056991577\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:25<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[89] loss: 0.001868\n",
      "epoch time : 205.8828604221344\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:25<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[90] loss: 0.003173\n",
      "epoch time : 206.6053056716919\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:25<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[91] loss: 0.018986\n",
      "epoch time : 205.94432663917542\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:25<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[92] loss: 0.121482\n",
      "epoch time : 205.9958381652832\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:25<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[93] loss: 0.093551\n",
      "epoch time : 205.9699149131775\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[94] loss: 0.061506\n",
      "epoch time : 206.20453071594238\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[95] loss: 0.048023\n",
      "epoch time : 206.1497642993927\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.80it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[96] loss: 0.027716\n",
      "epoch time : 206.06160473823547\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[97] loss: 0.031693\n",
      "epoch time : 206.15884113311768\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[98] loss: 0.020755\n",
      "epoch time : 206.22947430610657\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n",
      "  0%|          | 0/782 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[99] loss: 0.019981\n",
      "epoch time : 206.10172963142395\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 782/782 [03:26<00:00,  3.79it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[100] loss: 0.017532\n",
      "epoch time : 207.0015926361084\n"
     ]
    }
   ],
   "source": [
    "# criterion. 변경 '가능'\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "# Optimizer\n",
    "optimizer = optim.SGD(model.parameters(), lr=learning_rate, momentum=momentum, weight_decay=weight_decay)\n",
    "\n",
    "# Learning Rate Scheduler\n",
    "scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=50, verbose=False)\n",
    "\n",
    "loss_ = []\n",
    "n = len(trainloader)  # number of batches\n",
    "\n",
    "# Training\n",
    "for epoch in range(EPOCH): \n",
    "\n",
    "  running_loss = 0.0\n",
    "  start = time.time()\n",
    "  pbar = tqdm(trainloader, position=0, leave=True)\n",
    "  for batch, (X, y) in enumerate(pbar):\n",
    "    X, y = X.to(device), y.to(device)\n",
    "    optimizer.zero_grad() # 배치마다 optimizer 초기화\n",
    "    pred = model(X) \n",
    "    loss = criterion(pred, y) # 크로스 엔트로피 손실함수 계산 \n",
    "    loss.backward() # backpropagation\n",
    "    optimizer.step() # 가중치 최적화\n",
    "    running_loss += loss.item()\n",
    "\n",
    "  scheduler.step()\n",
    "  scheduler.get_last_lr()\n",
    "  if (epoch + 1) % 10 == 0:\n",
    "    dir = './model_' + str(epoch) + '.pt'\n",
    "    torch.save(model.state_dict() , dir) # 모델 저장, path 수정\n",
    "  loss_.append(running_loss / n)\n",
    "  print('[%d] loss: %.6f' %(epoch + 1, running_loss / len(trainloader)))\n",
    "  print(\"epoch time :\", time.time()-start)\n",
    "  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sf9UIvloTtt2"
   },
   "source": [
    "# Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "amoq6d5NMYvA"
   },
   "outputs": [],
   "source": [
    "# Evaluation\n",
    "\n",
    "# 수정\n",
    "# net = ResNet152().to(device)\n",
    "net = model\n",
    "# net.load_state_dict(torch.load('/content/gdrive/MyDrive/Baseline//model_ckpt.pt')) # 저장된 모델 불러오기\n",
    "\n",
    "\n",
    "# 이하 전체 수정 불가\n",
    "correct = 0\n",
    "total = 0\n",
    "correct_pred = {classname: 0 for classname in classes}\n",
    "total_pred = {classname: 0 for classname in classes}\n",
    "# since we're not training, we don't need to calculate the gradients for our outputs\n",
    "with torch.no_grad():\n",
    "    for data in testloader:\n",
    "        images, labels = data[0].to(device), data[1].to(device)\n",
    "        # calculate outputs by running images through the network\n",
    "        outputs = net(images)\n",
    "        # the class with the highest energy is what we choose as prediction\n",
    "        _, predicted = torch.max(outputs.data, 1)\n",
    "        total += labels.size(0)\n",
    "        correct += (predicted == labels).sum().item()\n",
    "        # collect the correct predictions for each class\n",
    "        for label, prediction in zip(labels, predicted):\n",
    "            if label == prediction:\n",
    "                correct_pred[classes[label]] += 1\n",
    "            total_pred[classes[label]] += 1\n",
    "\n",
    "print('Accuracy of the network on the 10000 test images: %d %%' % (\n",
    "    100 * correct / total))\n",
    "\n",
    "# print accuracy for each class\n",
    "for classname, correct_count in correct_pred.items():\n",
    "    accuracy = 100 * float(correct_count) / total_pred[classname]\n",
    "    print(\"Accuracy for class {:5s} is: {:.1f} %\".format(classname,\n",
    "                                                   accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1lUnuy-g4Aiq"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "def imshow(img):\n",
    "    img = img / 2 + 0.5     # unnormalize\n",
    "    npimg = img.numpy()\n",
    "    plt.figure(figsize=(20,10))\n",
    "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
    "    plt.show()\n",
    "\n",
    "dataiter = iter(testloader)\n",
    "images, labels = dataiter.next()\n",
    "\n",
    "# print images\n",
    "imshow(torchvision.utils.make_grid(images))\n",
    "print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(8)))\n",
    "\n",
    "net = model\n",
    "# net = Net()\n",
    "# net.to(device)\n",
    "# net.load_state_dict(torch.load('/content/gdrive/MyDrive/Baseline//model_ckpt.pt'))\n",
    "\n",
    "_, predicted = torch.max(outputs, 1)\n",
    "\n",
    "print('Predicted: ', ' '.join('%5s' % classes[predicted[j]]\n",
    "                              for j in range(8)))"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "CNN_Baseline (1).ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "01d13bc99df7471dab7e0d451f4ca306": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ab9a872a12be43a38c78fd5ad08466d5",
       "IPY_MODEL_348af209c82b43939c37268d7f3a8f7e",
       "IPY_MODEL_ddac62df1b4d4601b956158c4121060b"
      ],
      "layout": "IPY_MODEL_0e1cdd7636d14f4e99fe337a60def6d2"
     }
    },
    "0e1cdd7636d14f4e99fe337a60def6d2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "348af209c82b43939c37268d7f3a8f7e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d69f95f0416749d497733f019dfffc56",
      "max": 170498071,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_c9628333ac2845928bc5eaebf9b1192f",
      "value": 170498071
     }
    },
    "4787f83577b84d3e90b3c5796dcf9b37": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "63bb95b28d404b259eb3cdbbfa036e2c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "70e762fedc8b4cdeacd5061008af36f9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ab9a872a12be43a38c78fd5ad08466d5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4787f83577b84d3e90b3c5796dcf9b37",
      "placeholder": "​",
      "style": "IPY_MODEL_70e762fedc8b4cdeacd5061008af36f9",
      "value": ""
     }
    },
    "c9628333ac2845928bc5eaebf9b1192f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "d69f95f0416749d497733f019dfffc56": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ddac62df1b4d4601b956158c4121060b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f863cac857d64166b6db9ae7dffca5aa",
      "placeholder": "​",
      "style": "IPY_MODEL_63bb95b28d404b259eb3cdbbfa036e2c",
      "value": " 170499072/? [00:03&lt;00:00, 51840147.98it/s]"
     }
    },
    "f863cac857d64166b6db9ae7dffca5aa": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}