Lec5/Assignment4_CIFAR-10_with_CNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment4. CIFAR-10 Classification with CNN\n",
    "\n",
    "Lab7에서는 assignment3-CIFAR-10 Classification with MLP 코드의 MLP로 정의된 model arcitecture 부분을 수정하여 CNN으로 바꾸고, 이를 통해 CIFAR-10 dataset의 10-way classification을 CNN을 통해 해보았습니다. 그 과정에서 MLP에서 그러했듯, 다양한 hyperparameter들을 변수화하여 CNN model을 customize할 수 있게하였고, 그 결과 hyperparameter tunning 없이도 test accuracy 73%를 얻어낼 수 있었습니다. 그러나 loss graph 상에서 train / validation loss가 벌어지는 overfitting 현상 또한 볼 수 있었습니다.\n",
    "\n",
    "본 과제에서는 kernel size등 더 다양한 hyperparamteter들을 변수화하여 control할 수 있게하고, l2 regularization, fully connected layer의 dropout 등 다양한 tecnique들을 적용해 CNN 모델의 성능을 더 높여보도록 합시다.\n",
    "\n",
    "\n",
    "### 실험을 완료하신 뒤에는\n",
    "주피터 노트북 파일의 끝 부분에 최종적으로 최적화한 hyperparameter set과 그에 따른 test/validation/train accuracy를 리포트해 주신 뒤 github 링크를 [과제 제출 및 피드백 링크](https://goo.gl/forms/LpHtAMinhIYiiKCY2) 를 통해 제출해주시면 감사하겠습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>  \n",
    "\n",
    "# 시작하기\n",
    "\n",
    "[제가 미리 만들어놓은 이 링크](https://colab.research.google.com/github/heartcored98/Standalone-DeepLearning/blob/master/Lec5/Assignment4_CIFAR-10_with_CNN.ipynb)를 통해 Colab에서 바로 작업하실 수 있습니다!  \n",
    "만약 코드가 알록달록 하지 않다면(Syntax Highlighting이 안 된다면)  \n",
    "그냥 새로운 python3, GPU 런타임으로 Colab 파일을 만드신 후 코드를 복붙하시는게 빠를 겁니다.   \n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아래 코드를 통해 Colab에 연결되어 있는 서버 컴퓨터 내에 결과들을 저장할 results 폴더를 만들어봅시다.  \n",
    "만약 이미 만들었다면 **File exists** 와 같은 에러가 뜨면서 새로운 폴더 생성이 불가능할텐데 폴더만 있다면 괜찮으니 넘어가줍시다.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘results’: File exists\r\n"
     ]
    }
   ],
   "source": [
    "!mkdir results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PD4cIKKvKFCC"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import argparse\n",
    "import numpy as np\n",
    "import time\n",
    "from copy import deepcopy # Add Deepcopy for args\n",
    "import seaborn as sns \n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "29UainWPco7Y"
   },
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 73
    },
    "colab_type": "code",
    "id": "Cu753dPPKGkV",
    "outputId": "43b9be12-27a2-48ac-d9d4-08ea5dab0658"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "transform = transforms.Compose(\n",
    "    [transforms.ToTensor(),\n",
    "     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n",
    "\n",
    "trainset = torchvision.datasets.CIFAR10(root='./data', train=True,\n",
    "                                        download=True, transform=transform)\n",
    "trainset, valset = torch.utils.data.random_split(trainset, [40000, 10000])\n",
    "testset = torchvision.datasets.CIFAR10(root='./data', train=False,\n",
    "                                       download=True, transform=transform)\n",
    "partition = {'train': trainset, 'val':valset, 'test':testset}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RxnfFJwBcsAv"
   },
   "source": [
    "## Model Architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "cfg = {\n",
    "    'VGG11': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],\n",
    "    'VGG13': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],\n",
    "    'VGG16': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],\n",
    "    'VGG19': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M']\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CNN(nn.Module):\n",
    "    \n",
    "    def __init__(self, model_code, in_channels):\n",
    "        super(CNN, self).__init__()\n",
    "        \n",
    "        self.layers = self._make_layers(model_code, in_channels)\n",
    "        self.classifier = nn.Sequential(nn.Linear(512, 4096),\n",
    "                                        nn.ReLU(),\n",
    "                                        nn.Linear(4096, 10))\n",
    "        \n",
    "    def forward(self, x):\n",
    "        for layer in self.layers:\n",
    "            x = layer(x)\n",
    "        x = x.view(x.size(0), -1)\n",
    "        x = self.classifier(x)\n",
    "        return x\n",
    "\n",
    "    def _make_layers(self, model_code, in_channels):\n",
    "        layers = nn.ModuleList()\n",
    "        for x in cfg[model_code]:\n",
    "            if x != 'M':\n",
    "                layers.append(nn.Conv2d(in_channels=in_channels,\n",
    "                                        out_channels=x,\n",
    "                                        kernel_size=3,\n",
    "                                        stride=1,\n",
    "                                        padding=1))\n",
    "                layers.append(nn.ReLU())\n",
    "                in_channels = x\n",
    "            else:\n",
    "                layers.append(nn.MaxPool2d(kernel_size=2,\n",
    "                                           stride=2))\n",
    "        return layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5)\n",
    "        self.pool = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        self.fc1 = nn.Linear(16 * 5 * 5, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.pool(F.relu(self.conv1(x)))\n",
    "        x = self.pool(F.relu(self.conv2(x)))\n",
    "        x = x.view(-1, 16 * 5 * 5)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "def dimesion_check():\n",
    "    net = CNN('VGG19', 3)\n",
    "    x = torch.randn(2, 3, 32, 32)\n",
    "    y = net(x)\n",
    "    print(y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 10])\n"
     ]
    }
   ],
   "source": [
    "dimesion_check()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "itGsp6jDWs_a"
   },
   "source": [
    "## Train, Validate, Test and Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(net, partition, optimizer, criterion, args):\n",
    "    trainloader = torch.utils.data.DataLoader(partition['train'], \n",
    "                                              batch_size=args.train_batch_size, \n",
    "                                              shuffle=True, num_workers=2)\n",
    "    net.train()\n",
    "\n",
    "    correct = 0\n",
    "    total = 0\n",
    "    train_loss = 0.0\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        optimizer.zero_grad() # [21.01.05 오류 수정] 매 Epoch 마다 .zero_grad()가 실행되는 것을 매 iteration 마다 실행되도록 수정했습니다. \n",
    "\n",
    "        # get the inputs\n",
    "        inputs, labels = data\n",
    "        inputs = inputs.cuda()\n",
    "        labels = labels.cuda()\n",
    "        outputs = net(inputs)\n",
    "\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        train_loss += loss.item()\n",
    "        _, predicted = torch.max(outputs.data, 1)\n",
    "        total += labels.size(0)\n",
    "        correct += (predicted == labels).sum().item()\n",
    "\n",
    "    train_loss = train_loss / len(trainloader)\n",
    "    train_acc = 100 * correct / total\n",
    "    return net, train_loss, train_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate(net, partition, criterion, args):\n",
    "    valloader = torch.utils.data.DataLoader(partition['val'], \n",
    "                                            batch_size=args.test_batch_size, \n",
    "                                            shuffle=False, num_workers=2)\n",
    "    net.eval()\n",
    "\n",
    "    correct = 0\n",
    "    total = 0\n",
    "    val_loss = 0 \n",
    "    with torch.no_grad():\n",
    "        for data in valloader:\n",
    "            images, labels = data\n",
    "            images = images.cuda()\n",
    "            labels = labels.cuda()\n",
    "            outputs = net(images)\n",
    "\n",
    "            loss = criterion(outputs, labels)\n",
    "            \n",
    "            val_loss += loss.item()\n",
    "            _, predicted = torch.max(outputs.data, 1)\n",
    "            total += labels.size(0)\n",
    "            correct += (predicted == labels).sum().item()\n",
    "\n",
    "        val_loss = val_loss / len(valloader)\n",
    "        val_acc = 100 * correct / total\n",
    "    return val_loss, val_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(net, partition, args):\n",
    "    testloader = torch.utils.data.DataLoader(partition['test'], \n",
    "                                             batch_size=args.test_batch_size, \n",
    "                                             shuffle=False, num_workers=2)\n",
    "    net.eval()\n",
    "    \n",
    "    correct = 0\n",
    "    total = 0\n",
    "    with torch.no_grad():\n",
    "        for data in testloader:\n",
    "            images, labels = data\n",
    "            images = images.cuda()\n",
    "            labels = labels.cuda()\n",
    "\n",
    "            outputs = net(images)\n",
    "            _, predicted = torch.max(outputs.data, 1)\n",
    "            total += labels.size(0)\n",
    "            correct += (predicted == labels).sum().item()\n",
    "\n",
    "        test_acc = 100 * correct / total\n",
    "    return test_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LiOCP6TqWw2V"
   },
   "outputs": [],
   "source": [
    "def experiment(partition, args):\n",
    "  \n",
    "    net = net = CNN('VGG19', 3)\n",
    "    net.cuda()\n",
    "\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "    if args.optim == 'SGD':\n",
    "        optimizer = optim.SGD(net.parameters(), lr=args.lr, weight_decay=args.l2)\n",
    "    elif args.optim == 'RMSprop':\n",
    "        optimizer = optim.RMSprop(net.parameters(), lr=args.lr, weight_decay=args.l2)\n",
    "    elif args.optim == 'Adam':\n",
    "        optimizer = optim.Adam(net.parameters(), lr=args.lr, weight_decay=args.l2)\n",
    "    else:\n",
    "        raise ValueError('In-valid optimizer choice')\n",
    "    \n",
    "    train_losses = []\n",
    "    val_losses = []\n",
    "    train_accs = []\n",
    "    val_accs = []\n",
    "        \n",
    "    for epoch in range(args.epoch):  # loop over the dataset multiple times\n",
    "        ts = time.time()\n",
    "        net, train_loss, train_acc = train(net, partition, optimizer, criterion, args)\n",
    "        val_loss, val_acc = validate(net, partition, criterion, args)\n",
    "        te = time.time()\n",
    "        \n",
    "        train_losses.append(train_loss)\n",
    "        val_losses.append(val_loss)\n",
    "        train_accs.append(train_acc)\n",
    "        val_accs.append(val_acc)\n",
    "        \n",
    "        print('Epoch {}, Acc(train/val): {:2.2f}/{:2.2f}, Loss(train/val) {:2.2f}/{:2.2f}. Took {:2.2f} sec'.format(epoch, train_acc, val_acc, train_loss, val_loss, te-ts))\n",
    "        \n",
    "    test_acc = test(net, partition, args)    \n",
    "    \n",
    "    result = {}\n",
    "    result['train_losses'] = train_losses\n",
    "    result['val_losses'] = val_losses\n",
    "    result['train_accs'] = train_accs\n",
    "    result['val_accs'] = val_accs\n",
    "    result['train_acc'] = train_acc\n",
    "    result['val_acc'] = val_acc\n",
    "    result['test_acc'] = test_acc\n",
    "    return vars(args), result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Manage Experiment Result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "import hashlib\n",
    "import json\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "import pandas as pd\n",
    "\n",
    "def save_exp_result(setting, result):\n",
    "    exp_name = setting['exp_name']\n",
    "    del setting['epoch']\n",
    "    del setting['test_batch_size']\n",
    "\n",
    "    hash_key = hashlib.sha1(str(setting).encode()).hexdigest()[:6]\n",
    "    filename = './results/{}-{}.json'.format(exp_name, hash_key)\n",
    "    result.update(setting)\n",
    "    with open(filename, 'w') as f:\n",
    "        json.dump(result, f)\n",
    "\n",
    "    \n",
    "def load_exp_result(exp_name):\n",
    "    dir_path = './results'\n",
    "    filenames = [f for f in listdir(dir_path) if isfile(join(dir_path, f)) if '.json' in f]\n",
    "    list_result = []\n",
    "    for filename in filenames:\n",
    "        if exp_name in filename:\n",
    "            with open(join(dir_path, filename), 'r') as infile:\n",
    "                results = json.load(infile)\n",
    "                list_result.append(results)\n",
    "    df = pd.DataFrame(list_result) # .drop(columns=[])\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualization Utility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_acc(var1, var2, df):\n",
    "\n",
    "    fig, ax = plt.subplots(1, 3)\n",
    "    fig.set_size_inches(15, 6)\n",
    "    sns.set_style(\"darkgrid\", {\"axes.facecolor\": \".9\"})\n",
    "\n",
    "    sns.barplot(x=var1, y='train_acc', hue=var2, data=df, ax=ax[0])\n",
    "    sns.barplot(x=var1, y='val_acc', hue=var2, data=df, ax=ax[1])\n",
    "    sns.barplot(x=var1, y='test_acc', hue=var2, data=df, ax=ax[2])\n",
    "    \n",
    "    ax[0].set_title('Train Accuracy')\n",
    "    ax[1].set_title('Validation Accuracy')\n",
    "    ax[2].set_title('Test Accuracy')\n",
    "\n",
    "    \n",
    "def plot_loss_variation(var1, var2, df, **kwargs):\n",
    "\n",
    "    list_v1 = df[var1].unique()\n",
    "    list_v2 = df[var2].unique()\n",
    "    list_data = []\n",
    "\n",
    "    for value1 in list_v1:\n",
    "        for value2 in list_v2:\n",
    "            row = df.loc[df[var1]==value1]\n",
    "            row = row.loc[df[var2]==value2]\n",
    "\n",
    "            train_losses = list(row.train_losses)[0]\n",
    "            val_losses = list(row.val_losses)[0]\n",
    "\n",
    "            for epoch, train_loss in enumerate(train_losses):\n",
    "                list_data.append({'type':'train', 'loss':train_loss, 'epoch':epoch, var1:value1, var2:value2})\n",
    "            for epoch, val_loss in enumerate(val_losses):\n",
    "                list_data.append({'type':'val', 'loss':val_loss, 'epoch':epoch, var1:value1, var2:value2})\n",
    "\n",
    "    df = pd.DataFrame(list_data)\n",
    "    g = sns.FacetGrid(df, row=var2, col=var1, hue='type', **kwargs)\n",
    "    g = g.map(plt.plot, 'epoch', 'loss', marker='.')\n",
    "    g.add_legend()\n",
    "    g.fig.suptitle('Train loss vs Val loss')\n",
    "    plt.subplots_adjust(top=0.89) # 만약 Title이 그래프랑 겹친다면 top 값을 조정해주면 됩니다! 함수 인자로 받으면 그래프마다 조절할 수 있겠죠?\n",
    "\n",
    "\n",
    "def plot_acc_variation(var1, var2, df, **kwargs):\n",
    "    list_v1 = df[var1].unique()\n",
    "    list_v2 = df[var2].unique()\n",
    "    list_data = []\n",
    "\n",
    "    for value1 in list_v1:\n",
    "        for value2 in list_v2:\n",
    "            row = df.loc[df[var1]==value1]\n",
    "            row = row.loc[df[var2]==value2]\n",
    "\n",
    "            train_accs = list(row.train_accs)[0]\n",
    "            val_accs = list(row.val_accs)[0]\n",
    "            test_acc = list(row.test_acc)[0]\n",
    "\n",
    "            for epoch, train_acc in enumerate(train_accs):\n",
    "                list_data.append({'type':'train', 'Acc':train_acc, 'test_acc':test_acc, 'epoch':epoch, var1:value1, var2:value2})\n",
    "            for epoch, val_acc in enumerate(val_accs):\n",
    "                list_data.append({'type':'val', 'Acc':val_acc, 'test_acc':test_acc, 'epoch':epoch, var1:value1, var2:value2})\n",
    "\n",
    "    df = pd.DataFrame(list_data)\n",
    "    g = sns.FacetGrid(df, row=var2, col=var1, hue='type', **kwargs)\n",
    "    g = g.map(plt.plot, 'epoch', 'Acc', marker='.')\n",
    "\n",
    "    def show_acc(x, y, metric, **kwargs):\n",
    "        plt.scatter(x, y, alpha=0.3, s=1)\n",
    "        metric = \"Test Acc: {:1.3f}\".format(list(metric.values)[0])\n",
    "        plt.text(0.05, 0.95, metric,  horizontalalignment='left', verticalalignment='center', transform=plt.gca().transAxes, bbox=dict(facecolor='yellow', alpha=0.5, boxstyle=\"round,pad=0.1\"))\n",
    "    g = g.map(show_acc, 'epoch', 'Acc', 'test_acc')\n",
    "\n",
    "    g.add_legend()\n",
    "    g.fig.suptitle('Train Accuracy vs Val Accuracy')\n",
    "    plt.subplots_adjust(top=0.89)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "omgExzmQgU1J"
   },
   "source": [
    "## Experiment1. N_layer vs Hidden_Dim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DRoOy_B3Wu7B"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Namespace(act='relu', epoch=10, exp_name='exp1_lr', l2=1e-05, lr=0.0001, optim='RMSprop', test_batch_size=1024, train_batch_size=512)\n",
      "Epoch 0, Acc(train/val): 10.04/9.82, Loss(train/val) 2.30/2.30. Took 13.75 sec\n",
      "Epoch 1, Acc(train/val): 9.69/9.65, Loss(train/val) 2.30/2.30. Took 13.69 sec\n",
      "Epoch 2, Acc(train/val): 9.85/9.65, Loss(train/val) 2.30/2.30. Took 13.72 sec\n",
      "Epoch 3, Acc(train/val): 9.97/9.94, Loss(train/val) 2.30/2.30. Took 13.73 sec\n",
      "Epoch 4, Acc(train/val): 9.97/9.65, Loss(train/val) 2.30/2.30. Took 13.74 sec\n",
      "Epoch 5, Acc(train/val): 9.95/9.77, Loss(train/val) 2.30/2.30. Took 13.77 sec\n",
      "Epoch 6, Acc(train/val): 9.92/9.82, Loss(train/val) 2.30/2.30. Took 13.76 sec\n",
      "Epoch 7, Acc(train/val): 9.91/9.77, Loss(train/val) 2.30/2.30. Took 13.68 sec\n"
     ]
    }
   ],
   "source": [
    "# ====== Random Seed Initialization ====== #\n",
    "seed = 123\n",
    "np.random.seed(seed)\n",
    "torch.manual_seed(seed)\n",
    "\n",
    "parser = argparse.ArgumentParser()\n",
    "args = parser.parse_args(\"\")\n",
    "args.exp_name = \"exp1_lr\"\n",
    "\n",
    "# ====== Model Capacity ====== #\n",
    "args.act = 'relu'\n",
    "\n",
    "# ====== Regularization ======= #\n",
    "args.l2 = 0.00001\n",
    "\n",
    "# ====== Optimizer & Training ====== #\n",
    "args.optim = 'RMSprop' #'RMSprop' #SGD, RMSprop, ADAM...\n",
    "args.lr = 0.0015\n",
    "args.epoch = 10\n",
    "\n",
    "args.train_batch_size = 512\n",
    "args.test_batch_size = 1024\n",
    "\n",
    "# ====== Experiment Variable ====== #\n",
    "name_var1 = 'lr'\n",
    "name_var2 = 'optim'\n",
    "list_var1 = [0.0001, 0.00001]\n",
    "list_var2 = ['RMSprop']\n",
    "\n",
    "\n",
    "for var1 in list_var1:\n",
    "    for var2 in list_var2:\n",
    "        setattr(args, name_var1, var1)\n",
    "        setattr(args, name_var2, var2)\n",
    "        print(args)\n",
    "                \n",
    "        setting, result = experiment(partition, deepcopy(args))\n",
    "        save_exp_result(setting, result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var1 = 'lr'\n",
    "var2 = 'optim'\n",
    "df = load_exp_result('exp1')\n",
    "\n",
    "plot_acc(var1, var2, df)\n",
    "plot_loss_variation(var1, var2, df, sharey=False) #sharey를 True로 하면 모둔 subplot의 y축의 스케일이 같아집니다.\n",
    "plot_acc_variation(var1, var2, df, margin_titles=True, sharey=True) #margin_titles를 True로 하면 그래프의 가장자리에 var1과 var2 값이 표시되고 False로 하면 각 subplot 위에 표시됩니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "Untitled4.ipynb의 사본",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python (comet2)",
   "language": "python",
   "name": "comet2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}