2.sweep/Sweep_pytorch_tutorial.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Sweep pytorch tutorial\n",
        "\n",
        "📢 해당 노트북은 Wandb 공식 홈페이지 중 다음 내용을 참고했습니다. \n",
        "  \n",
        "[Organizing Hyperparameter Sweeps in PyTorch with W&B](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/pytorch/Organizing_Hyperparameter_Sweeps_in_PyTorch_with_W%26B.ipynb#scrollTo=6IgMNFFT2qe_)  \n",
        "[Sweep from Jupyter Notebook](https://docs.wandb.ai/v/ko/sweeps/python-api)  \n",
        "\n",
        "\n",
        "### 목차  \n",
        "Step 0: W&B Setup하기  \n",
        "Step 1: Congfig 정의하기  \n",
        "Step 2: Dataloader 함수 정의하기  \n",
        "Step 3: 모델 정의하기  \n",
        "Step 4: Train 함수와 Vaild 함수 정의하기  \n",
        "Step 5: Sweep 실행하기  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Step 0: W&B Setup하기\n",
        "colab에서 WandB를 사용하려면 wandb modul을 install 해야합니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "8TVDfr4lSZwq"
      },
      "outputs": [],
      "source": [
        "from torchvision import datasets, transforms\n",
        "from torch.utils.data import DataLoader\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "import wandb\n",
        "import math\n",
        "DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%capture\n",
        "!pip install wandb --upgrade"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "1KOburhAShIc"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33masdqwdasdqwdzxcxzv\u001b[0m (use `wandb login --relogin` to force relogin)\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import wandb\n",
        "\n",
        "wandb.login()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kHyx1L5STTaW"
      },
      "source": [
        "## Step1. Config 정의하기\n",
        "\n",
        "sweep에 필요한 config 파일을 정의합니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "oCtuPcO7ShFm"
      },
      "outputs": [],
      "source": [
        "sweep_config = {\n",
        "    'name' : 'bayes-test',\n",
        "    'method': 'random',\n",
        "    'metric' : {\n",
        "        'name': 'val_loss',\n",
        "        'goal': 'minimize'   \n",
        "        },\n",
        "    'parameters' : {\n",
        "        'optimizer': {\n",
        "            'values': ['adam', 'sgd']\n",
        "            },\n",
        "        'dropout': {\n",
        "            'values': [0.3, 0.4]\n",
        "            },\n",
        "        'learning_rate': {\n",
        "            'distribution': 'uniform',\n",
        "            'min': 0,\n",
        "            'max': 0.1\n",
        "            },\n",
        "        'epochs': {\n",
        "            'values': [5, 6]\n",
        "            },\n",
        "        'batch_size': {\n",
        "            'distribution': 'q_log_uniform',\n",
        "            'q': 1,\n",
        "            'min': math.log(32),\n",
        "            'max': math.log(256),\n",
        "            }\n",
        "        }\n",
        "    }"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TBBWJbcYaDzb"
      },
      "source": [
        "## Step2. Dataloader 함수 정의하기\n",
        "mnist 함수를 사용해서 dataloader를 구성하겠습니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "id": "h3NwvXGdShDO"
      },
      "outputs": [],
      "source": [
        "def SweepDataset(batch_size):\n",
        "    transform = transforms.Compose(\n",
        "        [transforms.ToTensor(),\n",
        "         transforms.Normalize((0.1307,), (0.3081,))])\n",
        "    \n",
        "    train_data = datasets.MNIST(\".\", \n",
        "                train=True, \n",
        "                download=True,\n",
        "                transform=transform)\n",
        "    \n",
        "    test_data = datasets.MNIST(\".\", \n",
        "                train=False, \n",
        "                download=True,\n",
        "                transform=transform)\n",
        "    \n",
        "    \n",
        "    train_loader = DataLoader(train_data, batch_size=batch_size)\n",
        "    test_loader = DataLoader(test_data, batch_size=batch_size)\n",
        "\n",
        "    return train_loader, test_loader"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bc2GfRccaH7c"
      },
      "source": [
        "## Step3. 모델 정의하기\n",
        "모델과 optimizer를 정의해줍니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "id": "om_xkH-_ShAk"
      },
      "outputs": [],
      "source": [
        "class ConvNet(nn.Module):\n",
        "    def __init__(self, dropout):\n",
        "        super(ConvNet, self).__init__()\n",
        "        self.layer1 = nn.Sequential(\n",
        "            nn.Conv2d(1, 32, 3, 1, 1), nn.ReLU(),\n",
        "            nn.MaxPool2d(2, 2))\n",
        "        self.layer2 = nn.Sequential(\n",
        "            nn.Conv2d(32, 64, 3, 1, 1), nn.ReLU(),\n",
        "            nn.MaxPool2d(2, 2))\n",
        "        self.layer3 = nn.Sequential(\n",
        "            nn.Linear(64 * 7 * 7, 128, bias=True), nn.ReLU(),\n",
        "            nn.Dropout2d(p=dropout))\n",
        "        self.layer4 = nn.Sequential(\n",
        "            nn.Linear(128, 84), nn.ReLU(),\n",
        "            nn.Dropout2d(p=dropout))\n",
        "        self.fc3 = nn.Linear(84, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.layer1(x)\n",
        "        x = self.layer2(x)\n",
        "        x = x.view(x.size(0),-1) \n",
        "        x = self.layer3(x)\n",
        "        x = self.layer4(x)\n",
        "        x = self.fc3(x)\n",
        "        return x"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "id": "bzCoH_hpT7qI"
      },
      "outputs": [],
      "source": [
        "def build_optimizer(network, optimizer, learning_rate):\n",
        "    if optimizer == \"sgd\":\n",
        "        optimizer = optim.SGD(network.parameters(),\n",
        "                              lr=learning_rate, momentum=0.9)\n",
        "    elif optimizer == \"adam\":\n",
        "        optimizer = optim.Adam(network.parameters(),\n",
        "                               lr=learning_rate)\n",
        "    return optimizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CYZAPlIKaMZ2"
      },
      "source": [
        "## Step4. Train 함수와 Vaild함수 정의하기\n",
        "train함수는 학습을 진행하면서 loss를 출력합니다.  \n",
        "vaild함수는 학습 중간에 Accuracy를 사용해서 평가합니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 48,
      "metadata": {
        "id": "SFX6aIXpSg9s"
      },
      "outputs": [],
      "source": [
        "def train(model, loader, criterion, optimizer, device, config, wandb):\n",
        "    model.train()\n",
        "    wandb.watch(model, criterion, log=\"all\", log_freq=10)\n",
        "\n",
        "    for epoch in range(config.epochs):\n",
        "        cumu_loss = 0\n",
        "        for images, labels in loader:\n",
        "            images, labels  = images.to(device), labels.to(device)\n",
        "\n",
        "            output = model(images)\n",
        "            loss = criterion(output, labels)\n",
        "            cumu_loss += loss.item()\n",
        "\n",
        "            optimizer.zero_grad()\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "\n",
        "        avg_loss = cumu_loss / len(loader)\n",
        "        wandb.log({\"train_loss\": avg_loss}, step=epoch)  \n",
        "        print(f\"TRAIN: EPOCH {epoch + 1:04d} / {config.epochs:04d} | Epoch LOSS {avg_loss:.4f}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "metadata": {
        "id": "LLehH0A5pKER"
      },
      "outputs": [],
      "source": [
        "def vaild(model, loader, criterion, device,  wandb):\n",
        "    model.eval()\n",
        "\n",
        "    with torch.no_grad():\n",
        "        correct, test_loss = 0, 0\n",
        "        for data, target in loader:\n",
        "            data, target = data.to(device), target.to(device)\n",
        "            output = model(data)\n",
        "            \n",
        "            test_loss += criterion(output, target).item()\n",
        "            \n",
        "            pred = output.max(1, keepdim=True)[1]\n",
        "            correct += pred.eq(target.view_as(pred)).sum().item()\n",
        "            \n",
        "    \n",
        "    val_loss = test_loss / len(loader)\n",
        "    print(f\"VALID: LOSS {val_loss:.4f} | Accuracy {val_loss:.4f} \")\n",
        "    wandb.log({\n",
        "        \"val_acc\": 100. * correct / len(loader.dataset),\n",
        "        \"val_loss\": val_loss})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lmJSYYDzYpu3"
      },
      "source": [
        "## Step5. Sweep 실행하기"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "metadata": {
        "id": "SMADlVI6Sg7H"
      },
      "outputs": [],
      "source": [
        "def run_sweeep(config=None):\n",
        "    wandb.init(config=config)\n",
        "    \n",
        "    w_config = wandb.config\n",
        "    \n",
        "    criterion = nn.CrossEntropyLoss()\n",
        "    train_loader, vaild_loader = SweepDataset(w_config.batch_size)\n",
        "    model = ConvNet(w_config.dropout).to(DEVICE)\n",
        "    optimizer = build_optimizer(model, w_config.optimizer, w_config.learning_rate)\n",
        "    \n",
        "    train(model, train_loader, criterion, optimizer, DEVICE, w_config, wandb)\n",
        "    vaild(model, vaild_loader, criterion, DEVICE, wandb)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "metadata": {
        "id": "gi7BCOC3Sgro"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Create sweep with ID: 8mruz4pk\n",
            "Sweep URL: https://wandb.ai/pebpung/sweep_tutorial/sweeps/8mruz4pk\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: wdakaxg2 with config:\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 224\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tdropout: 0.3\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 5\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 0.0791329183434094\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \toptimizer: adam\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                    Syncing run <strong><a href=\"https://wandb.ai/pebpung/sweep_tutorial/runs/wdakaxg2\" target=\"_blank\">vital-sweep-1</a></strong> to <a href=\"https://wandb.ai/pebpung/sweep_tutorial\" target=\"_blank\">Weights & Biases</a> (<a href=\"https://docs.wandb.com/integrations/jupyter.html\" target=\"_blank\">docs</a>).<br/>\n",
              "Sweep page: <a href=\"https://wandb.ai/pebpung/sweep_tutorial/sweeps/8mruz4pk\" target=\"_blank\">https://wandb.ai/pebpung/sweep_tutorial/sweeps/8mruz4pk</a><br/>\n",
              "\n",
              "                "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TRAIN: EPOCH 0001 / 0005 | Epoch LOSS 6.6979\n",
            "TRAIN: EPOCH 0002 / 0005 | Epoch LOSS 2.3035\n",
            "TRAIN: EPOCH 0003 / 0005 | Epoch LOSS 2.3036\n",
            "TRAIN: EPOCH 0004 / 0005 | Epoch LOSS 2.3036\n",
            "TRAIN: EPOCH 0005 / 0005 | Epoch LOSS 2.3036\n",
            "VALID: LOSS 2.3016 | Accuracy 2.3016 \n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<br/>Waiting for W&B process to finish, PID 32219... <strong style=\"color:green\">(success).</strong>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "41619a2086fb4d51abb2e1e50f177e9d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "VBox(children=(Label(value=' 0.00MB of 0.00MB uploaded (0.00MB deduped)\\r'), FloatProgress(value=1.0, max=1.0)…"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<style>\n",
              "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: right }\n",
              "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; width: 100% }\n",
              "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
              "    </style>\n",
              "<div class=\"wandb-row\"><div class=\"wandb-col\">\n",
              "<h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>train_loss</td><td>█▁▁▁▁</td></tr><tr><td>val_acc</td><td>▁</td></tr><tr><td>val_loss</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\">\n",
              "<h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>train_loss</td><td>2.30363</td></tr><tr><td>val_acc</td><td>11.35</td></tr><tr><td>val_loss</td><td>2.30157</td></tr></table>\n",
              "</div></div>\n",
              "Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)\n",
              "<br/>Synced <strong style=\"color:#cdcd00\">vital-sweep-1</strong>: <a href=\"https://wandb.ai/pebpung/sweep_tutorial/runs/wdakaxg2\" target=\"_blank\">https://wandb.ai/pebpung/sweep_tutorial/runs/wdakaxg2</a><br/>\n",
              "Find logs at: <code>./wandb/run-20211103_122901-wdakaxg2/logs</code><br/>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: ulezb7kv with config:\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 159\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tdropout: 0.4\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 5\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 0.00936485889539045\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \toptimizer: sgd\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                    Syncing run <strong><a href=\"https://wandb.ai/pebpung/sweep_tutorial/runs/ulezb7kv\" target=\"_blank\">playful-sweep-2</a></strong> to <a href=\"https://wandb.ai/pebpung/sweep_tutorial\" target=\"_blank\">Weights & Biases</a> (<a href=\"https://docs.wandb.com/integrations/jupyter.html\" target=\"_blank\">docs</a>).<br/>\n",
              "Sweep page: <a href=\"https://wandb.ai/pebpung/sweep_tutorial/sweeps/8mruz4pk\" target=\"_blank\">https://wandb.ai/pebpung/sweep_tutorial/sweeps/8mruz4pk</a><br/>\n",
              "\n",
              "                "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TRAIN: EPOCH 0001 / 0005 | Epoch LOSS 0.6703\n",
            "TRAIN: EPOCH 0002 / 0005 | Epoch LOSS 0.1695\n",
            "TRAIN: EPOCH 0003 / 0005 | Epoch LOSS 0.1176\n",
            "TRAIN: EPOCH 0004 / 0005 | Epoch LOSS 0.0958\n",
            "TRAIN: EPOCH 0005 / 0005 | Epoch LOSS 0.0806\n",
            "VALID: LOSS 0.0408 | Accuracy 0.0408 \n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<br/>Waiting for W&B process to finish, PID 32434... <strong style=\"color:green\">(success).</strong>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a6f374c04eb44520b081af75f8e79d57",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "VBox(children=(Label(value=' 0.00MB of 0.00MB uploaded (0.00MB deduped)\\r'), FloatProgress(value=1.0, max=1.0)…"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<style>\n",
              "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: right }\n",
              "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; width: 100% }\n",
              "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
              "    </style>\n",
              "<div class=\"wandb-row\"><div class=\"wandb-col\">\n",
              "<h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>train_loss</td><td>█▂▁▁▁</td></tr><tr><td>val_acc</td><td>▁</td></tr><tr><td>val_loss</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\">\n",
              "<h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>train_loss</td><td>0.08062</td></tr><tr><td>val_acc</td><td>98.73</td></tr><tr><td>val_loss</td><td>0.04078</td></tr></table>\n",
              "</div></div>\n",
              "Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)\n",
              "<br/>Synced <strong style=\"color:#cdcd00\">playful-sweep-2</strong>: <a href=\"https://wandb.ai/pebpung/sweep_tutorial/runs/ulezb7kv\" target=\"_blank\">https://wandb.ai/pebpung/sweep_tutorial/runs/ulezb7kv</a><br/>\n",
              "Find logs at: <code>./wandb/run-20211103_122954-ulezb7kv/logs</code><br/>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: g0riigbo with config:\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 232\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tdropout: 0.3\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 5\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 0.09271816749021068\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \toptimizer: sgd\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                    Syncing run <strong><a href=\"https://wandb.ai/pebpung/sweep_tutorial/runs/g0riigbo\" target=\"_blank\">stoic-sweep-3</a></strong> to <a href=\"https://wandb.ai/pebpung/sweep_tutorial\" target=\"_blank\">Weights & Biases</a> (<a href=\"https://docs.wandb.com/integrations/jupyter.html\" target=\"_blank\">docs</a>).<br/>\n",
              "Sweep page: <a href=\"https://wandb.ai/pebpung/sweep_tutorial/sweeps/8mruz4pk\" target=\"_blank\">https://wandb.ai/pebpung/sweep_tutorial/sweeps/8mruz4pk</a><br/>\n",
              "\n",
              "                "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TRAIN: EPOCH 0001 / 0005 | Epoch LOSS 0.3774\n",
            "TRAIN: EPOCH 0002 / 0005 | Epoch LOSS 0.0993\n",
            "TRAIN: EPOCH 0003 / 0005 | Epoch LOSS 0.0723\n",
            "TRAIN: EPOCH 0004 / 0005 | Epoch LOSS 0.0568\n",
            "TRAIN: EPOCH 0005 / 0005 | Epoch LOSS 0.0480\n",
            "VALID: LOSS 0.0373 | Accuracy 0.0373 \n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<br/>Waiting for W&B process to finish, PID 32620... <strong style=\"color:green\">(success).</strong>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2bf2b70f42c8451489a05cd9c0dcd075",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "VBox(children=(Label(value=' 0.00MB of 0.00MB uploaded (0.00MB deduped)\\r'), FloatProgress(value=1.0, max=1.0)…"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<style>\n",
              "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: right }\n",
              "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; width: 100% }\n",
              "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
              "    </style>\n",
              "<div class=\"wandb-row\"><div class=\"wandb-col\">\n",
              "<h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>train_loss</td><td>█▂▂▁▁</td></tr><tr><td>val_acc</td><td>▁</td></tr><tr><td>val_loss</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\">\n",
              "<h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>train_loss</td><td>0.04796</td></tr><tr><td>val_acc</td><td>98.79</td></tr><tr><td>val_loss</td><td>0.0373</td></tr></table>\n",
              "</div></div>\n",
              "Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)\n",
              "<br/>Synced <strong style=\"color:#cdcd00\">stoic-sweep-3</strong>: <a href=\"https://wandb.ai/pebpung/sweep_tutorial/runs/g0riigbo\" target=\"_blank\">https://wandb.ai/pebpung/sweep_tutorial/runs/g0riigbo</a><br/>\n",
              "Find logs at: <code>./wandb/run-20211103_123048-g0riigbo/logs</code><br/>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: fdbyrxt0 with config:\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 93\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tdropout: 0.3\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 5\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 0.09129809703051509\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \toptimizer: adam\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                    Syncing run <strong><a href=\"https://wandb.ai/pebpung/sweep_tutorial/runs/fdbyrxt0\" target=\"_blank\">effortless-sweep-4</a></strong> to <a href=\"https://wandb.ai/pebpung/sweep_tutorial\" target=\"_blank\">Weights & Biases</a> (<a href=\"https://docs.wandb.com/integrations/jupyter.html\" target=\"_blank\">docs</a>).<br/>\n",
              "Sweep page: <a href=\"https://wandb.ai/pebpung/sweep_tutorial/sweeps/8mruz4pk\" target=\"_blank\">https://wandb.ai/pebpung/sweep_tutorial/sweeps/8mruz4pk</a><br/>\n",
              "\n",
              "                "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TRAIN: EPOCH 0001 / 0005 | Epoch LOSS 7.6609\n",
            "TRAIN: EPOCH 0002 / 0005 | Epoch LOSS 2.3060\n",
            "TRAIN: EPOCH 0003 / 0005 | Epoch LOSS 2.3061\n",
            "TRAIN: EPOCH 0004 / 0005 | Epoch LOSS 2.3061\n",
            "TRAIN: EPOCH 0005 / 0005 | Epoch LOSS 2.3062\n",
            "VALID: LOSS 2.3050 | Accuracy 2.3050 \n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<br/>Waiting for W&B process to finish, PID 390... <strong style=\"color:green\">(success).</strong>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ba91a41f5a85456d8e50871f9ce41faa",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "VBox(children=(Label(value=' 0.00MB of 0.00MB uploaded (0.00MB deduped)\\r'), FloatProgress(value=1.0, max=1.0)…"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<style>\n",
              "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: right }\n",
              "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; width: 100% }\n",
              "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
              "    </style>\n",
              "<div class=\"wandb-row\"><div class=\"wandb-col\">\n",
              "<h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>train_loss</td><td>█▁▁▁▁</td></tr><tr><td>val_acc</td><td>▁</td></tr><tr><td>val_loss</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\">\n",
              "<h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>train_loss</td><td>2.30615</td></tr><tr><td>val_acc</td><td>9.58</td></tr><tr><td>val_loss</td><td>2.30497</td></tr></table>\n",
              "</div></div>\n",
              "Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)\n",
              "<br/>Synced <strong style=\"color:#cdcd00\">effortless-sweep-4</strong>: <a href=\"https://wandb.ai/pebpung/sweep_tutorial/runs/fdbyrxt0\" target=\"_blank\">https://wandb.ai/pebpung/sweep_tutorial/runs/fdbyrxt0</a><br/>\n",
              "Find logs at: <code>./wandb/run-20211103_123139-fdbyrxt0/logs</code><br/>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: zrkzryi4 with config:\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 105\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tdropout: 0.4\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 5\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 0.04847267863519518\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \toptimizer: adam\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                    Syncing run <strong><a href=\"https://wandb.ai/pebpung/sweep_tutorial/runs/zrkzryi4\" target=\"_blank\">rosy-sweep-5</a></strong> to <a href=\"https://wandb.ai/pebpung/sweep_tutorial\" target=\"_blank\">Weights & Biases</a> (<a href=\"https://docs.wandb.com/integrations/jupyter.html\" target=\"_blank\">docs</a>).<br/>\n",
              "Sweep page: <a href=\"https://wandb.ai/pebpung/sweep_tutorial/sweeps/8mruz4pk\" target=\"_blank\">https://wandb.ai/pebpung/sweep_tutorial/sweeps/8mruz4pk</a><br/>\n",
              "\n",
              "                "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TRAIN: EPOCH 0001 / 0005 | Epoch LOSS 2.9021\n",
            "TRAIN: EPOCH 0002 / 0005 | Epoch LOSS 2.3035\n",
            "TRAIN: EPOCH 0003 / 0005 | Epoch LOSS 2.3035\n",
            "TRAIN: EPOCH 0004 / 0005 | Epoch LOSS 2.3036\n",
            "TRAIN: EPOCH 0005 / 0005 | Epoch LOSS 2.3036\n",
            "VALID: LOSS 2.3021 | Accuracy 2.3021 \n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<br/>Waiting for W&B process to finish, PID 644... <strong style=\"color:green\">(success).</strong>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c1ef2b0049514522a37a3a1a31b62015",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "VBox(children=(Label(value=' 0.00MB of 0.00MB uploaded (0.00MB deduped)\\r'), FloatProgress(value=1.0, max=1.0)…"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<style>\n",
              "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: right }\n",
              "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; width: 100% }\n",
              "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
              "    </style>\n",
              "<div class=\"wandb-row\"><div class=\"wandb-col\">\n",
              "<h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>train_loss</td><td>█▁▁▁▁</td></tr><tr><td>val_acc</td><td>▁</td></tr><tr><td>val_loss</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\">\n",
              "<h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>train_loss</td><td>2.30357</td></tr><tr><td>val_acc</td><td>10.1</td></tr><tr><td>val_loss</td><td>2.30207</td></tr></table>\n",
              "</div></div>\n",
              "Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)\n",
              "<br/>Synced <strong style=\"color:#cdcd00\">rosy-sweep-5</strong>: <a href=\"https://wandb.ai/pebpung/sweep_tutorial/runs/zrkzryi4\" target=\"_blank\">https://wandb.ai/pebpung/sweep_tutorial/runs/zrkzryi4</a><br/>\n",
              "Find logs at: <code>./wandb/run-20211103_123241-zrkzryi4/logs</code><br/>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: Agent Starting Run: qw10xfkb with config:\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tbatch_size: 68\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tdropout: 0.3\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tepochs: 5\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \tlearning_rate: 0.021868136115833214\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: \toptimizer: adam\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                    Syncing run <strong><a href=\"https://wandb.ai/pebpung/sweep_tutorial/runs/qw10xfkb\" target=\"_blank\">rural-sweep-6</a></strong> to <a href=\"https://wandb.ai/pebpung/sweep_tutorial\" target=\"_blank\">Weights & Biases</a> (<a href=\"https://docs.wandb.com/integrations/jupyter.html\" target=\"_blank\">docs</a>).<br/>\n",
              "Sweep page: <a href=\"https://wandb.ai/pebpung/sweep_tutorial/sweeps/8mruz4pk\" target=\"_blank\">https://wandb.ai/pebpung/sweep_tutorial/sweeps/8mruz4pk</a><br/>\n",
              "\n",
              "                "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "TRAIN: EPOCH 0001 / 0005 | Epoch LOSS 2.3277\n",
            "TRAIN: EPOCH 0002 / 0005 | Epoch LOSS 2.3025\n",
            "TRAIN: EPOCH 0003 / 0005 | Epoch LOSS 2.3025\n",
            "TRAIN: EPOCH 0004 / 0005 | Epoch LOSS 2.3025\n",
            "TRAIN: EPOCH 0005 / 0005 | Epoch LOSS 2.3025\n",
            "VALID: LOSS 2.3016 | Accuracy 2.3016 \n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<br/>Waiting for W&B process to finish, PID 958... <strong style=\"color:green\">(success).</strong>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ed16b9b7fbb94913af3afb5be6b04f80",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "VBox(children=(Label(value=' 0.00MB of 0.00MB uploaded (0.00MB deduped)\\r'), FloatProgress(value=1.0, max=1.0)…"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<style>\n",
              "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: right }\n",
              "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; width: 100% }\n",
              "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
              "    </style>\n",
              "<div class=\"wandb-row\"><div class=\"wandb-col\">\n",
              "<h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>train_loss</td><td>█▁▁▁▁</td></tr><tr><td>val_acc</td><td>▁</td></tr><tr><td>val_loss</td><td>▁</td></tr></table><br/></div><div class=\"wandb-col\">\n",
              "<h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>train_loss</td><td>2.30254</td></tr><tr><td>val_acc</td><td>11.35</td></tr><tr><td>val_loss</td><td>2.30159</td></tr></table>\n",
              "</div></div>\n",
              "Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)\n",
              "<br/>Synced <strong style=\"color:#cdcd00\">rural-sweep-6</strong>: <a href=\"https://wandb.ai/pebpung/sweep_tutorial/runs/qw10xfkb\" target=\"_blank\">https://wandb.ai/pebpung/sweep_tutorial/runs/qw10xfkb</a><br/>\n",
              "Find logs at: <code>./wandb/run-20211103_123340-qw10xfkb/logs</code><br/>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sweep_id = wandb.sweep(sweep_config, project=\"sweep_tutorial\", entity='pebpung')\n",
        "wandb.agent(sweep_id, run_sweeep, count=6)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "Sweep_pytorch_tutorial.ipynb",
      "private_outputs": true,
      "provenance": []
    },
    "interpreter": {
      "hash": "8e5e08de07fba0fb72df28d1d02d73caaafb877a45fd93aa5b25ed7e0e8c6fec"
    },
    "kernelspec": {
      "display_name": "Python 3.8.0 64-bit ('torch': conda)",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}