HW1_strong_baseline.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "HW1-strong baseline",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/PolarisRisingWar/lihongyi-2021-colab/blob/main/HW1_strong_baseline.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mz0_QVkxCrX3"
      },
      "source": [
        "# **Homework 1: COVID-19 Cases Prediction (Regression)**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZeZnPAiwDRWG"
      },
      "source": [
        "本notebook文件改自助教提供的simple baseline原文件。  \n",
        "在kaggle平台可以看到strong baseline在private leaderboard上是0.89266，在public leaderboard上是0.88017\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MbkbC-84LIdu"
      },
      "source": [
        "**整体策略**\n",
        "1. 特征工程<br>\n",
        "&nbsp;1. 正则化<br>\n",
        "&nbsp;2. 选择表现能力更强的特征<br>\n",
        "2. DNN architecture<br>\n",
        "&nbsp;1. layers<br>\n",
        "&nbsp;2. dimension<br>\n",
        "&nbsp;3. activation function<br>\n",
        "3. Training<br>\n",
        "&nbsp;1. mini-batch<br>\n",
        "&nbsp;2. optimizer<br>\n",
        "&nbsp;3. learning rate<br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jx3x1nDkG-Uy"
      },
      "source": [
        "# **Download Data**\n",
        "\n",
        "\n",
        "If the Google drive links are dead, you can download data from [kaggle](https://www.kaggle.com/c/ml2021spring-hw1/data), and upload data manually to the workspace."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tMj55YDKG6ch",
        "outputId": "0aa91192-6344-4609-977b-64fc308fe394"
      },
      "source": [
        "tr_path = 'covid.train.csv'  # path to training data\n",
        "tt_path = 'covid.test.csv'   # path to testing data\n",
        "\n",
        "!gdown --id '19CCyCgJrUxtvgZF53vnctJiOJ23T5mqF' --output covid.train.csv\n",
        "!gdown --id '1CE240jLm2npU-tdz81-oVKEF3T2yfT1O' --output covid.test.csv"
      ],
      "execution_count": 496,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=19CCyCgJrUxtvgZF53vnctJiOJ23T5mqF\n",
            "To: /content/covid.train.csv\n",
            "100% 2.00M/2.00M [00:00<00:00, 52.9MB/s]\n",
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1CE240jLm2npU-tdz81-oVKEF3T2yfT1O\n",
            "To: /content/covid.test.csv\n",
            "100% 651k/651k [00:00<00:00, 43.3MB/s]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wS_4-77xHk44"
      },
      "source": [
        "# **Import Some Packages**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k-onQd4JNA5H"
      },
      "source": [
        "# PyTorch\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.utils.data import Dataset, DataLoader\n",
        "\n",
        "# For data preprocess\n",
        "import numpy as np\n",
        "import csv\n",
        "import os\n",
        "\n",
        "# For plotting\n",
        "import matplotlib.pyplot as plt\n",
        "from matplotlib.pyplot import figure\n",
        "\n",
        "#PRW\n",
        "import pandas as pd\n",
        "import seaborn as sns\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.feature_selection import SelectKBest\n",
        "from sklearn.feature_selection import f_regression\n",
        "\n",
        "myseed = 42069  # set a random seed for reproducibility\n",
        "torch.backends.cudnn.deterministic = True\n",
        "#torch.backends.cudnn.benchmark = False  #PRW：我把这行去掉了，因为我觉得我不是很追求这个可复现性，我更追求结果表现\n",
        "np.random.seed(myseed)\n",
        "torch.manual_seed(myseed)\n",
        "if torch.cuda.is_available():\n",
        "    torch.cuda.manual_seed_all(myseed)"
      ],
      "execution_count": 497,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vHwpMAyCPvQv"
      },
      "source": [
        "# 数据探查"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7nuUYeidPzJ8"
      },
      "source": [
        "train_df=pd.read_csv(tr_path)"
      ],
      "execution_count": 498,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_bLtwfkKxhTt"
      },
      "source": [
        "#train_df.info()  没有缺失值\n",
        "#第一列是id，93列特征，train最后一列是数值型因变量\n",
        "train_df.drop(['id'],axis=1,inplace=True)  #删除id列"
      ],
      "execution_count": 499,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3EDU1s5JyMMQ"
      },
      "source": [
        "#sns.pairplot要跑好久\n",
        "#train_df.iloc[:,40:]  除40个state之外的列\n",
        "#train_df.iloc[:,40:].describe()\n",
        "bestfeatures = SelectKBest(score_func=f_regression, k=20)\n",
        "bestfeatures.fit(train_df.iloc[:,:-1].values,train_df.iloc[:,-1].values)\n",
        "selected_feature_columns=list(bestfeatures.get_support(True))\n"
      ],
      "execution_count": 500,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5b-zNqSxRq0n"
      },
      "source": [
        ""
      ],
      "execution_count": 500,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f95mLAPqX0IK"
      },
      "source": [
        ""
      ],
      "execution_count": 500,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yisAaqXMRpjg"
      },
      "source": [
        "#len(selected_feature_columns)"
      ],
      "execution_count": 501,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X9WXVZmmW7TM"
      },
      "source": [
        ""
      ],
      "execution_count": 501,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BtE3b6JEH7rw"
      },
      "source": [
        "# **Some Utilities**\n",
        "\n",
        "You do not need to modify this part."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FWMT3uf1NGQp"
      },
      "source": [
        "def get_device():\n",
        "    ''' Get device (if GPU is available, use GPU) '''\n",
        "    return 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "\n",
        "def plot_learning_curve(loss_record, title=''):\n",
        "    ''' Plot learning curve of your DNN (train & dev loss) '''\n",
        "    total_steps = len(loss_record['train'])\n",
        "    x_1 = range(total_steps)\n",
        "    x_2 = x_1[::len(loss_record['train']) // len(loss_record['dev'])]\n",
        "    figure(figsize=(6, 4))\n",
        "    plt.plot(x_1, loss_record['train'], c='tab:red', label='train')\n",
        "    plt.plot(x_2, loss_record['dev'], c='tab:cyan', label='dev')\n",
        "    plt.ylim(0.0, 5.)\n",
        "    plt.xlabel('Training steps')\n",
        "    plt.ylabel('MSE loss')\n",
        "    plt.title('Learning curve of {}'.format(title))\n",
        "    plt.legend()\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "def plot_pred(dv_set, model, device, lim=35., preds=None, targets=None):\n",
        "    ''' Plot prediction of your DNN '''\n",
        "    if preds is None or targets is None:\n",
        "        model.eval()\n",
        "        preds, targets = [], []\n",
        "        for x, y in dv_set:\n",
        "            x, y = x.to(device), y.to(device)\n",
        "            with torch.no_grad():\n",
        "                pred = model(x)\n",
        "                preds.append(pred.detach().cpu())\n",
        "                targets.append(y.detach().cpu())\n",
        "        preds = torch.cat(preds, dim=0).numpy()\n",
        "        targets = torch.cat(targets, dim=0).numpy()\n",
        "\n",
        "    figure(figsize=(5, 5))\n",
        "    plt.scatter(targets, preds, c='r', alpha=0.5)\n",
        "    plt.plot([-0.2, lim], [-0.2, lim], c='b')\n",
        "    plt.xlim(-0.2, lim)\n",
        "    plt.ylim(-0.2, lim)\n",
        "    plt.xlabel('ground truth value')\n",
        "    plt.ylabel('predicted value')\n",
        "    plt.title('Ground Truth v.s. Prediction')\n",
        "    plt.show()"
      ],
      "execution_count": 502,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "39U_XFX6KOoj"
      },
      "source": [
        "# **Preprocess**\n",
        "\n",
        "We have three kinds of datasets:\n",
        "* `train`: for training\n",
        "* `dev`: for validation\n",
        "* `test`: for testing (w/o target value)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TQ-MdwpLL7Dt"
      },
      "source": [
        "## **Dataset**\n",
        "\n",
        "The `COVID19Dataset` below does:\n",
        "* read `.csv` files\n",
        "* extract features\n",
        "* split `covid.train.csv` into train/dev sets\n",
        "* normalize features\n",
        "\n",
        "Finishing `TODO` below might make you pass medium baseline."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-JW_Osq-CXdI"
      },
      "source": [
        "#PRW\n",
        "with open(r'/content/covid.test.csv', 'r') as fp:\n",
        "    data = list(csv.reader(fp))\n",
        "    #data是一个由list组成的list，第一行是表头，之后的每一行是一个观测\n",
        "    data = np.array(data[1:])[:, 1:].astype(float)  #去掉表头行和id列"
      ],
      "execution_count": 503,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vJ4lUNqHDeU2",
        "outputId": "6f8548c8-1a12-4b31-c7f9-55f907edffe5"
      },
      "source": [
        "#PRW\n",
        "list_example=list(range(40))\n",
        "list_example.append(1)\n",
        "len(list_example)"
      ],
      "execution_count": 504,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "41"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 504
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0zlpIp9ANJRU"
      },
      "source": [
        "class COVID19Dataset(Dataset):\n",
        "    ''' Dataset for loading and preprocessing the COVID19 dataset '''\n",
        "    def __init__(self,\n",
        "                 path,\n",
        "                 mode='train',\n",
        "                 target_only=False):\n",
        "        #PRW：target_only=False就是simple baseline的情况，即直接使用全部93个特征硬上\n",
        "        self.mode = mode\n",
        "\n",
        "        # Read data into numpy arrays\n",
        "        with open(path, 'r') as fp:\n",
        "            data = list(csv.reader(fp))\n",
        "            #PRW：data是一个由list组成的list，第一行是表头，之后的每一行是一个观测\n",
        "            data = np.array(data[1:])[:, 1:].astype(float)  #PRW：去掉表头行和id列\n",
        "        \n",
        "        if not target_only:\n",
        "            feats = list(range(93))\n",
        "        else:\n",
        "            #medium baseline: Using 40 states & 2 tested_positive features (indices = 57 & 75)\n",
        "            feats=list(range(40))\n",
        "            feats.append(57)\n",
        "            feats.append(75)\n",
        "\n",
        "            #strong baseline调整特征\n",
        "            feats=list(set(feats+selected_feature_columns))\n",
        "            #feats=selected_feature_columns\n",
        "\n",
        "        if mode == 'test':\n",
        "            # Testing data\n",
        "            # data: 893 x len(feats)\n",
        "            data = data[:, feats]\n",
        "            self.data = torch.FloatTensor(data)\n",
        "        else:\n",
        "            # Training data (train/dev sets)\n",
        "            # data: 2700 x len(feats)\n",
        "            target = data[:, -1]\n",
        "            data = data[:, feats]\n",
        "            \n",
        "            # Splitting training data into train & dev sets\n",
        "            if mode == 'train':\n",
        "                indices = [i for i in range(len(data)) if i % 10 != 0]\n",
        "            elif mode == 'dev':\n",
        "                indices = [i for i in range(len(data)) if i % 10 == 0]\n",
        "            \n",
        "            # Convert data into PyTorch tensors\n",
        "            self.data = torch.FloatTensor(data[indices])\n",
        "            self.target = torch.FloatTensor(target[indices])\n",
        "\n",
        "        # Normalize features (you may remove this part to see what will happen)\n",
        "        self.data[:, 40:] = \\\n",
        "            (self.data[:, 40:] - self.data[:, 40:].mean(dim=0, keepdim=True)) \\\n",
        "            / self.data[:, 40:].std(dim=0, keepdim=True)\n",
        "\n",
        "        self.dim = self.data.shape[1]\n",
        "\n",
        "        print('Finished reading the {} set of COVID19 Dataset ({} samples found, each dim = {})'\n",
        "              .format(mode, len(self.data), self.dim))\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        # Returns one sample at a time\n",
        "        if self.mode in ['train', 'dev']:\n",
        "            # For training\n",
        "            return self.data[index], self.target[index]\n",
        "        else:\n",
        "            # For testing (no target)\n",
        "            return self.data[index]\n",
        "\n",
        "    def __len__(self):\n",
        "        # Returns the size of the dataset\n",
        "        return len(self.data)"
      ],
      "execution_count": 505,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AlhTlkE7MDo3"
      },
      "source": [
        "## **DataLoader**\n",
        "\n",
        "A `DataLoader` loads data from a given `Dataset` into batches.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hlhLk5t6MBX3"
      },
      "source": [
        "def prep_dataloader(path, mode, batch_size, n_jobs=0, target_only=False):\n",
        "    ''' Generates a dataset, then is put into a dataloader. '''\n",
        "    dataset = COVID19Dataset(path, mode=mode, target_only=target_only)  # Construct dataset\n",
        "    dataloader = DataLoader(\n",
        "        dataset, batch_size,\n",
        "        shuffle=(mode == 'train'), drop_last=False,\n",
        "        num_workers=n_jobs, pin_memory=True)  # Construct dataloader\n",
        "    return dataloader"
      ],
      "execution_count": 506,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SGuycwR0MeQB"
      },
      "source": [
        "# **Deep Neural Network**\n",
        "\n",
        "`NeuralNet` is an `nn.Module` designed for regression.\n",
        "The DNN consists of 2 fully-connected layers with ReLU activation.\n",
        "This module also included a function `cal_loss` for calculating loss.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "49-uXYovOAI0"
      },
      "source": [
        "class NeuralNet(nn.Module):\n",
        "    ''' A simple fully-connected deep neural network '''\n",
        "    def __init__(self, input_dim):\n",
        "        super(NeuralNet, self).__init__()\n",
        "\n",
        "        # Define your neural network here\n",
        "        self.net = nn.Sequential(\n",
        "            nn.Linear(input_dim, 47),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(47, 1)\n",
        "        )\n",
        "\n",
        "        # Mean squared error loss\n",
        "        self.criterion = nn.MSELoss(reduction='mean')\n",
        "\n",
        "    def forward(self, x):\n",
        "        ''' Given input of size (batch_size x input_dim), compute output of the network '''\n",
        "        return self.net(x).squeeze(1)\n",
        "\n",
        "    def cal_loss(self, pred, target):\n",
        "        ''' Calculate loss '''\n",
        "        # TODO: you may implement L1/L2 regularization here\n",
        "        return self.criterion(pred, target)"
      ],
      "execution_count": 555,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DvFWVjZ5Nvga"
      },
      "source": [
        "# **Train/Dev/Test**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MAM8QecJOyqn"
      },
      "source": [
        "## **Training**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lOqcmYzMO7jB"
      },
      "source": [
        "def train(tr_set, dv_set, model, config, device):\n",
        "    ''' DNN training '''\n",
        "\n",
        "    n_epochs = config['n_epochs']  # Maximum number of epochs\n",
        "\n",
        "    # Setup optimizer\n",
        "    optimizer = getattr(torch.optim, config['optimizer'])(\n",
        "        model.parameters(), **config['optim_hparas'])\n",
        "\n",
        "    min_mse = 1000.\n",
        "    loss_record = {'train': [], 'dev': []}      # for recording training loss\n",
        "    early_stop_cnt = 0\n",
        "    epoch = 0\n",
        "    while epoch < n_epochs:\n",
        "        model.train()                           # set model to training mode\n",
        "        for x, y in tr_set:                     # iterate through the dataloader\n",
        "            optimizer.zero_grad()               # set gradient to zero\n",
        "            x, y = x.to(device), y.to(device)   # move data to device (cpu/cuda)\n",
        "            pred = model(x)                     # forward pass (compute output)\n",
        "            mse_loss = model.cal_loss(pred, y)  # compute loss\n",
        "            mse_loss.backward()                 # compute gradient (backpropagation)\n",
        "            optimizer.step()                    # update model with optimizer\n",
        "            loss_record['train'].append(mse_loss.detach().cpu().item())\n",
        "\n",
        "        # After each epoch, test your model on the validation (development) set.\n",
        "        dev_mse = dev(dv_set, model, device)\n",
        "        if dev_mse < min_mse:\n",
        "            # Save model if your model improved\n",
        "            min_mse = dev_mse\n",
        "            print('Saving model (epoch = {:4d}, loss = {:.4f})'\n",
        "                .format(epoch + 1, min_mse))\n",
        "            torch.save(model.state_dict(), config['save_path'])  # Save model to specified path\n",
        "            early_stop_cnt = 0\n",
        "        else:\n",
        "            early_stop_cnt += 1\n",
        "\n",
        "        epoch += 1\n",
        "        loss_record['dev'].append(dev_mse)\n",
        "        if early_stop_cnt > config['early_stop']:\n",
        "            # Stop training if your model stops improving for \"config['early_stop']\" epochs.\n",
        "            break\n",
        "\n",
        "    print('Finished training after {} epochs'.format(epoch))\n",
        "    return min_mse, loss_record"
      ],
      "execution_count": 556,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0hSd4Bn3O2PL"
      },
      "source": [
        "## **Validation**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yrxrD3YsN3U2"
      },
      "source": [
        "def dev(dv_set, model, device):\n",
        "    model.eval()                                # set model to evalutation mode\n",
        "    total_loss = 0\n",
        "    for x, y in dv_set:                         # iterate through the dataloader\n",
        "        x, y = x.to(device), y.to(device)       # move data to device (cpu/cuda)\n",
        "        with torch.no_grad():                   # disable gradient calculation\n",
        "            pred = model(x)                     # forward pass (compute output)\n",
        "            mse_loss = model.cal_loss(pred, y)  # compute loss\n",
        "        total_loss += mse_loss.detach().cpu().item() * len(x)  # accumulate loss\n",
        "    total_loss = total_loss / len(dv_set.dataset)              # compute averaged loss\n",
        "\n",
        "    return total_loss"
      ],
      "execution_count": 557,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g0pdrhQAO41L"
      },
      "source": [
        "## **Testing**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aSBMRFlYN5tB"
      },
      "source": [
        "def test(tt_set, model, device):\n",
        "    model.eval()                                # set model to evalutation mode\n",
        "    preds = []\n",
        "    for x in tt_set:                            # iterate through the dataloader\n",
        "        x = x.to(device)                        # move data to device (cpu/cuda)\n",
        "        with torch.no_grad():                   # disable gradient calculation\n",
        "            pred = model(x)                     # forward pass (compute output)\n",
        "            preds.append(pred.detach().cpu())   # collect prediction\n",
        "    preds = torch.cat(preds, dim=0).numpy()     # concatenate all predictions and convert to a numpy array\n",
        "    return preds"
      ],
      "execution_count": 558,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SvckkF5dvf0j"
      },
      "source": [
        "# **Setup Hyper-parameters**\n",
        "\n",
        "`config` contains hyper-parameters for training and the path to save your model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NPXpdumwPjE7"
      },
      "source": [
        "device = get_device()                 # get the current available device ('cpu' or 'cuda')\n",
        "os.makedirs('models', exist_ok=True)  # The trained model will be saved to ./models/\n",
        "target_only = True                   # TODO: Using 40 states & 2 tested_positive features\n",
        "\n",
        "# TODO: How to tune these hyper-parameters to improve your model's performance?\n",
        "config = {\n",
        "    'n_epochs': 3000,                # maximum number of epochs\n",
        "    'batch_size': 100,               # mini-batch size for dataloader\n",
        "    'optimizer': 'SGD',              # optimization algorithm (optimizer in torch.optim)\n",
        "    'optim_hparas': {                # hyper-parameters for the optimizer (depends on which optimizer you are using)\n",
        "        'lr': 0.001,                 # learning rate of SGD\n",
        "        'momentum': 0.9              # momentum for SGD\n",
        "    },\n",
        "    'early_stop': 200,               # early stopping epochs (the number epochs since your model's last improvement)\n",
        "    'save_path': 'models/model.pth'  # your model will be saved here\n",
        "}"
      ],
      "execution_count": 627,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6j1eOV3TOH-j"
      },
      "source": [
        "# **Load data and model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eNrYBMmePLKm",
        "outputId": "de22b569-093e-4bd4-bd96-f5530be0f0a0"
      },
      "source": [
        "tr_set = prep_dataloader(tr_path, 'train', config['batch_size'], target_only=target_only)\n",
        "dv_set = prep_dataloader(tr_path, 'dev', config['batch_size'], target_only=target_only)\n",
        "tt_set = prep_dataloader(tt_path, 'test', config['batch_size'], target_only=target_only)"
      ],
      "execution_count": 628,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Finished reading the train set of COVID19 Dataset (2430 samples found, each dim = 60)\n",
            "Finished reading the dev set of COVID19 Dataset (270 samples found, each dim = 60)\n",
            "Finished reading the test set of COVID19 Dataset (893 samples found, each dim = 60)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FHylSirLP9oh"
      },
      "source": [
        "model = NeuralNet(tr_set.dataset.dim).to(device)  # Construct model and move to device"
      ],
      "execution_count": 629,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sX2B_zgSOPTJ"
      },
      "source": [
        "# **Start Training!**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GrEbUxazQAAZ",
        "outputId": "20e3fe71-0c85-4645-f606-ef3cd8068abe"
      },
      "source": [
        "model_loss, model_loss_record = train(tr_set, dv_set, model, config, device)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving model (epoch =    1, loss = 268.1602)\n",
            "Saving model (epoch =    2, loss = 94.9778)\n",
            "Saving model (epoch =    3, loss = 71.3720)\n",
            "Saving model (epoch =    4, loss = 53.1429)\n",
            "Saving model (epoch =    5, loss = 39.8516)\n",
            "Saving model (epoch =    6, loss = 29.9270)\n",
            "Saving model (epoch =    7, loss = 21.5908)\n",
            "Saving model (epoch =    8, loss = 15.6021)\n",
            "Saving model (epoch =    9, loss = 11.5713)\n",
            "Saving model (epoch =   10, loss = 8.7897)\n",
            "Saving model (epoch =   11, loss = 6.7749)\n",
            "Saving model (epoch =   12, loss = 5.3356)\n",
            "Saving model (epoch =   13, loss = 4.3302)\n",
            "Saving model (epoch =   14, loss = 3.6346)\n",
            "Saving model (epoch =   15, loss = 3.1028)\n",
            "Saving model (epoch =   16, loss = 2.7002)\n",
            "Saving model (epoch =   17, loss = 2.3759)\n",
            "Saving model (epoch =   18, loss = 2.1693)\n",
            "Saving model (epoch =   19, loss = 1.9399)\n",
            "Saving model (epoch =   20, loss = 1.7955)\n",
            "Saving model (epoch =   21, loss = 1.7110)\n",
            "Saving model (epoch =   22, loss = 1.6355)\n",
            "Saving model (epoch =   23, loss = 1.5441)\n",
            "Saving model (epoch =   24, loss = 1.5114)\n",
            "Saving model (epoch =   25, loss = 1.4592)\n",
            "Saving model (epoch =   26, loss = 1.4180)\n",
            "Saving model (epoch =   27, loss = 1.3976)\n",
            "Saving model (epoch =   28, loss = 1.3610)\n",
            "Saving model (epoch =   29, loss = 1.3475)\n",
            "Saving model (epoch =   30, loss = 1.3236)\n",
            "Saving model (epoch =   32, loss = 1.3086)\n",
            "Saving model (epoch =   33, loss = 1.2904)\n",
            "Saving model (epoch =   34, loss = 1.2860)\n",
            "Saving model (epoch =   35, loss = 1.2700)\n",
            "Saving model (epoch =   36, loss = 1.2639)\n",
            "Saving model (epoch =   37, loss = 1.2419)\n",
            "Saving model (epoch =   39, loss = 1.2373)\n",
            "Saving model (epoch =   40, loss = 1.2131)\n",
            "Saving model (epoch =   41, loss = 1.2089)\n",
            "Saving model (epoch =   42, loss = 1.2083)\n",
            "Saving model (epoch =   43, loss = 1.1956)\n",
            "Saving model (epoch =   44, loss = 1.1896)\n",
            "Saving model (epoch =   45, loss = 1.1766)\n",
            "Saving model (epoch =   47, loss = 1.1697)\n",
            "Saving model (epoch =   48, loss = 1.1622)\n",
            "Saving model (epoch =   50, loss = 1.1469)\n",
            "Saving model (epoch =   53, loss = 1.1356)\n",
            "Saving model (epoch =   55, loss = 1.1260)\n",
            "Saving model (epoch =   57, loss = 1.1227)\n",
            "Saving model (epoch =   58, loss = 1.1174)\n",
            "Saving model (epoch =   59, loss = 1.1167)\n",
            "Saving model (epoch =   60, loss = 1.1117)\n",
            "Saving model (epoch =   61, loss = 1.1084)\n",
            "Saving model (epoch =   62, loss = 1.1067)\n",
            "Saving model (epoch =   64, loss = 1.1031)\n",
            "Saving model (epoch =   65, loss = 1.0952)\n",
            "Saving model (epoch =   66, loss = 1.0944)\n",
            "Saving model (epoch =   67, loss = 1.0889)\n",
            "Saving model (epoch =   68, loss = 1.0801)\n",
            "Saving model (epoch =   71, loss = 1.0778)\n",
            "Saving model (epoch =   72, loss = 1.0674)\n",
            "Saving model (epoch =   73, loss = 1.0651)\n",
            "Saving model (epoch =   77, loss = 1.0634)\n",
            "Saving model (epoch =   78, loss = 1.0543)\n",
            "Saving model (epoch =   81, loss = 1.0433)\n",
            "Saving model (epoch =   82, loss = 1.0410)\n",
            "Saving model (epoch =   85, loss = 1.0406)\n",
            "Saving model (epoch =   86, loss = 1.0321)\n",
            "Saving model (epoch =   91, loss = 1.0316)\n",
            "Saving model (epoch =   92, loss = 1.0234)\n",
            "Saving model (epoch =   94, loss = 1.0188)\n",
            "Saving model (epoch =   96, loss = 1.0121)\n",
            "Saving model (epoch =   98, loss = 1.0104)\n",
            "Saving model (epoch =  101, loss = 1.0053)\n",
            "Saving model (epoch =  103, loss = 1.0012)\n",
            "Saving model (epoch =  105, loss = 1.0011)\n",
            "Saving model (epoch =  107, loss = 1.0005)\n",
            "Saving model (epoch =  110, loss = 0.9956)\n",
            "Saving model (epoch =  111, loss = 0.9954)\n",
            "Saving model (epoch =  114, loss = 0.9861)\n",
            "Saving model (epoch =  115, loss = 0.9838)\n",
            "Saving model (epoch =  118, loss = 0.9808)\n",
            "Saving model (epoch =  123, loss = 0.9790)\n",
            "Saving model (epoch =  125, loss = 0.9757)\n",
            "Saving model (epoch =  126, loss = 0.9684)\n",
            "Saving model (epoch =  130, loss = 0.9659)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hsNO9nnXQBvP"
      },
      "source": [
        "plot_learning_curve(model_loss_record, title='deep model')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3iZTVn5WQFpX"
      },
      "source": [
        "del model\n",
        "model = NeuralNet(tr_set.dataset.dim).to(device)\n",
        "ckpt = torch.load(config['save_path'], map_location='cpu')  # Load your best model\n",
        "model.load_state_dict(ckpt)\n",
        "plot_pred(dv_set, model, device)  # Show prediction on the validation set"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aQikz3IPiyPf"
      },
      "source": [
        "# **Testing**\n",
        "The predictions of your model on testing set will be stored at `pred.csv`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O8cTuQjQQOon"
      },
      "source": [
        "def save_pred(preds, file):\n",
        "    ''' Save predictions to specified file '''\n",
        "    print('Saving results to {}'.format(file))\n",
        "    with open(file, 'w') as fp:\n",
        "        writer = csv.writer(fp)\n",
        "        writer.writerow(['id', 'tested_positive'])\n",
        "        for i, p in enumerate(preds):\n",
        "            writer.writerow([i, p])\n",
        "\n",
        "preds = test(tt_set, model, device)  # predict COVID-19 cases with your model\n",
        "save_pred(preds, 'pred.csv')         # save prediction file to pred.csv"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8kdpNPimq_ka"
      },
      "source": [
        "x=1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nfrVxqJanGpE"
      },
      "source": [
        "# **Hints**\n",
        "\n",
        "## **Simple Baseline**\n",
        "* Run sample code\n",
        "\n",
        "## **Medium Baseline**\n",
        "* Feature selection: 40 states + 2 `tested_positive` (`TODO` in dataset)\n",
        "\n",
        "## **Strong Baseline**\n",
        "* Feature selection (what other features are useful?)\n",
        "* DNN architecture (layers? dimension? activation function?)\n",
        "* Training (mini-batch? optimizer? learning rate?)\n",
        "* L2 regularization\n",
        "* There are some mistakes in the sample code, can you find them?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9tmCwXgpot3t"
      },
      "source": [
        "# **Reference**\n",
        "This code is completely written by Heng-Jui Chang @ NTUEE.  \n",
        "Copying or reusing this code is required to specify the original author. \n",
        "\n",
        "E.g.  \n",
        "Source: Heng-Jui Chang @ NTUEE (https://github.com/ga642381/ML2021-Spring/blob/main/HW01/HW01.ipynb)\n"
      ]
    }
  ]
}