hw/hw2/hw2_code.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "hw2_(1) (1).ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lcS4CQrKGLYA"
      },
      "source": [
        "# **Homework 2-1 Phoneme Classification**\n",
        "\n",
        "* Slides: https://speech.ee.ntu.edu.tw/~hylee/ml/ml2021-course-data/hw/HW02/HW02.pdf\n",
        "* Video (Chinese): https://youtu.be/PdjXnQbu2zo\n",
        "* Video (English): https://youtu.be/ESRr-VCykBs\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XcrJY3SDGRz0"
      },
      "source": [
        "# **The DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus (TIMIT)**\n",
        "The TIMIT corpus of reading speech has been designed to provide speech data for the acquisition of acoustic-phonetic knowledge and for the development and evaluation of automatic speech recognition systems.\n",
        "\n",
        "This homework is a multiclass classification task, \n",
        "we are going to train a deep neural network classifier to predict the phonemes for each frame from the speech corpus TIMIT.\n",
        "\n",
        "link: https://academictorrents.com/details/34e2b78745138186976cbc27939b1b34d18bd5b3"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uewNjZjGXZpZ"
      },
      "source": [
        "# **基於baseline的分析與改善**\n",
        "\n",
        "* **採樣**\n",
        "  * 做數據分析的時候，發現不同class的數據量差異很大，最多跟最少可以差到40倍，而觀察valid data的錯誤率，同樣發現數據量少的class判別的正確率更低\n",
        "  * 在數據少的class上做過採樣，原本是想把每一個class的data都擴展到十萬筆，保證數據量相差不大，後來礙於colab的ram有限，只把少的過採樣到一萬筆\n",
        "  * 後來聽說pytorch有內建的sampler，但跑一次要一個多小時，就沒有修改，相信更平均的數據效果會更好\n",
        "* **Optimizer**\n",
        "  * 先使用收斂速度快的adam，收斂到一定程度後改用穩定的SGDM\n",
        "  * 基本上是參考SWATS的邏輯，但切換的點是自己決定的\n",
        "* **激活函數**\n",
        "  * 用ReLU，效果比sigmioid好得多\n",
        "* **network架構**\n",
        "  * 加上batch normalization，加速收斂，為了避免batch不能代表數據分佈，將batch size調大\n",
        "  * 加上dropout，雖然batch normalization一定程度上可以替代dropout的效果，但還是有使\b用\n",
        "* **Train跟Valid dataset切分**\n",
        "  * 改為隨機分佈，保證資料分佈相同\n",
        "  * 事實上，沒有什麼差別，原本的資料分布好像已經夠亂了\n",
        "* **L1/L2正則，避免overfitting**\n",
        "\n",
        "# **Result**\n",
        "  * Public: 0.75487\n",
        "  * Private: 0.75484\n",
        "  * 離strong baseline還有0.006的差距，或許可以透過平均採樣跟增加epoch達成\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PuTiV-5VGZsl"
      },
      "source": [
        "# **Download Data**\n",
        "Download data from google drive, then unzip it.\n",
        "\n",
        "You should have `timit_11/train_11.npy`, `timit_11/train_label_11.npy`, and `timit_11/test_11.npy` after running this block.<br><br>\n",
        "`timit_11/`\n",
        "- `train_11.npy`: training data<br>\n",
        "- `train_label_11.npy`: training label<br>\n",
        "- `test_11.npy`:  testing data<br><br>\n",
        "\n",
        "**notes: if the google drive link is dead, you can download the data directly from Kaggle and upload it to the workspace**\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9iT39_2uClcy",
        "outputId": "79318feb-3480-4d2a-ff7f-0a807e80beaf"
      },
      "source": [
        "!gdown --id \"1HPkcmQmFGu-3OknddKIa5dNDsR05lIQR\" --output data.zip\n",
        "!unzip data.zip\n",
        "!ls "
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1HPkcmQmFGu-3OknddKIa5dNDsR05lIQR\n",
            "To: /content/data.zip\n",
            "372MB [00:02, 125MB/s]\n",
            "Archive:  data.zip\n",
            "   creating: timit_11/\n",
            "  inflating: timit_11/train_11.npy   \n",
            "  inflating: timit_11/test_11.npy    \n",
            "  inflating: timit_11/train_label_11.npy  \n",
            "data.zip  sample_data  timit_11\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yU2B8RgCGpa7"
      },
      "source": [
        "# **Import Packages**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cz2bAEzmIIG_"
      },
      "source": [
        "# PyTorch\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.utils.data import Dataset, DataLoader\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# For data preprocess\n",
        "import numpy as np\n",
        "import csv\n",
        "import os\n",
        "\n",
        "# Utility\n",
        "import gc\n",
        "\n",
        "my_seed = 0\n",
        "torch.backends.cudnn.deterministic = True\n",
        "torch.backends.cudnn.benchmark = False\n",
        "np.random.seed(my_seed)\n",
        "torch.manual_seed(my_seed)\n",
        "if torch.cuda.is_available():\n",
        "    torch.cuda.manual_seed_all(my_seed)"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e3ZAb_xpG2f2"
      },
      "source": [
        "# **Download Data**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o1Nk6S5-CqKl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a07c9967-e784-41ed-9999-dc11c692f9a1"
      },
      "source": [
        "print (\"Loading data ...\")\n",
        "\n",
        "data_root = \"./timit_11/\"\n",
        "train_data = np.load(data_root + \"train_11.npy\")\n",
        "train_label = np.load(data_root + \"train_label_11.npy\")\n",
        "test_data = np.load(data_root + \"test_11.npy\")\n",
        "\n",
        "print(\"Size of training data: {}\".format(train_data.shape))\n",
        "print(\"Size of testing data: {}\".format(test_data.shape))"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loading data ...\n",
            "Size of training data: (1229932, 429)\n",
            "Size of testing data: (451552, 429)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AlPAAK1zG9I1"
      },
      "source": [
        "# **Data Analysis**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c3eOg1QNDNF9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dba7f829-6c73-444a-b9d2-50c877fe5d22"
      },
      "source": [
        "# 統計每個類別的數量\n",
        "print (\"Total number {:d}\".format(train_label.shape[0]))\n",
        "\n",
        "train_cnt = np.zeros((39), dtype=int)\n",
        "for i in range(39):\n",
        "    train_cnt[i] = np.sum(train_label == str(i))\n",
        "\n",
        "sum = np.sum(train_cnt)\n",
        "print (\"\\n   class   count    rate\")\n",
        "for i in range(39):\n",
        "    print (\"{:8d}\".format(i), end='')\n",
        "    print (\"{:8d}\".format(train_cnt[i]), end='')\n",
        "    print (\"  {:.4f}\".format(train_cnt[i] / sum))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Total number 1229932\n",
            "\n",
            "   class   count    rate\n",
            "       0   62708  0.0510\n",
            "       1   83746  0.0681\n",
            "       2   35048  0.0285\n",
            "       3   59031  0.0480\n",
            "       4   38930  0.0317\n",
            "       5   26380  0.0214\n",
            "       6    4038  0.0033\n",
            "       7   73827  0.0600\n",
            "       8   28797  0.0234\n",
            "       9   34289  0.0279\n",
            "      10   11028  0.0090\n",
            "      11   11711  0.0095\n",
            "      12   26790  0.0218\n",
            "      13   43410  0.0353\n",
            "      14   39583  0.0322\n",
            "      15   11342  0.0092\n",
            "      16   20922  0.0170\n",
            "      17   51533  0.0419\n",
            "      18   24938  0.0203\n",
            "      19   47059  0.0383\n",
            "      20    8508  0.0069\n",
            "      21    7083  0.0058\n",
            "      22    7050  0.0057\n",
            "      23   10663  0.0087\n",
            "      24    3883  0.0032\n",
            "      25    8219  0.0067\n",
            "      26    7825  0.0064\n",
            "      27    6059  0.0049\n",
            "      28   11492  0.0093\n",
            "      29   21012  0.0171\n",
            "      30   25094  0.0204\n",
            "      31   31618  0.0257\n",
            "      32   12003  0.0098\n",
            "      33   22907  0.0186\n",
            "      34    6920  0.0056\n",
            "      35   84521  0.0687\n",
            "      36   27088  0.0220\n",
            "      37   14164  0.0115\n",
            "      38  178713  0.1453\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SEDRaijXHLGb"
      },
      "source": [
        "# **Dataset**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZY3N4d_BGQg3"
      },
      "source": [
        "class TIMITDataset(Dataset):\n",
        "    def __init__(self, x, y=None):\n",
        "        self.x = torch.from_numpy(x).float()\n",
        "        if y is not None:\n",
        "            y = y.astype(np.int)\n",
        "            self.y = torch.LongTensor(y)\n",
        "        else:\n",
        "            self.y = None\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        if self.y is not None:\n",
        "            return self.x[index], self.y[index]\n",
        "        else:\n",
        "            return self.x[index]\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.x)\n"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sVfUOD45HSPe"
      },
      "source": [
        "# **Setup Hyper-parameters**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KdtxzenUAbN8"
      },
      "source": [
        "# 超參數\n",
        "valid_rate = 0.01           \n",
        "num_epoch = 100             \n",
        "learning_rate = 0.0001      \n",
        "weight_decay_l1 = 0.0\n",
        "weight_decay_l2 = 0.001\n",
        "batch_size = 2048           # 原本設64，跑起來更慢，且沒辦法代表分佈\n",
        "number = 10000              # 每個類別最少補到一萬筆data\n",
        "\n",
        "model_path = \"./model.ckpt\""
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W2PIs43JHZAG"
      },
      "source": [
        "# **Shuffle Training and Validation Data**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UFs29VPfHNQX"
      },
      "source": [
        "# 打亂分佈，隨機選取training data跟valid data\n",
        "train_indices, valid_indices = train_test_split([i for i in range(train_data.shape[0])], test_size=valid_rate, random_state=1)\n",
        "train_x = train_data[train_indices, :]\n",
        "train_y = train_label[train_indices]\n",
        "valid_x = train_data[valid_indices, :]\n",
        "valid_y = train_label[valid_indices]"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pu9BKvlYImie"
      },
      "source": [
        "# **Data Process**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6L76t-nmKVCd"
      },
      "source": [
        "# 紀錄每個類的id，方便sample\n",
        "train_class = []\n",
        "id = np.arange(train_x.shape[0])\n",
        "for i in range(39):\n",
        "    train_class.append(id[train_y == str(i)])"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2V1gaxWPI8C1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7232203d-b57c-48c2-ff83-78bad6823f17"
      },
      "source": [
        "del train_data, train_label\n",
        "gc.collect()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "252"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wkhdhRMqIs95"
      },
      "source": [
        "# **Deep Neural Network**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6QJSxlqtQXcz"
      },
      "source": [
        "class Classifier(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Classifier, self).__init__()\n",
        "\n",
        "        self.layer1 = nn.Linear(429, 2048)\n",
        "        self.layer2 = nn.Linear(2048, 2048)\n",
        "        self.layer3 = nn.Linear(2048, 2048)\n",
        "        self.layer4 = nn.Linear(2048, 1024)\n",
        "        self.layer5 = nn.Linear(1024, 512)\n",
        "        self.layer6 = nn.Linear(512, 128)\n",
        "\n",
        "        self.bn1 = nn.BatchNorm1d(2048)\n",
        "        self.bn2 = nn.BatchNorm1d(2048)\n",
        "        self.bn3 = nn.BatchNorm1d(2048)\n",
        "        self.bn4 = nn.BatchNorm1d(1024)\n",
        "        self.bn5 = nn.BatchNorm1d(512)\n",
        "        self.bn6 = nn.BatchNorm1d(128)\n",
        "\n",
        "        self.out = nn.Linear(128, 39) \n",
        "        \n",
        "        self.drop = nn.Dropout(0.5)\n",
        "        self.act_fn = nn.ReLU()\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.layer1(x)\n",
        "        x = self.act_fn(x)\n",
        "        x = self.bn1(x)\n",
        "        x = self.drop(x)\n",
        "\n",
        "        x = self.layer2(x)\n",
        "        x = self.act_fn(x)\n",
        "        x = self.bn2(x)\n",
        "        x = self.drop(x)\n",
        "\n",
        "        x = self.layer3(x)\n",
        "        x = self.act_fn(x)\n",
        "        x = self.bn3(x)\n",
        "        x = self.drop(x)\n",
        "\n",
        "        x = self.layer4(x)\n",
        "        x = self.act_fn(x)\n",
        "        x = self.bn4(x)\n",
        "        x = self.drop(x)\n",
        "\n",
        "        x = self.layer5(x)\n",
        "        x = self.act_fn(x)\n",
        "        x = self.bn5(x)\n",
        "        x = self.drop(x)\n",
        "\n",
        "        x = self.layer6(x)\n",
        "        x = self.act_fn(x)\n",
        "        x = self.bn6(x)\n",
        "        x = self.drop(x)\n",
        "\n",
        "        x = self.out(x)\n",
        "        \n",
        "        return x"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B7u_PUAEI0Gc"
      },
      "source": [
        "# **Utility**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6qGhyYRqQtBD"
      },
      "source": [
        "# check device\n",
        "def get_device():\n",
        "    return \"cuda\" if torch.cuda.is_available() else \"cpu\""
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GtuC2rjYI9GR"
      },
      "source": [
        "# **Training**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JjfE1-_XTKYz"
      },
      "source": [
        "def cal_regularization(model, weight_decay_l1, weight_decay_l2):\n",
        "    l1 = 0\n",
        "    l2 = 0\n",
        "    for i in model.parameters():\n",
        "        l1 += torch.sum(abs(i))\n",
        "        l2 += torch.sum(torch.pow(i, 2))\n",
        "    return weight_decay_l1 * l1 + weight_decay_l2 * l2\n",
        "\n",
        "def train_model(num_epoch, learning_rate, weight_decay_l1, weight_decay_l2,\n",
        "                train_dataset, train_dataloader,\n",
        "                valid_dataset, valid_dataloader):\n",
        "    model = Classifier().to(device)\n",
        "    criterion = nn.CrossEntropyLoss()\n",
        "\n",
        "    best_acc = 0.0\n",
        "    for epoch in range(num_epoch):\n",
        "        # 前面使用adam，收斂快，後面使用SGDM，穩定且偏差小\n",
        "        if epoch == 0:\n",
        "            optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
        "        elif epoch == 35:\n",
        "            optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9)\n",
        "\n",
        "        train_acc = 0.0\n",
        "        train_loss = 0.0\n",
        "        val_acc = 0.0\n",
        "        val_loss = 0.0\n",
        "\n",
        "        # training\n",
        "        model.train() # set the model to training mode\n",
        "        for i, data in enumerate(train_dataloader):\n",
        "            inputs, labels = data\n",
        "            inputs, labels = inputs.to(device), labels.to(device)\n",
        "    \n",
        "            optimizer.zero_grad() \n",
        "            outputs = model(inputs) \n",
        "    \n",
        "            batch_loss = criterion(outputs, labels)\n",
        "            _, train_pred = torch.max(outputs, 1) # get the index of the class with the highest probability\n",
        "            (batch_loss + cal_regularization(model, weight_decay_l1, weight_decay_l2)).backward() \n",
        "\n",
        "            optimizer.step() \n",
        "    \n",
        "            train_acc += (train_pred.cpu() == labels.cpu()).sum().item()\n",
        "            train_loss += batch_loss.item()\n",
        "    \n",
        "        # validation\n",
        "        if len(valid_dataset) > 0:\n",
        "            model.eval() # set the model to evaluation mode\n",
        "            with torch.no_grad():\n",
        "                for i, data in enumerate(valid_dataloader):\n",
        "                    inputs, labels = data\n",
        "                    inputs, labels = inputs.to(device), labels.to(device)\n",
        "                    outputs = model(inputs)\n",
        "                    batch_loss = criterion(outputs, labels) \n",
        "                    _, val_pred = torch.max(outputs, 1) \n",
        "                \n",
        "                    val_acc += (val_pred.cpu() == labels.cpu()).sum().item() # get the index of the class with the highest probability\n",
        "                    val_loss += batch_loss.item()\n",
        "    \n",
        "                print (\"[{:03d}/{:03d}] Train Acc: {:3.6f} Loss: {:3.6f} | Val Acc: {:3.6f} loss: {:3.6f}\".format(\n",
        "                    epoch + 1, num_epoch, train_acc / len(train_dataset), train_loss / len(train_dataloader), val_acc / len(valid_dataset), val_loss / len(valid_dataloader)\n",
        "                ))\n",
        "    \n",
        "                # if the model improves, save a checkpoint at this epoch\n",
        "                if val_acc > best_acc:\n",
        "                    best_acc = val_acc\n",
        "                    torch.save(model.state_dict(), model_path)\n",
        "                    print (\"saving model with acc {:.3f}\".format(best_acc / len(valid_dataset)))\n",
        "        else:\n",
        "            print(\"[{:03d}/{:03d}] Train Acc: {:3.6f} Loss: {:3.6f}\".format(\n",
        "                epoch + 1, num_epoch, train_acc / len(train_dataset), train_loss / len(train_dataloaders)\n",
        "            ))\n",
        "\n",
        "    # if not validating, save the last epoch\n",
        "    if len(valid_dataset) == 0:\n",
        "        torch.save(model.state_dict(), model_path)\n",
        "        print(\"saving model at last epoch\")"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5tY6xgYIAu32",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e97e5409-6410-41ce-8bea-984c1a2614c0"
      },
      "source": [
        "# get device \n",
        "device = get_device()\n",
        "print(f\"DEVICE: {device}\")"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "DEVICE: cuda\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vuWRBPDBAzrw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "27ab40f1-74ba-4782-8d84-f108f931a035"
      },
      "source": [
        "valid_dataset = TIMITDataset(valid_x, valid_y)\n",
        "valid_dataloader = DataLoader(valid_dataset, batch_size=batch_size, shuffle=False)\n",
        "\n",
        "del valid_x, valid_y\n",
        "gc.collect()"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "100"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v1YXlr3yA5Qm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e1dbf229-1d64-4737-bc9d-12e709346b91"
      },
      "source": [
        "print (\"Sample data:\")\n",
        "print (\"\\n   class   count\")\n",
        "for i in range(len(train_class)):\n",
        "    if (train_class[i].shape[0] < number):\n",
        "        print (\"{:8d}\".format(i), end='')\n",
        "        print (\"{:8d}\".format(number - train_class[i].shape[0]))\n",
        "\n",
        "        id = np.random.choice(train_class[i], size=number-train_class[i].shape[0])\n",
        "        train_x = np.vstack((train_x, train_x[id]))\n",
        "        label = np.empty((id.shape[0]), dtype=int)\n",
        "        train_y = np.append(train_y, label)\n",
        "        train_y[-id.shape[0]:] = int(i)\n",
        "\n",
        "print (\"\\n\", train_x.shape, train_y.shape)\n",
        "train_dataset = TIMITDataset(train_x, train_y)\n",
        "train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, drop_last=True)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sample data:\n",
            "\n",
            "   class   count\n",
            "       6    6001\n",
            "      20    1575\n",
            "      21    3004\n",
            "      22    3022\n",
            "      24    6158\n",
            "      25    1860\n",
            "      26    2261\n",
            "      27    3995\n",
            "      34    3140\n",
            "\n",
            " (1248648, 429) (1248648,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "giCHzVWXTZTV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fab8cf71-c376-42f6-9bdb-4896dcf8295f"
      },
      "source": [
        "train_model(num_epoch, learning_rate, weight_decay_l1, weight_decay_l2, train_dataset, train_dataloader, valid_dataset, valid_dataloader)\n",
        "\n",
        "del train_x, train_y, train_dataset, train_dataloader, valid_dataset, valid_dataloader\n",
        "gc.collect()"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[001/100] Train Acc: 0.389344 Loss: 2.256582 | Val Acc: 0.565285 loss: 1.588015\n",
            "saving model with acc 0.565\n",
            "[002/100] Train Acc: 0.536640 Loss: 1.613519 | Val Acc: 0.634634 loss: 1.338159\n",
            "saving model with acc 0.635\n",
            "[003/100] Train Acc: 0.587108 Loss: 1.419193 | Val Acc: 0.672927 loss: 1.192538\n",
            "saving model with acc 0.673\n",
            "[004/100] Train Acc: 0.616298 Loss: 1.308941 | Val Acc: 0.691545 loss: 1.121740\n",
            "saving model with acc 0.692\n",
            "[005/100] Train Acc: 0.634932 Loss: 1.238852 | Val Acc: 0.705122 loss: 1.062977\n",
            "saving model with acc 0.705\n",
            "[006/100] Train Acc: 0.647512 Loss: 1.190279 | Val Acc: 0.715203 loss: 1.054551\n",
            "saving model with acc 0.715\n",
            "[007/100] Train Acc: 0.657168 Loss: 1.154711 | Val Acc: 0.720976 loss: 1.004870\n",
            "saving model with acc 0.721\n",
            "[008/100] Train Acc: 0.664730 Loss: 1.124434 | Val Acc: 0.726179 loss: 0.972580\n",
            "saving model with acc 0.726\n",
            "[009/100] Train Acc: 0.671096 Loss: 1.101223 | Val Acc: 0.731138 loss: 0.968435\n",
            "saving model with acc 0.731\n",
            "[010/100] Train Acc: 0.676651 Loss: 1.080788 | Val Acc: 0.735285 loss: 0.941801\n",
            "saving model with acc 0.735\n",
            "[011/100] Train Acc: 0.681672 Loss: 1.061125 | Val Acc: 0.739593 loss: 0.943365\n",
            "saving model with acc 0.740\n",
            "[012/100] Train Acc: 0.686473 Loss: 1.044287 | Val Acc: 0.743902 loss: 0.931258\n",
            "saving model with acc 0.744\n",
            "[013/100] Train Acc: 0.690360 Loss: 1.029013 | Val Acc: 0.747724 loss: 0.880717\n",
            "saving model with acc 0.748\n",
            "[014/100] Train Acc: 0.694887 Loss: 1.013947 | Val Acc: 0.750000 loss: 0.881336\n",
            "saving model with acc 0.750\n",
            "[015/100] Train Acc: 0.698133 Loss: 1.002385 | Val Acc: 0.754309 loss: 0.872487\n",
            "saving model with acc 0.754\n",
            "[016/100] Train Acc: 0.701507 Loss: 0.989204 | Val Acc: 0.756179 loss: 0.853143\n",
            "saving model with acc 0.756\n",
            "[017/100] Train Acc: 0.703970 Loss: 0.979489 | Val Acc: 0.761789 loss: 0.828186\n",
            "saving model with acc 0.762\n",
            "[018/100] Train Acc: 0.706209 Loss: 0.970606 | Val Acc: 0.762033 loss: 0.827178\n",
            "saving model with acc 0.762\n",
            "[019/100] Train Acc: 0.708866 Loss: 0.961312 | Val Acc: 0.762846 loss: 0.841049\n",
            "saving model with acc 0.763\n",
            "[020/100] Train Acc: 0.710563 Loss: 0.954998 | Val Acc: 0.768537 loss: 0.812924\n",
            "saving model with acc 0.769\n",
            "[021/100] Train Acc: 0.712929 Loss: 0.946723 | Val Acc: 0.764797 loss: 0.857419\n",
            "[022/100] Train Acc: 0.715120 Loss: 0.938514 | Val Acc: 0.769675 loss: 0.812915\n",
            "saving model with acc 0.770\n",
            "[023/100] Train Acc: 0.715529 Loss: 0.934974 | Val Acc: 0.773659 loss: 0.826160\n",
            "saving model with acc 0.774\n",
            "[024/100] Train Acc: 0.717359 Loss: 0.928576 | Val Acc: 0.773740 loss: 0.773099\n",
            "saving model with acc 0.774\n",
            "[025/100] Train Acc: 0.717961 Loss: 0.924665 | Val Acc: 0.774146 loss: 0.796645\n",
            "saving model with acc 0.774\n",
            "[026/100] Train Acc: 0.719919 Loss: 0.919602 | Val Acc: 0.776179 loss: 0.789958\n",
            "saving model with acc 0.776\n",
            "[027/100] Train Acc: 0.721333 Loss: 0.915037 | Val Acc: 0.776667 loss: 0.784889\n",
            "saving model with acc 0.777\n",
            "[028/100] Train Acc: 0.722454 Loss: 0.910882 | Val Acc: 0.779187 loss: 0.762660\n",
            "saving model with acc 0.779\n",
            "[029/100] Train Acc: 0.722877 Loss: 0.907737 | Val Acc: 0.775203 loss: 0.780451\n",
            "[030/100] Train Acc: 0.724137 Loss: 0.903159 | Val Acc: 0.779593 loss: 0.753174\n",
            "saving model with acc 0.780\n",
            "[031/100] Train Acc: 0.724229 Loss: 0.901828 | Val Acc: 0.778537 loss: 0.781070\n",
            "[032/100] Train Acc: 0.725578 Loss: 0.898495 | Val Acc: 0.779431 loss: 0.763686\n",
            "[033/100] Train Acc: 0.726242 Loss: 0.894566 | Val Acc: 0.780569 loss: 0.763771\n",
            "saving model with acc 0.781\n",
            "[034/100] Train Acc: 0.727033 Loss: 0.891935 | Val Acc: 0.783171 loss: 0.770577\n",
            "saving model with acc 0.783\n",
            "[035/100] Train Acc: 0.727874 Loss: 0.888978 | Val Acc: 0.781545 loss: 0.728925\n",
            "[036/100] Train Acc: 0.736030 Loss: 0.861019 | Val Acc: 0.786748 loss: 0.719773\n",
            "saving model with acc 0.787\n",
            "[037/100] Train Acc: 0.739819 Loss: 0.847752 | Val Acc: 0.788618 loss: 0.718324\n",
            "saving model with acc 0.789\n",
            "[038/100] Train Acc: 0.742351 Loss: 0.839852 | Val Acc: 0.789187 loss: 0.717786\n",
            "saving model with acc 0.789\n",
            "[039/100] Train Acc: 0.744196 Loss: 0.833240 | Val Acc: 0.790813 loss: 0.716452\n",
            "saving model with acc 0.791\n",
            "[040/100] Train Acc: 0.745919 Loss: 0.829431 | Val Acc: 0.792358 loss: 0.713735\n",
            "saving model with acc 0.792\n",
            "[041/100] Train Acc: 0.747427 Loss: 0.823772 | Val Acc: 0.792358 loss: 0.710393\n",
            "[042/100] Train Acc: 0.747614 Loss: 0.821676 | Val Acc: 0.793496 loss: 0.704609\n",
            "saving model with acc 0.793\n",
            "[043/100] Train Acc: 0.748441 Loss: 0.819105 | Val Acc: 0.793659 loss: 0.703914\n",
            "saving model with acc 0.794\n",
            "[044/100] Train Acc: 0.749827 Loss: 0.814412 | Val Acc: 0.793171 loss: 0.702027\n",
            "[045/100] Train Acc: 0.750403 Loss: 0.811880 | Val Acc: 0.795041 loss: 0.699948\n",
            "saving model with acc 0.795\n",
            "[046/100] Train Acc: 0.750994 Loss: 0.809760 | Val Acc: 0.795772 loss: 0.698347\n",
            "saving model with acc 0.796\n",
            "[047/100] Train Acc: 0.752117 Loss: 0.807050 | Val Acc: 0.795691 loss: 0.697200\n",
            "[048/100] Train Acc: 0.752713 Loss: 0.804286 | Val Acc: 0.795610 loss: 0.697433\n",
            "[049/100] Train Acc: 0.753739 Loss: 0.801283 | Val Acc: 0.795935 loss: 0.695318\n",
            "saving model with acc 0.796\n",
            "[050/100] Train Acc: 0.754414 Loss: 0.800144 | Val Acc: 0.796748 loss: 0.694588\n",
            "saving model with acc 0.797\n",
            "[051/100] Train Acc: 0.755305 Loss: 0.797413 | Val Acc: 0.797886 loss: 0.693050\n",
            "saving model with acc 0.798\n",
            "[052/100] Train Acc: 0.755212 Loss: 0.796450 | Val Acc: 0.797642 loss: 0.691419\n",
            "[053/100] Train Acc: 0.755677 Loss: 0.794271 | Val Acc: 0.796992 loss: 0.691589\n",
            "[054/100] Train Acc: 0.756573 Loss: 0.791168 | Val Acc: 0.798049 loss: 0.690342\n",
            "saving model with acc 0.798\n",
            "[055/100] Train Acc: 0.757176 Loss: 0.790175 | Val Acc: 0.798699 loss: 0.687924\n",
            "saving model with acc 0.799\n",
            "[056/100] Train Acc: 0.757133 Loss: 0.789056 | Val Acc: 0.799106 loss: 0.687956\n",
            "saving model with acc 0.799\n",
            "[057/100] Train Acc: 0.757995 Loss: 0.786743 | Val Acc: 0.799512 loss: 0.689253\n",
            "saving model with acc 0.800\n",
            "[058/100] Train Acc: 0.758030 Loss: 0.785693 | Val Acc: 0.799919 loss: 0.688411\n",
            "saving model with acc 0.800\n",
            "[059/100] Train Acc: 0.758875 Loss: 0.783327 | Val Acc: 0.798943 loss: 0.684820\n",
            "[060/100] Train Acc: 0.759381 Loss: 0.782320 | Val Acc: 0.800650 loss: 0.684700\n",
            "saving model with acc 0.801\n",
            "[061/100] Train Acc: 0.759656 Loss: 0.782020 | Val Acc: 0.800081 loss: 0.683625\n",
            "[062/100] Train Acc: 0.760221 Loss: 0.779084 | Val Acc: 0.801220 loss: 0.684590\n",
            "saving model with acc 0.801\n",
            "[063/100] Train Acc: 0.760358 Loss: 0.777531 | Val Acc: 0.800488 loss: 0.681648\n",
            "[064/100] Train Acc: 0.761444 Loss: 0.776285 | Val Acc: 0.801707 loss: 0.681056\n",
            "saving model with acc 0.802\n",
            "[065/100] Train Acc: 0.761577 Loss: 0.774901 | Val Acc: 0.801951 loss: 0.680395\n",
            "saving model with acc 0.802\n",
            "[066/100] Train Acc: 0.761983 Loss: 0.773620 | Val Acc: 0.802846 loss: 0.679145\n",
            "saving model with acc 0.803\n",
            "[067/100] Train Acc: 0.762214 Loss: 0.772493 | Val Acc: 0.801870 loss: 0.677218\n",
            "[068/100] Train Acc: 0.762132 Loss: 0.771907 | Val Acc: 0.801382 loss: 0.681407\n",
            "[069/100] Train Acc: 0.762779 Loss: 0.770878 | Val Acc: 0.803008 loss: 0.679403\n",
            "saving model with acc 0.803\n",
            "[070/100] Train Acc: 0.763140 Loss: 0.769770 | Val Acc: 0.802520 loss: 0.676844\n",
            "[071/100] Train Acc: 0.763696 Loss: 0.768218 | Val Acc: 0.803496 loss: 0.678676\n",
            "saving model with acc 0.803\n",
            "[072/100] Train Acc: 0.763674 Loss: 0.767323 | Val Acc: 0.803171 loss: 0.677583\n",
            "[073/100] Train Acc: 0.764143 Loss: 0.765506 | Val Acc: 0.803821 loss: 0.676305\n",
            "saving model with acc 0.804\n",
            "[074/100] Train Acc: 0.764011 Loss: 0.765181 | Val Acc: 0.804634 loss: 0.678746\n",
            "saving model with acc 0.805\n",
            "[075/100] Train Acc: 0.764274 Loss: 0.764919 | Val Acc: 0.803740 loss: 0.675561\n",
            "[076/100] Train Acc: 0.765030 Loss: 0.762609 | Val Acc: 0.804228 loss: 0.674722\n",
            "[077/100] Train Acc: 0.765502 Loss: 0.760864 | Val Acc: 0.804715 loss: 0.674503\n",
            "saving model with acc 0.805\n",
            "[078/100] Train Acc: 0.765345 Loss: 0.759918 | Val Acc: 0.804878 loss: 0.672734\n",
            "saving model with acc 0.805\n",
            "[079/100] Train Acc: 0.765681 Loss: 0.759146 | Val Acc: 0.804634 loss: 0.670801\n",
            "[080/100] Train Acc: 0.766269 Loss: 0.757950 | Val Acc: 0.804715 loss: 0.671173\n",
            "[081/100] Train Acc: 0.766609 Loss: 0.757261 | Val Acc: 0.805203 loss: 0.672493\n",
            "saving model with acc 0.805\n",
            "[082/100] Train Acc: 0.766449 Loss: 0.756868 | Val Acc: 0.805285 loss: 0.668045\n",
            "saving model with acc 0.805\n",
            "[083/100] Train Acc: 0.766680 Loss: 0.756317 | Val Acc: 0.805691 loss: 0.670359\n",
            "saving model with acc 0.806\n",
            "[084/100] Train Acc: 0.767037 Loss: 0.755253 | Val Acc: 0.805041 loss: 0.667609\n",
            "[085/100] Train Acc: 0.767595 Loss: 0.752742 | Val Acc: 0.805447 loss: 0.668022\n",
            "[086/100] Train Acc: 0.767935 Loss: 0.752226 | Val Acc: 0.805610 loss: 0.666593\n",
            "[087/100] Train Acc: 0.768334 Loss: 0.750991 | Val Acc: 0.806260 loss: 0.666835\n",
            "saving model with acc 0.806\n",
            "[088/100] Train Acc: 0.769039 Loss: 0.749727 | Val Acc: 0.806341 loss: 0.665262\n",
            "saving model with acc 0.806\n",
            "[089/100] Train Acc: 0.768641 Loss: 0.749939 | Val Acc: 0.805122 loss: 0.666017\n",
            "[090/100] Train Acc: 0.769104 Loss: 0.748697 | Val Acc: 0.807642 loss: 0.665228\n",
            "saving model with acc 0.808\n",
            "[091/100] Train Acc: 0.768983 Loss: 0.748445 | Val Acc: 0.806667 loss: 0.663487\n",
            "[092/100] Train Acc: 0.769025 Loss: 0.747745 | Val Acc: 0.806748 loss: 0.663551\n",
            "[093/100] Train Acc: 0.769815 Loss: 0.745595 | Val Acc: 0.807724 loss: 0.661833\n",
            "saving model with acc 0.808\n",
            "[094/100] Train Acc: 0.770217 Loss: 0.745127 | Val Acc: 0.807398 loss: 0.661418\n",
            "[095/100] Train Acc: 0.770272 Loss: 0.744282 | Val Acc: 0.807642 loss: 0.664240\n",
            "[096/100] Train Acc: 0.769971 Loss: 0.744532 | Val Acc: 0.807967 loss: 0.661788\n",
            "saving model with acc 0.808\n",
            "[097/100] Train Acc: 0.770920 Loss: 0.742241 | Val Acc: 0.808618 loss: 0.660625\n",
            "saving model with acc 0.809\n",
            "[098/100] Train Acc: 0.771145 Loss: 0.741589 | Val Acc: 0.808699 loss: 0.663168\n",
            "saving model with acc 0.809\n",
            "[099/100] Train Acc: 0.771188 Loss: 0.741139 | Val Acc: 0.809512 loss: 0.662392\n",
            "saving model with acc 0.810\n",
            "[100/100] Train Acc: 0.771223 Loss: 0.740754 | Val Acc: 0.808537 loss: 0.660993\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pn5Br9XZJSQl"
      },
      "source": [
        "# **Save Output**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_vE4kN6qJdZk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "746a872a-0414-4501-af35-e1b4a45c99b1"
      },
      "source": [
        "# create testing dataset\n",
        "test_dataset = TIMITDataset(test_data, None)\n",
        "test_dataloader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)\n",
        "\n",
        "# create model and load weights from checkpoint\n",
        "model = Classifier().to(device)\n",
        "model.load_state_dict(torch.load(model_path))"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<All keys matched successfully>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fzHfFe_uT7NS"
      },
      "source": [
        "predict = []\n",
        "model.eval()\n",
        "\n",
        "with torch.no_grad():\n",
        "    for i, data in enumerate(test_dataloader):\n",
        "        inputs = data\n",
        "        inputs = inputs.to(device)\n",
        "        outputs = model(inputs)\n",
        "        _, test_pred = torch.max(outputs, 1) \n",
        "\n",
        "        for y in test_pred.cpu().numpy():\n",
        "            predict.append(y)"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NkB9bjt1BRNS"
      },
      "source": [
        "with open(\"prediction.csv\", 'w') as f:\n",
        "    f.write(\"Id,Class\\n\")\n",
        "    for i, y in enumerate(predict):\n",
        "        f.write(\"{},{}\\n\".format(i, y))"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "QlbFYkJ5jUVs",
        "outputId": "cc9a2d9f-8785-46f9-ed0d-be947c676d2d"
      },
      "source": [
        "from google.colab import files\n",
        "\n",
        "files.download(\"prediction.csv\")"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_0321c125-9a31-4a33-b249-ed7719478834\", \"prediction.csv\", 4241655)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    }
  ]
}