Natural_Language_Processing/Semantic_Similarity_with_BERT/Semantic_Similarity_with_BERT_HJH.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Semantic Similarity is the task of determining how similar two sentences are, in terms of what they mean. This example demonstrates the use of SNLI (Stanford Natural Language Inference) Corpus to predict sentence semantic similarity with Transformers. We will fine-tune a BERT model that takes two sentences as inputs and that outputs a similarity score for these two sentences.\n",
    "\n",
    "## Dataset\n",
    "\n",
    "- [SNLI](https://nlp.stanford.edu/projects/snli/)\n",
    "\n",
    " Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). [[pdf](https://nlp.stanford.edu/pubs/snli_paper.pdf)]\n",
    "\n",
    "<img width=\"685\" alt=\"image\" src=\"https://user-images.githubusercontent.com/37654013/111018451-2d4dc200-83fc-11eb-9f23-11ec849d85e4.png\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "transformers version:  4.3.3\n"
     ]
    }
   ],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import transformers\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(action='ignore')\n",
    "\n",
    "print('transformers version: ',transformers.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import progress_bar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(f'cuda:1' if torch.cuda.is_available() else 'cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_length = 128 # Maximun length of input sentence to the model\n",
    "batch_size = 32\n",
    "epochs = 2\n",
    "\n",
    "# Labels in our dataset\n",
    "labels = [\"contradiction\", \"entailment\", \"neutral\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !curl -LO https://raw.githubusercontent.com/MohamadMerchant/SNLI/master/data.tar.gz\n",
    "# !tar -xvzf data.tar.gza"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total train samples: 100000\n",
      "Total validation samples: 10000\n",
      "Total test samples: 10000\n"
     ]
    }
   ],
   "source": [
    "# There are more than 550k samples in total; we will use 100k for this example.\n",
    "train_df = pd.read_csv(\"SNLI_Corpus/snli_1.0_train.csv\", nrows=100000)\n",
    "valid_df = pd.read_csv(\"SNLI_Corpus/snli_1.0_dev.csv\")\n",
    "test_df = pd.read_csv(\"SNLI_Corpus/snli_1.0_test.csv\")\n",
    "\n",
    "# Shape of the data\n",
    "print(f\"Total train samples: {train_df.shape[0]}\")\n",
    "print(f\"Total validation samples: {valid_df.shape[0]}\")\n",
    "print(f\"Total test samples: {test_df.shape[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset Overview:\n",
    "\n",
    "- sentence1: The premise caption that was supplied to the author of the pair.\n",
    "- sentence2: The hypothesis caption that was written by the author of the pair.\n",
    "- similarity: This is the label chosen by the majority of annotators. Where no majority exists, the label \"-\" is used (we will skip such samples here).\n",
    "\n",
    "Here are the \"similarity\" label values in our dataset:\n",
    "\n",
    "- Contradiction: The sentences share no similarity.\n",
    "- Entailment: The sentences have similar meaning.\n",
    "- Neutral: The sentences are neutral.\n",
    "\n",
    "Let's look at one sample from the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence1: A person on a horse jumps over a broken down airplane.\n",
      "Sentence2: A person is at a diner, ordering an omelette.\n",
      "Similarity: contradiction\n"
     ]
    }
   ],
   "source": [
    "print(f\"Sentence1: {train_df.loc[1, 'sentence1']}\")\n",
    "print(f\"Sentence2: {train_df.loc[1, 'sentence2']}\")\n",
    "print(f\"Similarity: {train_df.loc[1, 'similarity']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 100000 entries, 0 to 99999\n",
      "Data columns (total 3 columns):\n",
      " #   Column      Non-Null Count   Dtype \n",
      "---  ------      --------------   ----- \n",
      " 0   similarity  100000 non-null  object\n",
      " 1   sentence1   100000 non-null  object\n",
      " 2   sentence2   99997 non-null   object\n",
      "dtypes: object(3)\n",
      "memory usage: 2.3+ MB\n"
     ]
    }
   ],
   "source": [
    "train_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>similarity</th>\n",
       "      <th>sentence1</th>\n",
       "      <th>sentence2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>91479</th>\n",
       "      <td>neutral</td>\n",
       "      <td>Cannot see picture to describe.</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91480</th>\n",
       "      <td>entailment</td>\n",
       "      <td>Cannot see picture to describe.</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91481</th>\n",
       "      <td>contradiction</td>\n",
       "      <td>Cannot see picture to describe.</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          similarity                        sentence1 sentence2\n",
       "91479        neutral  Cannot see picture to describe.       NaN\n",
       "91480     entailment  Cannot see picture to describe.       NaN\n",
       "91481  contradiction  Cannot see picture to describe.       NaN"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[train_df.sentence2.isnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of missing values\n",
      "similarity    0\n",
      "sentence1     0\n",
      "sentence2     3\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# We have some NaN entries in our train data, we will simply drop them.\n",
    "print(\"Number of missing values\")\n",
    "print(train_df.isnull().sum())\n",
    "train_df.dropna(axis=0, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Distribution of our training targets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Target Distribution\n",
      "entailment       33384\n",
      "contradiction    33310\n",
      "neutral          33193\n",
      "-                  110\n",
      "Name: similarity, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(\"Train Target Distribution\")\n",
    "print(train_df.similarity.value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Distribution of our validation targets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation Target Distribution\n",
      "entailment       3329\n",
      "contradiction    3278\n",
      "neutral          3235\n",
      "-                 158\n",
      "Name: similarity, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(\"Validation Target Distribution\")\n",
    "print(valid_df.similarity.value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The value \"-\" appears as part of our training and validation targets. We will skip these samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = (\n",
    "    train_df[train_df.similarity != \"-\"]\n",
    "    .sample(frac=1.0, random_state=42)\n",
    "    .reset_index(drop=True)\n",
    ")\n",
    "\n",
    "valid_df = (\n",
    "    valid_df[valid_df.similarity != \"-\"]\n",
    "    .sample(frac=1.0, random_state=42)\n",
    "    .reset_index(drop=True)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Label encoding train, valid and test labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder = {\n",
    "    'contradiction':0,\n",
    "    'entailment':1,\n",
    "    'neutral':2\n",
    "}\n",
    "\n",
    "train_df['similarity'] = train_df['similarity'].map(encoder)\n",
    "valid_df['similarity'] = valid_df['similarity'].map(encoder)\n",
    "test_df['similarity'] = test_df['similarity'].map(encoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create a custom data generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokeinzer Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tokenizer return values description**\n",
    "- **input_ids** : 입력값으로 들어간 각 토큰에 대한 id\n",
    "- **attention_mask** : 인코딩된 토큰 중에서 focus를 취해야하는 곳은 '1'로 아닌 곳은 '0'으로 나타냄 \n",
    "- **token_type_ids** : Sequence classification 또는 QA 같은 task 경우 여러 sentence를 함께 encoding하는데 각 sentence를 구분하기 위해 나타냄. 첫 sentence는 '0', 두 번째는 '1'. 이런식으로 구분함\n",
    "\n",
    "**source:** https://huggingface.co/transformers/glossary.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = transformers.BertTokenizer.from_pretrained(\n",
    "    \"bert-base-uncased\", do_lower_case=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.\n"
     ]
    }
   ],
   "source": [
    "example = train_df[['sentence1','sentence2']].values.astype('str')[0].tolist()\n",
    "encoded = tokenizer.encode_plus(\n",
    "    text=example[0],\n",
    "    text_pair=example[1],\n",
    "    add_special_tokens=True, # Such as '[CLS]', '[SEP]'\n",
    "    max_length=max_length, # maximun length \n",
    "    return_attention_mask=True, # whether returns attention_mask\n",
    "    return_token_type_ids=True, # whether returns token_type_ids\n",
    "    pad_to_max_length=True, # padding\n",
    "    return_tensors=\"pt\" # 'pt': pytorch, 'tf': tensorflow\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[input]\n",
      "Sentence1: Two male clowns, one in a plaid suit and the other in black, performing a musical number in a theater setting.\n",
      "Sentence2: The clowns are in the dressing room.\n",
      "\n",
      "[encoded]\n",
      "input_ids: tensor([[  101,  2048,  3287, 15912,  2015,  1010,  2028,  1999,  1037, 26488,\n",
      "          4848,  1998,  1996,  2060,  1999,  2304,  1010,  4488,  1037,  3315,\n",
      "          2193,  1999,  1037,  4258,  4292,  1012,   102,  1996, 15912,  2015,\n",
      "          2024,  1999,  1996, 11225,  2282,  1012,   102,     0,     0,     0,\n",
      "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
      "             0,     0,     0,     0,     0,     0,     0,     0]])\n",
      "attention_mask:  tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
      "         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0]])\n",
      "token_type_ids:  tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
      "         0, 0, 0, 0, 0, 0, 0, 0]])\n",
      "\n",
      "[decoded]\n",
      "decode: ['[CLS]', 'two', 'male', 'clown', '##s', ',', 'one', 'in', 'a', 'plaid', 'suit', 'and', 'the', 'other', 'in', 'black', ',', 'performing', 'a', 'musical', 'number', 'in', 'a', 'theater', 'setting', '.', '[SEP]', 'the', 'clown', '##s', 'are', 'in', 'the', 'dressing', 'room', '.', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]']\n"
     ]
    }
   ],
   "source": [
    "print('[input]')\n",
    "print(f'Sentence1: {example[0]}')\n",
    "print(f'Sentence2: {example[1]}')\n",
    "print('\\n[encoded]')\n",
    "print(f\"input_ids: {encoded['input_ids']}\")\n",
    "print(f'attention_mask: ',encoded['attention_mask'])\n",
    "print(f'token_type_ids: ',encoded['token_type_ids'])\n",
    "print('\\n[decoded]')\n",
    "print(f\"decode: {tokenizer.convert_ids_to_tokens(encoded['input_ids'][0])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create dataloader using tokeinzer\n",
    "\n",
    "1. Make a Dataset\n",
    "2. Build a DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BertSemanticDataset(torch.utils.data.Dataset):\n",
    "    \"\"\"Generates batches of data.\n",
    "\n",
    "    Args:\n",
    "        sentence_pairs: Array of premise and hypothesis input sentences.\n",
    "        targets: Array of labels.\n",
    "        max_length: maximun length of sentence\n",
    "        include_targets: boolean, whether to incude the labels.\n",
    "\n",
    "    Returns:\n",
    "        Dictionary keys : ['input_ids','attention_mask','token_type_ids','target']\n",
    "        (or just [input_ids, attention_mask, token_type_ids] if include_targets=False)\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(\n",
    "        self,\n",
    "        sentence_pairs,\n",
    "        tokenizer,\n",
    "        max_length,\n",
    "        targets=None,\n",
    "        include_targets=True,\n",
    "    ):\n",
    "        self.sentence_pairs = sentence_pairs\n",
    "        self.targets = targets\n",
    "        self.include_targets = include_targets\n",
    "        self.tokenizer = tokenizer\n",
    "        \n",
    "    def __len__(self):\n",
    "        # Denotes the number of sentence pairs\n",
    "        return len(self.sentence_pairs)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        encoded = self.tokenizer.encode_plus(\n",
    "            self.sentence_pairs[idx][0],\n",
    "            text_pair=self.sentence_pairs[idx][1],\n",
    "            add_special_tokens=True,\n",
    "            max_length=max_length,\n",
    "            return_attention_mask=True,\n",
    "            return_token_type_ids=True,\n",
    "            pad_to_max_length=True,\n",
    "            return_tensors=\"pt\",\n",
    "        )\n",
    "        \n",
    "        if self.include_targets:\n",
    "            return {\n",
    "                'input_ids':encoded['input_ids'][0],\n",
    "                'attention_mask':encoded['attention_mask'][0],\n",
    "                'token_type_ids':encoded['token_type_ids'][0],\n",
    "                'target': self.targets[idx]\n",
    "            }\n",
    "        else:\n",
    "            return {\n",
    "                'input_ids':encoded['input_ids'][0],\n",
    "                'attention_mask':encoded['attention_mask'][0],\n",
    "                'token_type_ids':encoded['token_type_ids'][0]\n",
    "            }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainset = BertSemanticDataset(\n",
    "    sentence_pairs=train_df[['sentence1','sentence2']].values.astype('str'),\n",
    "    targets=train_df['similarity'].values,\n",
    "    tokenizer=tokenizer,\n",
    "    max_length=max_length\n",
    ")\n",
    "\n",
    "validset = BertSemanticDataset(\n",
    "    sentence_pairs=valid_df[['sentence1','sentence2']].values.astype('str'),\n",
    "    targets=valid_df['similarity'].values,\n",
    "    tokenizer=tokenizer,\n",
    "    max_length=max_length\n",
    ")\n",
    "\n",
    "testset = BertSemanticDataset(\n",
    "    sentence_pairs=test_df[['sentence1','sentence2']].values.astype('str'),\n",
    "    targets=test_df['similarity'].values,\n",
    "    tokenizer=tokenizer,\n",
    "    max_length=max_length,\n",
    "    include_targets=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainloader = torch.utils.data.DataLoader(\n",
    "    dataset=trainset,\n",
    "    batch_size=batch_size,\n",
    "    shuffle=True,\n",
    "    num_workers=4\n",
    ")\n",
    "validloader = torch.utils.data.DataLoader(\n",
    "    dataset=validset,\n",
    "    batch_size=batch_size,\n",
    "    shuffle=False,\n",
    "    num_workers=4\n",
    ")\n",
    "testloader = torch.utils.data.DataLoader(\n",
    "    dataset=testset,\n",
    "    batch_size=batch_size,\n",
    "    shuffle=False,\n",
    "    num_workers=4\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_i = next(iter(trainloader))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build the model\n",
    "\n",
    "- **model.eval()** will notify all your layers that you are in eval mode, that way, batchnorm or dropout layers will work in eval mode instead of training mode.\n",
    "\n",
    "- **torch.no_grad()** impacts the autograd engine and deactivate it. It will reduce memory usage and speed up computations but you won’t be able to backprop (which you don’t want in an eval script)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Problem**\n",
    "- 학습이 제대로 진행히 안되고 33%에 수렴.\n",
    "\n",
    "**Reason**\n",
    "- `forward`에서 bert를 torch.no_grad()로 감싸줘야함. \n",
    "    - `__init__`에서 bert의 parameter를 requires_grad=False로 설정해줘도 안됨\n",
    "- fine tuning을 할때는 learning rate를 작게해줘야함. Bert가 크다보니 제대로 수렴이 되지않음"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BertSemanticModel(torch.nn.Module):\n",
    "    def __init__(self):\n",
    "        super(BertSemanticModel, self).__init__() \n",
    "        \n",
    "        self.fine_tuning = False\n",
    "        self.bert = transformers.BertModel.from_pretrained('bert-base-uncased')\n",
    "            \n",
    "        self.bi_lstm = torch.nn.LSTM(input_size=self.bert.config.hidden_size, \n",
    "                                     hidden_size=64,\n",
    "                                     bidirectional=True)\n",
    "        \n",
    "        self.linear = torch.nn.Linear(in_features=64*2*2, out_features=3) \n",
    "        self.dropout = torch.nn.Dropout(p=0.3)\n",
    "        \n",
    "    def forward(self, input_ids, attention_mask, token_type_ids):\n",
    "        \n",
    "        if self.fine_tuning:\n",
    "            embedding = self.bert(input_ids,\n",
    "                                  attention_mask=attention_mask,\n",
    "                                  token_type_ids=token_type_ids)\n",
    "        else:\n",
    "            with torch.no_grad():\n",
    "                embedding = self.bert(input_ids,\n",
    "                                      attention_mask=attention_mask,\n",
    "                                      token_type_ids=token_type_ids)\n",
    "        \n",
    "        # sequence_output (batch size x #token x hidden size) : (batch size x 128 x 768)        \n",
    "        # pooled_output (batch size x  hidden size)           : (batch size x  768) CLS token에 linear mapping 후 tanh 결과\n",
    "        sequence_output, pooled_output = embedding[0], embedding[1]\n",
    "        \n",
    "        # lstm_out (batch size x #token x hidden size)        : (batch size x 128 x 128)\n",
    "        lstm_out, _ = self.bi_lstm(sequence_output)\n",
    "\n",
    "        # gap_out (batch size x hidden size)                  : (batch size x 128)\n",
    "        gap_out = lstm_out.mean(dim=1) # GAP\n",
    "        \n",
    "        # gmp_out (batch size x hidden size)                  : (batch size x 128)\n",
    "        gmp_out, _ = lstm_out.max(dim=1) # GMP\n",
    "           \n",
    "        # out (batch size x hidden size)                      : (batch size x 256)\n",
    "        out = torch.cat([gap_out, gmp_out], dim=1)\n",
    "        out = self.dropout(out)\n",
    "        \n",
    "        # out (batch size x #class)                           : (batch size x 3)\n",
    "        out = self.linear(out)\n",
    "        \n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = BertSemanticModel().to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ther number of parameters from pytorch model:  109910019\n"
     ]
    }
   ],
   "source": [
    "print('Ther number of parameters from pytorch model: ',sum([np.prod(param.size()) for param in model.parameters()]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The difference of the number of model parameters between Pytorch and Keras\n",
      "109910019 - 109909507 =  512\n"
     ]
    }
   ],
   "source": [
    "print('The difference of the number of model parameters between Pytorch and Keras')\n",
    "print('109910019 - 109909507 = ',109910019 - 109909507)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The difference of the number of LSTM parameters between Pytorch and Keras\n",
      "427008 - 426496 = 512\n"
     ]
    }
   ],
   "source": [
    "lstm_params = sum([np.prod(param.size()) for param in model.bi_lstm.parameters()])\n",
    "print('The difference of the number of LSTM parameters between Pytorch and Keras')\n",
    "print(f'{lstm_params} - 426496 = {lstm_params - 426496}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, dataloader, criterion, optimizer, device):\n",
    "    total = 0\n",
    "    correct = 0 \n",
    "    total_loss = 0\n",
    "    \n",
    "    model.train()\n",
    "    for batch_idx, batch_i in enumerate(dataloader):\n",
    "        # inputs and targets\n",
    "        input_ids = batch_i['input_ids'].to(device)\n",
    "        attention_mask = batch_i['attention_mask'].to(device)\n",
    "        token_type_ids = batch_i['token_type_ids'].to(device)\n",
    "        targets = batch_i['target'].to(device)\n",
    "        \n",
    "        # reset optimizer\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        # model output\n",
    "        outputs = model(input_ids, attention_mask, token_type_ids)\n",
    "        \n",
    "        # accuracy\n",
    "        _, predict = outputs.max(1)\n",
    "        correct += predict.eq(targets.long()).cpu().float().sum().item()\n",
    "        total += input_ids.size(0)\n",
    "        \n",
    "        # loss\n",
    "        loss = criterion(outputs, targets)\n",
    "        loss.backward()\n",
    "        \n",
    "        # update optimizer\n",
    "        optimizer.step()\n",
    "        \n",
    "        total_loss += loss.item()\n",
    "    \n",
    "        \n",
    "        # massage\n",
    "        progress_bar(current=batch_idx, \n",
    "                     total=len(dataloader),\n",
    "                     msg='Loss: %.3f | Acc: %.3f%%' % (total_loss/(batch_idx + 1), \n",
    "                                                               100.*(correct/total)),\n",
    "                     term_width=100)\n",
    "        \n",
    "        \n",
    "def validation(model, dataloader, criterion, device):\n",
    "    total = 0\n",
    "    correct = 0 \n",
    "    total_loss = 0\n",
    "    \n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        for batch_idx, batch_i in enumerate(dataloader):\n",
    "            # inputs and targets\n",
    "            input_ids = batch_i['input_ids'].to(device)\n",
    "            attention_mask = batch_i['attention_mask'].to(device)\n",
    "            token_type_ids = batch_i['token_type_ids'].to(device)\n",
    "            targets = batch_i['target'].to(device)\n",
    "\n",
    "\n",
    "            # model output\n",
    "            outputs = model(input_ids, attention_mask, token_type_ids)\n",
    "\n",
    "            # accuracy\n",
    "            _, predict = outputs.max(1)\n",
    "            correct += predict.eq(targets.long()).cpu().float().sum().item()\n",
    "            total += input_ids.size(0)\n",
    "\n",
    "            # loss\n",
    "            loss = criterion(outputs, targets)\n",
    "            total_loss += loss.item()\n",
    "\n",
    "            # massage\n",
    "            progress_bar(current=batch_idx, \n",
    "                         total=len(dataloader),\n",
    "                         msg='Loss: %.3f | Acc: %.3f%%' % (total_loss/(batch_idx + 1), \n",
    "                                                                   100.*(correct/total)),\n",
    "                         term_width=100)\n",
    "            \n",
    "            \n",
    "def fit(model, epochs, trainloader, criterion, optimizer, device, validloader=None):\n",
    "    for epoch in range(epochs):\n",
    "        print('Fit start')\n",
    "        print(f'\\nEpochs: {epoch+1}/{epochs}')\n",
    "        train(model, trainloader, criterion, optimizer, device)\n",
    "        if validloader is not None:\n",
    "            validation(model, validloader, criterion, device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = torch.nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.Adam(model.parameters())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fit start\n",
      "\n",
      "Epochs: 1/1\n",
      " [================================================================>]  Step: 1ms | Tot: 1ms | Loss: 0.722 | Acc: 68.71 3122/3122 \n",
      " [================================================================>]  Step: 0ms | Tot: 0ms | Loss: 0.534 | Acc: 78.83 308/308 \n"
     ]
    }
   ],
   "source": [
    "fit(model=model,\n",
    "    epochs=1,\n",
    "    trainloader=trainloader,\n",
    "    validloader=validloader,\n",
    "    criterion=criterion,\n",
    "    optimizer=optimizer,\n",
    "    device=device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine-tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fine_tuning = True\n",
    "optimizer.param_groups[0]['lr'] = 0.00001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fit start\n",
      "\n",
      "Epochs: 1/1\n",
      " [================================================================>]  Step: 0ms | Tot: 0ms | Loss: 0.494 | Acc: 80.91 3122/3122 \n",
      " [================================================================>]  Step: 0ms | Tot: 0ms | Loss: 0.364 | Acc: 86.49 307/30 308/308 \n"
     ]
    }
   ],
   "source": [
    "fit(model=model,\n",
    "    epochs=1,\n",
    "    trainloader=trainloader,\n",
    "    validloader=validloader,\n",
    "    criterion=criterion,\n",
    "    optimizer=optimizer,\n",
    "    device=device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train the entire model end-to-end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluate model on the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(model, dataloader, device):\n",
    "    preds = np.zeros(len(dataloader.dataset))\n",
    "    \n",
    "    model.eval()\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        for batch_idx, batch_i in enumerate(dataloader):\n",
    "            # inputs and targets\n",
    "            input_ids = batch_i['input_ids'].to(device)\n",
    "            attention_mask = batch_i['attention_mask'].to(device)\n",
    "            token_type_ids = batch_i['token_type_ids'].to(device)\n",
    "\n",
    "            # model output\n",
    "            outputs = model(input_ids, attention_mask, token_type_ids)\n",
    "\n",
    "            # predict\n",
    "            _, predict = outputs.max(1)\n",
    "            \n",
    "            start_idx = batch_idx * input_ids.size(0)\n",
    "            end_idx = (batch_idx+1) * input_ids.size(0)\n",
    "            preds[start_idx:end_idx] = predict.detach().cpu().numpy()\n",
    "            \n",
    "            \n",
    "            # massage\n",
    "            progress_bar(current=batch_idx, \n",
    "                         total=len(testloader),\n",
    "                         term_width=100)\n",
    "                         \n",
    "    return preds\n",
    "                         \n",
    "def evaluate(preds, trues):\n",
    "    return np.sum(preds == trues) / len(trues)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " [================================================================>]  Step: 0ms | Tot: 0ms         313/313 \n"
     ]
    }
   ],
   "source": [
    "preds = predict(model, testloader, device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 84.670%\n"
     ]
    }
   ],
   "source": [
    "test_acc = evaluate(preds=preds, trues=test_df['similarity'].values)\n",
    "print('Test Accuracy: {0:.3%}'.format(test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference on custom sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_similarity(sentence1, sentence2, device):\n",
    "    labels = [\"contradiction\", \"entailment\", \"neutral\"]\n",
    "    \n",
    "    sentence_pairs = np.array([[str(sentence1), str(sentence2)]])\n",
    "    test_data = BertSemanticDataset(\n",
    "        sentence_pairs, \n",
    "        max_length=max_length, \n",
    "        tokenizer=tokenizer, \n",
    "        include_targets=False,\n",
    "    )\n",
    "    \n",
    "    testloader = torch.utils.data.DataLoader(\n",
    "        dataset=test_data,\n",
    "        batch_size=1,\n",
    "        shuffle=False,\n",
    "        num_workers=4\n",
    "    )\n",
    "\n",
    "    test_input = next(iter(testloader))\n",
    "    \n",
    "    model.eval()\n",
    "    output = model(test_input['input_ids'].to(device),\n",
    "                   attention_mask=test_input['attention_mask'].to(device),\n",
    "                   token_type_ids=test_input['token_type_ids'].to(device))\n",
    "    \n",
    "    output = torch.nn.functional.softmax(output, dim=1)\n",
    "    proba, idx = output.max(1)\n",
    "    \n",
    "    print('Sentence1: ',sentence1)\n",
    "    print('Sentence2: ',sentence2)\n",
    "    print(\"{0:}: {1: .2%}\".format(labels[idx].upper(), proba.item()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence1:  Two women are observing something together.\n",
      "Sentence2:  Two women are standing with their eyes closed.\n",
      "CONTRADICTION:  77.36%\n"
     ]
    }
   ],
   "source": [
    "sentence1 = \"Two women are observing something together.\"\n",
    "sentence2 = \"Two women are standing with their eyes closed.\"\n",
    "check_similarity(sentence1, sentence2, device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence1:  A smiling costumed woman is holding an umbrella\n",
      "Sentence2:  A happy woman in a fairy costume holds an umbrella\n",
      "NEUTRAL:  90.67%\n"
     ]
    }
   ],
   "source": [
    "sentence1 = \"A smiling costumed woman is holding an umbrella\"\n",
    "sentence2 = \"A happy woman in a fairy costume holds an umbrella\"\n",
    "check_similarity(sentence1, sentence2, device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence1:  A soccer game with multiple males playing\n",
      "Sentence2:  Some men are playing a sport\n",
      "ENTAILMENT:  94.69%\n"
     ]
    }
   ],
   "source": [
    "sentence1 = \"A soccer game with multiple males playing\"\n",
    "sentence2 = \"Some men are playing a sport\"\n",
    "check_similarity(sentence1, sentence2, device)"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}