Natural_Language_Processing/Question_Answering_Huggingface/QA_huggingface_KYK.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1680d6df-5137-4ca5-8b08-56143ca431f9",
   "metadata": {},
   "source": [
    "# Question Answering with Hugging Face Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96b7f80d-8165-48db-9ac1-332e11ac361f",
   "metadata": {},
   "source": [
    "[Original Keras Code](https://keras.io/examples/nlp/question_answering/)\n",
    "\n",
    "<br/>\n",
    "\n",
    "**Author**: Yookyung Kho\n",
    "\n",
    "**Date presented**: 2022/05/09, DSBA keras2torch Study\n",
    "\n",
    "**Task description**: Question Answering with pretrained `distilbert-base-cased` from HuggingFace\n",
    "\n",
    "**References**:\n",
    "\n",
    "- https://qa.fastforwardlabs.com/pytorch/hugging%20face/wikipedia/bert/transformers/2020/05/19/Getting_Started_with_QA.html\n",
    "\n",
    "- https://huggingface.co/transformers/v3.0.2/model_doc/auto.html\n",
    "\n",
    "- https://huggingface.co/course/chapter7/7?fw=tf\n",
    "\n",
    "- https://huggingface.co/course/chapter6/3b?fw=pt\n",
    "\n",
    "- https://huggingface.co/transformers/v3.2.0/custom_datasets.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "690ddfe4-7262-421d-af5b-42764d48759d",
   "metadata": {},
   "source": [
    "## 0. About QA"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08d07dc0-0b29-423d-b6fa-36a90e9568d7",
   "metadata": {},
   "source": [
    "- Task: **context-based question answering** - questions are asked from a given paragraph\n",
    "\n",
    "- Dataset: SQUAD v1.1\n",
    "\n",
    "<img src=\"qa_output.png\" width=\"1000\" height=\"600\">\n",
    "\n",
    "<img src=\"qa_input.png\" width=\"1000\" height=\"600\">\n",
    "\n",
    "[img source](https://blog.paperspace.com/how-to-train-question-answering-machine-learning-models/)\n",
    "\n",
    "- Input: Question(질문), Context(정답 span 포함)\n",
    "\n",
    "```\n",
    "[CLS] question [SEP] context [SEP]\n",
    "```\n",
    "\n",
    "- Output: Answer의 시작과 끝 토큰\n",
    "\n",
    "    - `start_logits`: (batch_size, sequence_length)\n",
    "    \n",
    "    - `end_logits`: (batch_size, sequence_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc79cab3-ef55-4331-81c5-74a0e8b236d3",
   "metadata": {},
   "source": [
    "## 1. Loading the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "947802bd-4851-4ef4-9b46-93f218606e1d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reusing dataset squad (/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "670864adb62a45708a4607c380bffed5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from datasets import load_dataset\n",
    "\n",
    "datasets = load_dataset(\"squad\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "68395017-d1a8-437e-8052-b0eb6e8ac20c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['id', 'title', 'context', 'question', 'answers'],\n",
       "        num_rows: 87599\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['id', 'title', 'context', 'question', 'answers'],\n",
       "        num_rows: 10570\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e9222c6b-5d4f-4e0a-954d-2ebe66b4805b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': '5733be284776f41900661182',\n",
       " 'title': 'University_of_Notre_Dame',\n",
       " 'context': 'Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',\n",
       " 'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',\n",
       " 'answers': {'text': ['Saint Bernadette Soubirous'], 'answer_start': [515]}}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex_train = datasets[\"train\"][0]\n",
    "ex_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fa26df20-58f1-4d4c-93fa-ab6e12edabcc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': '56be4db0acb8001400a502ec',\n",
       " 'title': 'Super_Bowl_50',\n",
       " 'context': 'Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi\\'s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.',\n",
       " 'question': 'Which NFL team represented the AFC at Super Bowl 50?',\n",
       " 'answers': {'text': ['Denver Broncos', 'Denver Broncos', 'Denver Broncos'],\n",
       "  'answer_start': [177, 177, 177]}}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex_valid = datasets[\"validation\"][0]\n",
    "ex_valid"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26fbced6-2a1a-4b29-a133-43e49ce28358",
   "metadata": {},
   "source": [
    "- train -> train, valid\n",
    "\n",
    "- validation -> test\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "99861395-1564-4bb1-b40a-125e2d059dc2",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached shuffled indices for dataset at /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453/cache-b7b1a28774d2c4ca.arrow\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data size: Train(8000), Valid(2000)\n"
     ]
    }
   ],
   "source": [
    "shuffled_set = datasets[\"train\"].shuffle(seed=602)\n",
    "\n",
    "small_set = shuffled_set.select(range(10000)) #10000개만 샘플링\n",
    "\n",
    "train_valid = small_set.train_test_split(test_size=0.2)\n",
    "\n",
    "train_examples = train_valid[\"train\"]\n",
    "valid_examples = train_valid[\"test\"]\n",
    "\n",
    "print(f\"Data size: Train({len(train_examples)}), Valid({len(valid_examples)})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0499a63b-310c-4964-89c6-924dd9b06440",
   "metadata": {},
   "source": [
    "## 2. Preprocessing the training data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd232d21-7272-46e7-a31c-bb87922f12fb",
   "metadata": {},
   "source": [
    "### 2.1. Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8429f49e-59ec-4fca-9bfc-9cdf3ebc63ef",
   "metadata": {},
   "source": [
    "How to deal with very **long context**?\n",
    "\n",
    "- 보통 최대 길이(max length)에 맞춰 자름\n",
    "\n",
    "- 하지만 QA에서는 주어진 context에서 정답을 찾아야 하기 때문에 context를 max len에 맞춰 잘라버리면 정답이 사라지는 문제가 발생할 수도 있음!\n",
    "\n",
    "- (해결책) **overlap을 허용하면서 context를 더 작은 chunk들로 분할**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e76d9cd5-f90c-4942-9d70-c4c67fb8e175",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "model_checkpoint = \"distilbert-base-cased\"\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bb95c96c-313b-45f2-ba7e-1ed5caa2896c",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_length = 384  # The maximum length of a feature (question and context)\n",
    "doc_stride = 128  # The authorized overlap between two part of the context when splitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "bb59685e-a0f8-4772-9047-db90a426ee89",
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_train_features(examples):\n",
    "    # Tokenize our examples with truncation and padding, but keep the overflows using a\n",
    "    # stride. This results in one example possible giving several features when a context is long,\n",
    "    # each of those features having a context that overlaps a bit the context of the previous\n",
    "    # feature.\n",
    "    examples[\"question\"] = [q.lstrip() for q in examples[\"question\"]]\n",
    "    examples[\"context\"] = [c.lstrip() for c in examples[\"context\"]]\n",
    "    \n",
    "    # 1) tokenizer로 분절 # [CLS] question [SEP] context [SEP]\n",
    "    tokenized_examples = tokenizer(\n",
    "        examples[\"question\"],\n",
    "        examples[\"context\"],\n",
    "        truncation=\"only_second\", ### question은 보존, context만 max len을 넘어가면 잘리도록\n",
    "        max_length=max_length,\n",
    "        stride=doc_stride, ### overlap 정도(128 토큰)\n",
    "        return_overflowing_tokens=True, ### let the tokenizer know we want the overflowing tokens\n",
    "        return_offsets_mapping=True, ### to compute the start_positions and end_positions\n",
    "        padding=\"max_length\",\n",
    "    )\n",
    "    # tokenized_examples: dictionary 반환\n",
    "    # {'input_ids': [101, 1706, ..., 102],\n",
    "    #  'attention_mask': [1, 1, 1 ..., 0],\n",
    "    #  'offset_mapping': [(0, 0), (0, 2), (3, 7), ...(694, 695), (0, 0)],\n",
    "    #  'overflow_to_sample_mapping': [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1]}\n",
    "    \n",
    "    \n",
    "    # 2) \"overflow_to_sample_mapping\"과 \"offset_mapping\" 삭제\n",
    "    ### input_ids와 attention mask만 model input으로 들어가기 때문에 나머지 pop~\n",
    "    ### tokenized_examples 딕셔너리에서 'overflow_to_sample_mapping'과 'offset_mapping'만 따로 빼서 저장해둠\n",
    "    \n",
    "    ## \"overflow_to_sample_mapping\": context가 길어서 여러 feature로 뽑히는 경우 각 feature가 몇번째 문장(샘플)에 속하는지 파악할 수 있음\n",
    "    sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n",
    "\n",
    "    ## \"offset_mapping\": 각 토큰의 첫 철자와 마지막 철자 위치(인덱스)가 튜플 형태로 표현됨\n",
    "    #### ex. \"cat\": (15,17)\n",
    "    offset_mapping = tokenized_examples.pop(\"offset_mapping\")\n",
    "\n",
    "    # 3) 정답의 시작, 끝 위치 라벨링\n",
    "    tokenized_examples[\"start_positions\"] = []\n",
    "    tokenized_examples[\"end_positions\"] = []\n",
    "\n",
    "    for i, offsets in enumerate(offset_mapping):\n",
    "        ### feature 내 정답 없으면 0으로 라벨링(시퀀스 맨 처음에 위치한 [CLS])\n",
    "        input_ids = tokenized_examples[\"input_ids\"][i] # i번째 span이자 feature\n",
    "        cls_index = input_ids.index(tokenizer.cls_token_id) # 0\n",
    "\n",
    "        ### 시퀀스 내 question, context 위치 파악\n",
    "        sequence_ids = tokenized_examples.sequence_ids(i) # [cls]와 [sep]은 None, question은 0, context는 1로 채워진 리스트\n",
    "\n",
    "        # One example can give several spans, this is the index of the example containing this span of text.\n",
    "        sample_index = sample_mapping[i] #몇번째 샘플(데이터, 문장)인지\n",
    "        answers = examples[\"answers\"][sample_index] #정답 'text'와 'answer_start'(start char idx) 포함하는 딕셔너리\n",
    "        \n",
    "        #정답이 주어지지 않은 경우 [cls]를 가상의 정답으로 간주\n",
    "        if len(answers[\"answer_start\"]) == 0:\n",
    "            tokenized_examples[\"start_positions\"].append(cls_index)\n",
    "            tokenized_examples[\"end_positions\"].append(cls_index)\n",
    "        else:\n",
    "            # Start/end character index of the answer in the text.\n",
    "            start_char = answers[\"answer_start\"][0]\n",
    "            end_char = start_char + len(answers[\"text\"][0])\n",
    "\n",
    "            # Start token index of the current span in the text.\n",
    "            token_start_index = 0\n",
    "            while sequence_ids[token_start_index] != 1:\n",
    "                token_start_index += 1\n",
    "\n",
    "            # End token index of the current span in the text.\n",
    "            token_end_index = len(input_ids) - 1 # max-len - 1\n",
    "            while sequence_ids[token_end_index] != 1: #None으로 채워진 [SEP] 자리만 거침\n",
    "                token_end_index -= 1\n",
    "\n",
    "            ## possible case: 해당 span의 첫번째 토큰의 첫 알파벳이 정답의 첫번째 철자보다 전에 위치하고\n",
    "            ## 마지막 토큰의 마지막 철자가  정답의 마지막 펄자보다 뒤에 위치해야 해당 span 내 정답 존재할 수 있음\n",
    "            ### 이 경우를 제외하고는 전부 [CLS]을 가상 정답으로 처리\n",
    "            if not (\n",
    "                offsets[token_start_index][0] <= start_char\n",
    "                and offsets[token_end_index][1] >= end_char\n",
    "            ):\n",
    "                tokenized_examples[\"start_positions\"].append(cls_index)\n",
    "                tokenized_examples[\"end_positions\"].append(cls_index)\n",
    "            else:\n",
    "                # Otherwise move the token_start_index and token_end_index to the two ends of the answer.\n",
    "                # Note: we could go after the last offset if the answer is the last word (edge case).\n",
    "                while (\n",
    "                    token_start_index < len(offsets) # max_len (시퀀스 내 최대 토큰 수)\n",
    "                    and offsets[token_start_index][0] <= start_char\n",
    "                ):\n",
    "                    token_start_index += 1 #정답 span의 시작 토큰으로 하나씩 접근 \n",
    "                tokenized_examples[\"start_positions\"].append(token_start_index - 1)\n",
    "                \n",
    "                while offsets[token_end_index][1] >= end_char:\n",
    "                    token_end_index -= 1\n",
    "                tokenized_examples[\"end_positions\"].append(token_end_index + 1)\n",
    "    \n",
    "    return tokenized_examples\n",
    "\n",
    "# [83, 51, 19, 0, 0, 64, 27, 0, 34, 0, 0, 0, 67, 34, 0, 0, 0, 0, 0] ## start positions\n",
    "# [85, 53, 21, 0, 0, 70, 33, 0, 40, 0, 0, 0, 68, 35, 0, 0, 0, 0, 0] ## end positions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b52dcdf1-d157-4715-abac-975614fdc7a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data size: 8000 -> 8103\n"
     ]
    }
   ],
   "source": [
    "# features: train_dataset\n",
    "train_dataset = train_examples.map(\n",
    "    prepare_train_features,\n",
    "    batched=True,\n",
    "    num_proc=3,\n",
    "    remove_columns=train_examples.column_names,\n",
    ")\n",
    "print(f\"Train data size: {len(train_examples)} -> {len(train_dataset)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "17be2888-2a5d-45c4-b55f-6d015311f0d4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['id', 'title', 'context', 'question', 'answers'],\n",
       "    num_rows: 8000\n",
       "})"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a2b2067b-57a4-4248-aba9-e80d3be099b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['input_ids', 'attention_mask', 'start_positions', 'end_positions'],\n",
       "    num_rows: 8103\n",
       "})"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0f93237e-1687-40a2-848d-82a473fc84a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input_ids': [101, 1327, 1202, 1103, 1697, 1105, 3732, 2031, 5299, 1116, 22417, 1106, 1146, 8678, 136, 102, 1398, 1433, 3099, 1104, 1103, 1244, 1311, 117, 1259, 1103, 1697, 117, 1103, 3302, 1116, 1104, 1103, 3732, 2031, 117, 1352, 7030, 1105, 27597, 117, 1105, 1155, 1484, 1104, 2757, 117, 20335, 1148, 1105, 17766, 1106, 1146, 8678, 1103, 5317, 119, 1636, 12749, 1116, 170, 3101, 3161, 1306, 1115, 1103, 3013, 1104, 1644, 1110, 7298, 1106, 1103, 3013, 1104, 1251, 1769, 2301, 119, 1335, 1103, 1269, 1159, 117, 1103, 2877, 1433, 1144, 5602, 21435, 131, 1103, 7663, 3392, 1110, 1714, 1106, 4958, 1184, 24026, 1122, 1209, 3593, 117, 1112, 1263, 1112, 1122, 12543, 1439, 1157, 4035, 15447, 5894, 3758, 1105, 18788, 1103, 7950, 1193, 4921, 2266, 1104, 2833, 119, 18872, 117, 1103, 9799, 3392, 1144, 170, 2178, 1104, 9799, 21435, 117, 1105, 1103, 3275, 3392, 1145, 1144, 1672, 21435, 3113, 3758, 1259, 16810, 2916, 21435, 119, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'start_positions': 54, 'end_positions': 55}\n"
     ]
    }
   ],
   "source": [
    "print(train_dataset[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8272a1ff-aaff-4932-9266-05bc23d3d925",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Valid data size: 2000 -> 2022\n"
     ]
    }
   ],
   "source": [
    "# valid도 동일하게 수행\n",
    "valid_dataset = valid_examples.map(\n",
    "    prepare_train_features,\n",
    "    batched=True,\n",
    "    num_proc=3,\n",
    "    remove_columns=valid_examples.column_names,\n",
    ")\n",
    "print(f\"Valid data size: {len(valid_examples)} -> {len(valid_dataset)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e2955322-a1f3-4f1d-8c77-611024e1852e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final Data size: Train(8103), Valid(2022)\n"
     ]
    }
   ],
   "source": [
    "print(f\"Final Data size: Train({len(train_dataset)}), Valid({len(valid_dataset)})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e16f9dcf-995a-47f4-bdb2-9867dd7e0870",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "fcdc9703-aac3-48bd-b591-4c42450079c5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': '56cd8d2762d2951400fa66e1',\n",
       " 'title': 'Sino-Tibetan_relations_during_the_Ming_dynasty',\n",
       " 'context': 'Van Praag states that the Ming court established diplomatic delegations with Tibet merely to secure urgently needed horses. Wang and Nyima argue that these were not diplomatic delegations at all, that Tibetan areas were ruled by the Ming since Tibetan leaders were granted positions as Ming officials, that horses were collected from Tibet as a mandatory \"corvée\" tax, and therefore Tibetans were \"undertaking domestic affairs, not foreign diplomacy\". Sperling writes that the Ming simultaneously bought horses in the Kham region while fighting Tibetan tribes in Amdo and receiving Tibetan embassies in Nanjing. He also argues that the embassies of Tibetan lamas visiting the Ming court were for the most part efforts to promote commercial transactions between the lamas\\' large, wealthy entourage and Ming Chinese merchants and officials. Kolmaš writes that while the Ming maintained a laissez-faire policy towards Tibet and limited the numbers of the Tibetan retinues, the Tibetans sought to maintain a tributary relationship with the Ming because imperial patronage provided them with wealth and power. Laird writes that Tibetans eagerly sought Ming court invitations since the gifts the Tibetans received for bringing tribute were much greater in value than the latter. As for the Yongle Emperor\\'s gifts to his Tibetan and Nepalese vassals such as silver wares, Buddha relics, utensils for Buddhist temples and religious ceremonies, and gowns and robes for monks, Tsai writes \"in his effort to draw neighboring states to the Ming orbit so that he could bask in glory, the Yongle Emperor was quite willing to pay a small price\". The Information Office of the State Council of the PRC lists the Tibetan tribute items as oxen, horses, camels, sheep, fur products, medical herbs, Tibetan incenses, thangkas (painted scrolls), and handicrafts; while the Ming awarded Tibetan tribute-bearers an equal value of gold, silver, satin and brocade, bolts of cloth, grains, and tea leaves. Silk workshops during the Ming also catered specifically to the Tibetan market with silk clothes and furnishings featuring Tibetan Buddhist iconography.',\n",
       " 'question': 'who were the Tibetan areas were ruled by?',\n",
       " 'answers': {'text': ['the Ming'], 'answer_start': [22]}}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_examples[53]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37511862-bb8a-4d98-aff2-c148e531cd79",
   "metadata": {},
   "source": [
    "max_len(384) 넘어가는 long sequence이므로 아래와 같이 두개의 feature로 분할됨\n",
    "\n",
    "- 1번째 feature가 정답을 포함하고 2번째 feature는 정답을 포함하지 않아 0(\\[CLS\\])으로 라벨링"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "2deff703-eb1f-4cdc-bf60-cf498c2e0725",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[CLS] who were the Tibetan areas were ruled by? [SEP] Van Praag states that the Ming court established diplomatic delegations with Tibet merely to secure urgently needed horses. Wang and Nyima argue that these were not diplomatic delegations at all, that Tibetan areas were ruled by the Ming since Tibetan leaders were granted positions as Ming officials, that horses were collected from Tibet as a mandatory \" corvée \" tax, and therefore Tibetans were \" undertaking domestic affairs, not foreign diplomacy \". Sperling writes that the Ming simultaneously bought horses in the Kham region while fighting Tibetan tribes in Amdo and receiving Tibetan embassies in Nanjing. He also argues that the embassies of Tibetan lamas visiting the Ming court were for the most part efforts to promote commercial transactions between the lamas\\'large, wealthy entourage and Ming Chinese merchants and officials. Kolmaš writes that while the Ming maintained a laissez - faire policy towards Tibet and limited the numbers of the Tibetan retinues, the Tibetans sought to maintain a tributary relationship with the Ming because imperial patronage provided them with wealth and power. Laird writes that Tibetans eagerly sought Ming court invitations since the gifts the Tibetans received for bringing tribute were much greater in value than the latter. As for the Yongle Emperor\\'s gifts to his Tibetan and Nepalese vassals such as silver wares, Buddha relics, utensils for Buddhist temples and religious ceremonies, and gowns and robes for monks, Tsai writes \" in his effort to draw neighboring states to the Ming orbit so that he could bask in glory, the Yongle Emperor was quite willing to pay a small price \". The Information Office of the State Council of the PRC lists the Tibetan tribute items as oxen, horses, camels, sheep, fur products, medical herbs, Tibetan incenses, thangkas ( painted [SEP]'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(train_dataset[53]['input_ids']) #max_len=100, stride=20 예시"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "4619b52a-438e-4e65-b9aa-9bb0978dc84a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[CLS] who were the Tibetan areas were ruled by? [SEP] the latter. As for the Yongle Emperor\\'s gifts to his Tibetan and Nepalese vassals such as silver wares, Buddha relics, utensils for Buddhist temples and religious ceremonies, and gowns and robes for monks, Tsai writes \" in his effort to draw neighboring states to the Ming orbit so that he could bask in glory, the Yongle Emperor was quite willing to pay a small price \". The Information Office of the State Council of the PRC lists the Tibetan tribute items as oxen, horses, camels, sheep, fur products, medical herbs, Tibetan incenses, thangkas ( painted scrolls ), and handicrafts ; while the Ming awarded Tibetan tribute - bearers an equal value of gold, silver, satin and brocade, bolts of cloth, grains, and tea leaves. Silk workshops during the Ming also catered specifically to the Tibetan market with silk clothes and furnishings featuring Tibetan Buddhist iconography. [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode(train_dataset[54]['input_ids'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "38474ce7-bd46-4dcd-86be-1d3582c08ae0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[17, 0]\n",
      "[18, 0]\n"
     ]
    }
   ],
   "source": [
    "print(train_dataset['start_positions'][53:55])\n",
    "print(train_dataset['end_positions'][53:55])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad206d60-9ea3-4322-aaa4-a5f7c09f42e8",
   "metadata": {},
   "source": [
    "### 2.2. DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "69d8d04a-5d57-48b3-90c8-c81c3e9c6ec5",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size=16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "0e2b2e65-7f68-4157-892b-c812979b6c6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n",
    "valid_dataloader = DataLoader(valid_dataset, batch_size=batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "05d7be72-608f-43d9-bea4-7565d95a3e7a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train iteration: 507, Valid iteration: 127\n"
     ]
    }
   ],
   "source": [
    "print(f\"Train iteration: {len(train_dataloader)}, Valid iteration: {len(valid_dataloader)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "299dcd05-1e21-4b0c-9014-5d28b075676e",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### 예시"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "600b0d75-074e-4f45-8d43-349f479baa10",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_ex_v = next(iter(valid_dataloader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "6f0aea71-9f9d-4960-b396-46a5a086615a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 384])"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reshape_tensor(batch_ex_v['input_ids'], batch_size=2).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "1e316d5f-a714-47e1-8ee1-0e28704cd396",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "384"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(batch_ex['input_ids'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "3c4fe5d8-2e54-4f58-a0d1-509d494ef183",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_ex = next(iter(train_dataloader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "bf70c95d-b389-4768-aebb-6ff610303f0f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 101, 4434, 1121,  ...,    0,    0,    0],\n",
       "        [ 101, 1327, 2578,  ...,    0,    0,    0],\n",
       "        [ 101, 1327, 1583,  ...,    0,    0,    0],\n",
       "        [ 101, 1731, 1242,  ...,    0,    0,    0]])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prob = batch_ex['input_ids']\n",
    "input_ids = torch.concat([prob[i][0].view(1) for i in range(len(prob))]).unsqueeze(0) # 384->[1, 384]\n",
    "\n",
    "for batch_idx in range(1, batch_size):\n",
    "    new_ids = torch.concat([prob[i][batch_idx].view(1) for i in range(len(prob))]).unsqueeze(0) # 384->[1, 384]\n",
    "    input_ids = torch.cat([input_ids , new_ids], dim=0)\n",
    "\n",
    "input_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "f0cbdc8f-8cee-40d6-8145-a6f233dc13ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_ex = next(iter(train_dataloader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "f519f740-803b-47df-81ba-6f6c7402373d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[tensor([101, 101, 101, 101]),\n",
       " tensor([1327, 1327, 2627, 1327]),\n",
       " tensor([1710, 1110, 2234, 1132]),\n",
       " tensor([1108,  170, 1103, 1103]),\n",
       " tensor([1103, 7224,  185, 3501]),\n",
       " tensor([ 8099,  1115, 19456,  1637]),\n",
       " tensor([ 4264, 18028, 21123,  3002]),\n",
       " tensor([1114, 1103, 3855, 1104]),\n",
       " tensor([ 136, 2860, 1154, 1103]),\n",
       " tensor([ 102, 1104, 3352, 7085])]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_ex['input_ids'][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "dd663b9b-cb45-41ee-b918-55336ace285c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[tensor([1, 1, 1, 1]),\n",
       " tensor([1, 1, 1, 1]),\n",
       " tensor([1, 1, 1, 1]),\n",
       " tensor([1, 1, 1, 1]),\n",
       " tensor([1, 1, 1, 1]),\n",
       " tensor([1, 1, 1, 1]),\n",
       " tensor([1, 1, 1, 1]),\n",
       " tensor([1, 1, 1, 1]),\n",
       " tensor([1, 1, 1, 1]),\n",
       " tensor([1, 1, 1, 1])]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_ex['attention_mask'][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "974cdc7f-076d-4f34-9907-2e7ddfc8fc73",
   "metadata": {},
   "outputs": [],
   "source": [
    "def reshape_tensor(org_tensor, batch_size):\n",
    "    new_tensor = torch.concat([org_tensor[i][0].view(1) for i in range(len(org_tensor))]).unsqueeze(0) # 384->[1, 384]\n",
    "    for batch_idx in range(1, batch_size):\n",
    "        new_ids = torch.concat([org_tensor[i][batch_idx].view(1) for i in range(len(org_tensor))]).unsqueeze(0) # 384->[1, 384]\n",
    "        new_tensor = torch.cat([new_tensor, new_ids], dim=0)\n",
    "    return new_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "887e5486-e42c-4a9c-be98-7d411ed3c4b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 101, 1327, 1710,  ...,    0,    0,    0],\n",
       "        [ 101, 1327, 1110,  ...,    0,    0,    0],\n",
       "        [ 101, 2627, 2234,  ...,    0,    0,    0],\n",
       "        [ 101, 1327, 1132,  ...,    0,    0,    0]])"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reshape_tensor(batch_ex['input_ids'], batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "459f125f-fad6-4ad1-a1d1-d6ca2ff4e5c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 384])"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_ids.shape #attention_mask도 똑같이 해라잉"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "13025bb7-4856-45e0-867a-07694c7c9ca1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1, 1, 1,  ..., 0, 0, 0],\n",
       "        [1, 1, 1,  ..., 0, 0, 0],\n",
       "        [1, 1, 1,  ..., 0, 0, 0],\n",
       "        [1, 1, 1,  ..., 0, 0, 0]])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reshape_tensor(batch_ex['attention_mask'], batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f882036-b6eb-4536-8340-a02e3f5bca57",
   "metadata": {},
   "source": [
    "## 3. Fine-tuning the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1f228549-5583-4ea9-bf29-7ead7e5094fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "00e3e805-d772-4112-a4dd-fdda665949b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-cased were not used when initializing DistilBertForQuestionAnswering: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_projector.bias']\n",
      "- This IS expected if you are initializing DistilBertForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForQuestionAnswering were not initialized from the model checkpoint at distilbert-base-cased and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModelForQuestionAnswering\n",
    "\n",
    "model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "14f864c0-f56a-4216-98c5-c676934643f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm.auto import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "1c3ac158-0358-43df-838f-de744f479860",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AdamW\n",
    "\n",
    "optimizer = AdamW(model.parameters(), lr=5e-5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4138ed3c-8258-41a6-a04a-5a907c569a66",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dataloader 구축 이후 input 형태 변화로 인해 tensor reshape로 input 형태 조정\n",
    "### model input 중 input_ids, attention_mask에 활용\n",
    "\n",
    "def reshape_tensor(org_tensor, batch_size):\n",
    "    new_tensor = torch.concat([org_tensor[i][0].view(1) for i in range(len(org_tensor))]).unsqueeze(0) # 384->[1, 384]\n",
    "    for batch_idx in range(1, batch_size):\n",
    "        new_ids = torch.concat([org_tensor[i][batch_idx].view(1) for i in range(len(org_tensor))]).unsqueeze(0) # 384->[1, 384]\n",
    "        new_tensor = torch.cat([new_tensor, new_ids], dim=0)\n",
    "    return new_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "be3392a5-1bfc-43ed-9a5f-dc840f550390",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epoch(model, dataloader, optimizer, device):\n",
    "    model.train()\n",
    "    losses = 0\n",
    "    \n",
    "    for batch_idx, batch in tqdm(enumerate(dataloader)):\n",
    "        # input\n",
    "        batch_size = batch['input_ids'][0].size(0)\n",
    "        \n",
    "        input_ids = reshape_tensor(batch['input_ids'], batch_size).to(device)\n",
    "        attention_mask = reshape_tensor(batch['attention_mask'], batch_size).to(device)\n",
    "        start_positions = batch['start_positions'].to(device)\n",
    "        end_positions = batch['end_positions'].to(device)\n",
    "        \n",
    "        # Output\n",
    "        outputs = model(input_ids, attention_mask=attention_mask, start_positions=start_positions, end_positions=end_positions)\n",
    "        \n",
    "        # Calculate loss, Update parameters\n",
    "        optimizer.zero_grad()\n",
    "        loss = outputs[0] # loss는 Model for QA output으로 바로 반환\n",
    "        \n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        losses += loss.item()\n",
    "    \n",
    "    train_loss = losses / len(dataloader)\n",
    "    \n",
    "    return train_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "5fcc3a80-bd32-4050-91ba-ce758969842c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def valid_epoch(model, dataloader, device):\n",
    "    model.eval()\n",
    "    losses = 0\n",
    "    \n",
    "    for batch_idx, batch in enumerate(dataloader):\n",
    "        # input\n",
    "        batch_size = batch['input_ids'][0].size(0)\n",
    "        \n",
    "        input_ids = reshape_tensor(batch['input_ids'], batch_size).to(device)\n",
    "        attention_mask = reshape_tensor(batch['attention_mask'], batch_size).to(device)\n",
    "        start_positions = batch['start_positions'].to(device)\n",
    "        end_positions = batch['end_positions'].to(device)\n",
    "        \n",
    "        # Output\n",
    "        outputs = model(input_ids, attention_mask=attention_mask, start_positions=start_positions, end_positions=end_positions)\n",
    "        \n",
    "        loss = outputs[0]\n",
    "        \n",
    "        losses += loss.item()\n",
    "        \n",
    "    valid_loss = losses / len(dataloader)\n",
    "    \n",
    "    return valid_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "589b0fcf-4bda-4053-83a6-a948741ce9ff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "83ca163b2172408b8b907e252a512df2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Epoch 0] Train loss: 2.644, Valid loss: 1.708, Epoch time = 135.889s\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a942cc3fd3ac433fbae70a4aee18f522",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Epoch 1] Train loss: 1.274, Valid loss: 1.614, Epoch time = 138.224s\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39556dcb891b42bb9301a4cd16cd0c37",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Epoch 2] Train loss: 0.659, Valid loss: 1.978, Epoch time = 139.553s\n"
     ]
    }
   ],
   "source": [
    "from timeit import default_timer as timer\n",
    "\n",
    "num_epochs = 3\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    start_time = timer()\n",
    "    train_loss = train_epoch(model, train_dataloader, optimizer, device)\n",
    "    end_time = timer()\n",
    "    valid_loss = valid_epoch(model, valid_dataloader, device)\n",
    "    \n",
    "    print((f\"[Epoch {epoch}] Train loss: {train_loss:.3f}, Valid loss: {valid_loss:.3f}, Epoch time = {(end_time - start_time):.3f}s\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "11c18e49-865d-4600-a621-a9cb508d32a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(model, 'distill_bert_qa.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bb9e134-5f3a-46c0-9bf2-92802c729e77",
   "metadata": {},
   "source": [
    "## 4. Inference(Test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "dbd8b90c-c02a-4eb7-b393-0c1694d464fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm.auto import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5f7f581a-b06f-4d40-a840-0056caf12ad8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reusing dataset squad (/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['id', 'title', 'context', 'question', 'answers'],\n",
       "    num_rows: 10570\n",
       "})"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "from datasets import load_dataset\n",
    "\n",
    "datasets = load_dataset(\"squad\", split=\"validation\")\n",
    "datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5921502a-f4d1-4353-8ece-b91a98037a2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 8\n",
    "\n",
    "device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d9a06026-2279-40ee-83cf-403fd6d901b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = torch.load('distill_bert_qa.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56658c2f-851c-466a-8082-07dd3e16c012",
   "metadata": {},
   "source": [
    "### Inference Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "4f6d6c1a-0926-48fc-be29-1511ca0e4a2c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached shuffled indices for dataset at /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453/cache-434e5e0fe7c25b62.arrow\n"
     ]
    }
   ],
   "source": [
    "shuffled_test = datasets.shuffle(seed=602)\n",
    "\n",
    "test_examples = shuffled_test.select([7]) #하나만 뽑아쓰기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "138f7b48-be9e-4f6d-89b6-584ef5ddde2f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': '5730de74f6cb411900e244fd',\n",
       " 'title': 'United_Methodist_Church',\n",
       " 'context': 'Unlike confirmation and profession of faith, Baptism is a sacrament in the UMC. The Book of Discipline of the United Methodist Church directs the local church to offer membership preparation or confirmation classes to all people, including adults. The term confirmation is generally reserved for youth, while some variation on membership class is generally used for adults wishing to join the church. The Book of Discipline normally allows any youth at least completing sixth grade to participate, although the pastor has discretionary authority to allow a younger person to participate. In confirmation and membership preparation classes, students learn about Church and the Methodist-Christian theological tradition in order to profess their ultimate faith in Christ.',\n",
       " 'question': 'How do students learn about the church?',\n",
       " 'answers': {'text': ['confirmation and membership preparation classes',\n",
       "   'In confirmation and membership preparation classes,',\n",
       "   'confirmation and membership preparation classes'],\n",
       "  'answer_start': [591, 588, 591]}}"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_examples[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a694ca3b-31f8-4bc0-b569-81c9f77911ef",
   "metadata": {},
   "source": [
    "- 3 gold answers : answer의 변형에도 모델을 견고히 하기 위해 세 사람에게 답변을 얻음\n",
    "\n",
    "- 평가 지표\n",
    "\n",
    "    - Exact match : 3개 중에 하나로 나왔으면 1, 아니면 0으로 binary accuracy\n",
    "    \n",
    "    - F1 : 단어 단위로 구한 F1-score 3개 중에 max one을 per-question F1-score로 두고 전체 macro average\n",
    "    \n",
    "    - `metric = load_metric(\"squad\")`로 쉽게 계산 가능\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "51c2312a-90fb-4fa9-b55c-d9bd2e5bffa1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_test_examples(examples):\n",
    "    questions = [q.strip() for q in examples[\"question\"]]\n",
    "    inputs = tokenizer(\n",
    "        questions,\n",
    "        examples[\"context\"],\n",
    "        max_length=max_length,\n",
    "        truncation=\"only_second\",\n",
    "        stride=doc_stride,\n",
    "        return_overflowing_tokens=True,\n",
    "        return_offsets_mapping=True,\n",
    "        padding=\"max_length\",\n",
    "    )\n",
    "\n",
    "    sample_map = inputs.pop(\"overflow_to_sample_mapping\")\n",
    "    example_ids = []\n",
    "\n",
    "    for i in range(len(inputs[\"input_ids\"])):\n",
    "        sample_idx = sample_map[i]\n",
    "        example_ids.append(examples[\"id\"][sample_idx])\n",
    "\n",
    "        sequence_ids = inputs.sequence_ids(i)\n",
    "        offset = inputs[\"offset_mapping\"][i]\n",
    "        inputs[\"offset_mapping\"][i] = [\n",
    "            o if sequence_ids[k] == 1 else None for k, o in enumerate(offset)\n",
    "        ]\n",
    "\n",
    "    inputs[\"example_id\"] = example_ids\n",
    "    return inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "277a3dac-0eb0-48bc-b250-f350fd0fd94b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7a6341c8e8064710a2e125b92774f9bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test data size: 1 -> 1\n"
     ]
    }
   ],
   "source": [
    "test_dataset = test_examples.map(\n",
    "    preprocess_test_examples,\n",
    "    batched=True,\n",
    "    remove_columns=test_examples.column_names,\n",
    ")\n",
    "\n",
    "print(f\"Test data size: {len(test_examples)} -> {len(test_dataset)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "fa5efca1-8757-4e97-b1c5-16d5990d579e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "test_dataset_for_model = test_dataset.remove_columns([\"example_id\", \"offset_mapping\"])\n",
    "\n",
    "test_dataloader = DataLoader(test_dataset_for_model, batch_size=batch_size, shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "4196cbf7-c6ea-47b2-86a5-499664b7558b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import collections\n",
    "import random\n",
    "import numpy as np\n",
    "from datasets import load_metric\n",
    "\n",
    "\n",
    "def compute_metrics(start_logits, end_logits, features, examples):\n",
    "    example_to_features = collections.defaultdict(list)\n",
    "    for idx, feature in enumerate(features):\n",
    "        example_to_features[feature[\"example_id\"]].append(idx)\n",
    "    # defaultdict(list,\n",
    "    #             {'5726398589a1e219009ac58b': [0],\n",
    "    #              '571cdcb85efbb31900334e0c': [1],\n",
    "    #              '5730aa52069b531400832221': [2],\n",
    "    #              '572684f5dd62a815002e87fe': [3],\n",
    "    #              '572732f8f1498d1400e8f476': [4],\n",
    "    #              '56beae423aeaaa14008c91f4': [5],\n",
    "    #              ...,\n",
    "    #              '5726fc63dd62a815002e9706': [38, 39],\n",
    "    #              ...,\n",
    "    #              '5729081d3f37b31900477fad': [100]})\n",
    "    metric = load_metric(\"squad\")\n",
    "    predicted_answers = []\n",
    "    n_best = 20\n",
    "    max_answer_length = 30\n",
    "    ex_idx = random.randint(0, len(test_examples)-1) ### 출력용 예시 인덱스 추출\n",
    "\n",
    "    for idx, example in tqdm(enumerate(examples)):\n",
    "        example_id = example[\"id\"]\n",
    "        context = example[\"context\"]\n",
    "        answers = []\n",
    "\n",
    "        # Loop through all features associated with that example\n",
    "        for feature_index in example_to_features[example_id]:\n",
    "            start_logit = start_logits[feature_index]\n",
    "            end_logit = end_logits[feature_index]\n",
    "            offsets = features[feature_index][\"offset_mapping\"]\n",
    "\n",
    "            start_indexes = np.argsort(start_logit)[-1 : -n_best - 1 : -1].tolist()\n",
    "            end_indexes = np.argsort(end_logit)[-1 : -n_best - 1 : -1].tolist()\n",
    "            for start_index in start_indexes:\n",
    "                for end_index in end_indexes:\n",
    "                    # Skip answers that are not fully in the context\n",
    "                    if offsets[start_index] is None or offsets[end_index] is None:\n",
    "                        continue\n",
    "                    # Skip answers with a length that is either < 0 or > max_answer_length\n",
    "                    if (\n",
    "                        end_index < start_index\n",
    "                        or end_index - start_index + 1 > max_answer_length\n",
    "                    ):\n",
    "                        continue\n",
    "\n",
    "                    answer = {\n",
    "                        \"text\": context[offsets[start_index][0] : offsets[end_index][1]],\n",
    "                        \"logit_score\": start_logit[start_index] + end_logit[end_index],\n",
    "                    }\n",
    "                    answers.append(answer)\n",
    "\n",
    "        # Select the answer with the best score\n",
    "        if len(answers) > 0:\n",
    "            best_answer = max(answers, key=lambda x: x[\"logit_score\"])\n",
    "            predicted_answers.append(\n",
    "                {\"id\": example_id, \"prediction_text\": best_answer[\"text\"]}\n",
    "            )\n",
    "        else:\n",
    "            predicted_answers.append({\"id\": example_id, \"prediction_text\": \"\"})\n",
    "        \n",
    "        ### 출력용\n",
    "        if idx == ex_idx:\n",
    "            print(f\"Inference Example\\n\")\n",
    "            print(f\"[Id] {example['id']}\\n[Context] {example['context']}\\n[Question] {example['question']}\")\n",
    "            print(f\"[Real Answers] {example['answers']}\")\n",
    "            print(f\"[Pred Answers] {best_answer}\")\n",
    "\n",
    "    theoretical_answers = [{\"id\": ex[\"id\"], \"answers\": ex[\"answers\"]} for ex in examples]\n",
    "    return metric.compute(predictions=predicted_answers, references=theoretical_answers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "d5d624d4-9087-4269-b43f-d0125afbef31",
   "metadata": {},
   "outputs": [],
   "source": [
    "def reshape_tensor(org_tensor, batch_size):\n",
    "    new_tensor = torch.concat([org_tensor[i][0].view(1) for i in range(len(org_tensor))]).unsqueeze(0) # 384->[1, 384]\n",
    "    for batch_idx in range(1, batch_size):\n",
    "        new_ids = torch.concat([org_tensor[i][batch_idx].view(1) for i in range(len(org_tensor))]).unsqueeze(0) # 384->[1, 384]\n",
    "        new_tensor = torch.cat([new_tensor, new_ids], dim=0)\n",
    "    return new_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "8eea6cfb-4d16-4973-a2a9-ee99b103fc6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_epoch(model, dataloader, features, examples, device):\n",
    "    model.eval()\n",
    "    \n",
    "    starts, ends = [], []\n",
    "    for batch_idx, batch in tqdm(enumerate(dataloader)):\n",
    "        # input\n",
    "        batch_size = batch['input_ids'][0].size(0)\n",
    "        \n",
    "        input_ids = reshape_tensor(batch['input_ids'], batch_size).to(device)\n",
    "        attention_mask = reshape_tensor(batch['attention_mask'], batch_size).to(device)\n",
    "        \n",
    "        # Output\n",
    "        outputs = model(input_ids, attention_mask=attention_mask)\n",
    "        \n",
    "        starts.append(outputs.start_logits) # start_logits: (batch_size, max_len)\n",
    "        ends.append(outputs.end_logits) # end_logits: (batch_size, max_len)\n",
    "        \n",
    "    all_start_logits = torch.cat(starts, dim=0).cpu().detach().numpy()\n",
    "    all_end_logits = torch.cat(ends, dim=0).cpu().detach().numpy()\n",
    "    \n",
    "    dict_metrics = compute_metrics(all_start_logits, all_end_logits, features, examples) ###\n",
    "    \n",
    "    return dict_metrics['exact_match'], dict_metrics['f1']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45d64af6-4359-48d2-a20c-c260fd751f6b",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Good Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "14c6c4b4-fbf5-45de-89cf-cbf34f08a70c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47414bfd271d4bfdb710034a23ad3523",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "97c804ce567d47df959ab650b1d373bc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inference Example\n",
      "\n",
      "[Id] 5729081d3f37b31900477fad\n",
      "[Context] Neutrophils and macrophages are phagocytes that travel throughout the body in pursuit of invading pathogens. Neutrophils are normally found in the bloodstream and are the most abundant type of phagocyte, normally representing 50% to 60% of the total circulating leukocytes. During the acute phase of inflammation, particularly as a result of bacterial infection, neutrophils migrate toward the site of inflammation in a process called chemotaxis, and are usually the first cells to arrive at the scene of infection. Macrophages are versatile cells that reside within tissues and produce a wide array of chemicals including enzymes, complement proteins, and regulatory factors such as interleukin 1. Macrophages also act as scavengers, ridding the body of worn-out cells and other debris, and as antigen-presenting cells that activate the adaptive immune system.\n",
      "[Question] What percentage of leukocytes do neutrophils represent?\n",
      "[Real Answers] {'text': ['50% to 60%', '50% to 60%', '50% to 60%'], 'answer_start': [226, 226, 226]}\n",
      "[Pred Answers] {'text': '50% to 60%', 'logit_score': 19.990376}\n",
      "[Test Result] Exact Match: 100.0, F1: 100.0\n"
     ]
    }
   ],
   "source": [
    "### good\n",
    "test_exact_match, test_f1 = test_epoch(model, test_dataloader, test_dataset, test_examples, device)\n",
    "\n",
    "print(f\"[Test Result] Exact Match: {test_exact_match}, F1: {test_f1}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "ffadd72a-0210-46ec-b680-edf09ced6445",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ceea0d61dc614549b4441930ace18c30",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "daf7fdd79bc24233a561c6f204fdf9c2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inference Example\n",
      "\n",
      "[Id] 572750e8dd62a815002e9af4\n",
      "[Context] The project must adhere to zoning and building code requirements. Constructing a project that fails to adhere to codes does not benefit the owner. Some legal requirements come from malum in se considerations, or the desire to prevent things that are indisputably bad – bridge collapses or explosions. Other legal requirements come from malum prohibitum considerations, or things that are a matter of custom or expectation, such as isolating businesses to a business district and residences to a residential district. An attorney may seek changes or exemptions in the law that governs the land where the building will be built, either by arguing that a rule is inapplicable (the bridge design will not cause a collapse), or that the custom is no longer needed (acceptance of live-work spaces has grown in the community).\n",
      "[Question] Who may seek changes or exemptions in the law that governs the land where the building will be built?\n",
      "[Real Answers] {'text': ['An attorney', 'attorney', 'An attorney'], 'answer_start': [517, 520, 517]}\n",
      "[Pred Answers] {'text': 'An attorney', 'logit_score': 19.63659}\n",
      "[Test Result] Exact Match: 100.0, F1: 100.0\n"
     ]
    }
   ],
   "source": [
    "### good\n",
    "test_exact_match, test_f1 = test_epoch(model, test_dataloader, test_dataset, test_examples, device)\n",
    "\n",
    "print(f\"[Test Result] Exact Match: {test_exact_match}, F1: {test_f1}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddfb6513-0e48-4f06-b61d-a00834f8de95",
   "metadata": {},
   "source": [
    "#### BAD Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "65b58ca0-1ab2-4653-a9c2-f905e00341fb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2eac071dd2c84e518f7b3953583e1a93",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4877510a64ec4b009de822a37e92231b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inference Example\n",
      "\n",
      "[Id] 5730de74f6cb411900e244fd\n",
      "[Context] Unlike confirmation and profession of faith, Baptism is a sacrament in the UMC. The Book of Discipline of the United Methodist Church directs the local church to offer membership preparation or confirmation classes to all people, including adults. The term confirmation is generally reserved for youth, while some variation on membership class is generally used for adults wishing to join the church. The Book of Discipline normally allows any youth at least completing sixth grade to participate, although the pastor has discretionary authority to allow a younger person to participate. In confirmation and membership preparation classes, students learn about Church and the Methodist-Christian theological tradition in order to profess their ultimate faith in Christ.\n",
      "[Question] How do students learn about the church?\n",
      "[Real Answers] {'text': ['confirmation and membership preparation classes', 'In confirmation and membership preparation classes,', 'confirmation and membership preparation classes'], 'answer_start': [591, 588, 591]}\n",
      "[Pred Answers] {'text': 'In confirmation and membership preparation classes, students learn about Church and the Methodist-Christian theological tradition', 'logit_score': 6.958498}\n",
      "[Test Result] Exact Match: 0.0, F1: 60.0\n"
     ]
    }
   ],
   "source": [
    "### bad\n",
    "test_exact_match, test_f1 = test_epoch(model, test_dataloader, test_dataset, test_examples, device)\n",
    "\n",
    "print(f\"[Test Result] Exact Match: {test_exact_match}, F1: {test_f1}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "70810921-1007-4e39-a0a2-1da04ecb4d74",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a4e3097d7d9c43e8ba329a6db0ab6514",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "deb07a304636480e8c763637ced9ddf2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inference Example\n",
      "\n",
      "[Id] 5726398589a1e219009ac58b\n",
      "[Context] Connection-oriented transmission requires a setup phase in each involved node before any packet is transferred to establish the parameters of communication. The packets include a connection identifier rather than address information and are negotiated between endpoints so that they are delivered in order and with error checking. Address information is only transferred to each node during the connection set-up phase, when the route to the destination is discovered and an entry is added to the switching table in each network node through which the connection passes. The signaling protocols used allow the application to specify its requirements and discover link parameters. Acceptable values for service parameters may be negotiated. Routing a packet requires the node to look up the connection id in a table. The packet header can be small, as it only needs to contain this code and any information, such as length, timestamp, or sequence number, which is different for different packets.\n",
      "[Question] Is the packet header long \n",
      "[Real Answers] {'text': ['The packet header can be small, as it only needs to contain this code and any information, such as length, timestamp, or sequence number', 'The packet header can be small', 'small'], 'answer_start': [816, 816, 841]}\n",
      "[Pred Answers] {'text': 'Routing a packet requires the node to look up the connection id in a table', 'logit_score': -0.50192857}\n",
      "[Test Result] Exact Match: 0.0, F1: 12.500000000000004\n"
     ]
    }
   ],
   "source": [
    "### bad\n",
    "\n",
    "test_exact_match, test_f1 = test_epoch(model, test_dataloader, test_dataset, test_examples, device)\n",
    "\n",
    "print(f\"[Test Result] Exact Match: {test_exact_match}, F1: {test_f1}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}