pytorch/06_bert/train_bert_nsmc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d7ad3f7f",
   "metadata": {},
   "source": [
    "- notebook : https://github.com/donarts/sourcecode/blob/main/pytorch/06_bert\n",
    "- train : https://swlock.blogspot.com/2023/03/bert-nsmc-pytorch-train.html\n",
    "- predict : https://swlock.blogspot.com/2023/03/bert-nsmc-pytorch-predict.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f7d55f63",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from Korpora import Korpora\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a09a8f5",
   "metadata": {},
   "source": [
    "fineturning할 nsmc 데이터를 가져옵니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a05b455e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    Korpora 는 다른 분들이 연구 목적으로 공유해주신 말뭉치들을\n",
      "    손쉽게 다운로드, 사용할 수 있는 기능만을 제공합니다.\n",
      "\n",
      "    말뭉치들을 공유해 주신 분들에게 감사드리며, 각 말뭉치 별 설명과 라이센스를 공유 드립니다.\n",
      "    해당 말뭉치에 대해 자세히 알고 싶으신 분은 아래의 description 을 참고,\n",
      "    해당 말뭉치를 연구/상용의 목적으로 이용하실 때에는 아래의 라이센스를 참고해 주시기 바랍니다.\n",
      "\n",
      "    # Description\n",
      "    Author : e9t@github\n",
      "    Repository : https://github.com/e9t/nsmc\n",
      "    References : www.lucypark.kr/docs/2015-pyconkr/#39\n",
      "\n",
      "    Naver sentiment movie corpus v1.0\n",
      "    This is a movie review dataset in the Korean language.\n",
      "    Reviews were scraped from Naver Movies.\n",
      "\n",
      "    The dataset construction is based on the method noted in\n",
      "    [Large movie review dataset][^1] from Maas et al., 2011.\n",
      "\n",
      "    [^1]: http://ai.stanford.edu/~amaas/data/sentiment/\n",
      "\n",
      "    # License\n",
      "    CC0 1.0 Universal (CC0 1.0) Public Domain Dedication\n",
      "    Details in https://creativecommons.org/publicdomain/zero/1.0/\n",
      "\n",
      "[Korpora] Corpus `nsmc` is already installed at C:\\Users\\jun\\Korpora\\nsmc\\ratings_train.txt\n",
      "[Korpora] Corpus `nsmc` is already installed at C:\\Users\\jun\\Korpora\\nsmc\\ratings_test.txt\n"
     ]
    }
   ],
   "source": [
    "NSMC = Korpora.load('nsmc')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "913825b6",
   "metadata": {},
   "source": [
    "dataframe 에 넣어봅니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a5efaf1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = pd.DataFrame({\"texts\":NSMC.train.texts, \"labels\":NSMC.train.labels})\n",
    "test_data = pd.DataFrame({\"texts\":NSMC.test.texts, \"labels\":NSMC.test.labels})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a09fec23",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>texts</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>아 더빙.. 진짜 짜증나네요 목소리</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>흠...포스터보고 초딩영화줄....오버연기조차 가볍지 않구나</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>너무재밓었다그래서보는것을추천한다</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>교도소 이야기구먼 ..솔직히 재미는 없다..평점 조정</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>사이몬페그의 익살스런 연기가 돋보였던 영화!스파이더맨에서 늙어보이기만 했던 커스틴 ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149995</th>\n",
       "      <td>인간이 문제지.. 소는 뭔죄인가..</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149996</th>\n",
       "      <td>평점이 너무 낮아서...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149997</th>\n",
       "      <td>이게 뭐요? 한국인은 거들먹거리고 필리핀 혼혈은 착하다?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149998</th>\n",
       "      <td>청춘 영화의 최고봉.방황과 우울했던 날들의 자화상</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149999</th>\n",
       "      <td>한국 영화 최초로 수간하는 내용이 담긴 영화</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>150000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    texts  labels\n",
       "0                                     아 더빙.. 진짜 짜증나네요 목소리       0\n",
       "1                       흠...포스터보고 초딩영화줄....오버연기조차 가볍지 않구나       1\n",
       "2                                       너무재밓었다그래서보는것을추천한다       0\n",
       "3                           교도소 이야기구먼 ..솔직히 재미는 없다..평점 조정       0\n",
       "4       사이몬페그의 익살스런 연기가 돋보였던 영화!스파이더맨에서 늙어보이기만 했던 커스틴 ...       1\n",
       "...                                                   ...     ...\n",
       "149995                                인간이 문제지.. 소는 뭔죄인가..       0\n",
       "149996                                      평점이 너무 낮아서...       1\n",
       "149997                    이게 뭐요? 한국인은 거들먹거리고 필리핀 혼혈은 착하다?       0\n",
       "149998                        청춘 영화의 최고봉.방황과 우울했던 날들의 자화상       1\n",
       "149999                           한국 영화 최초로 수간하는 내용이 담긴 영화       0\n",
       "\n",
       "[150000 rows x 2 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1fccb66a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>texts</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>굳 ㅋ</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GDNTOPCLASSINTHECLUB</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>뭐야 이 평점들은.... 나쁘진 않지만 10점 짜리는 더더욱 아니잖아</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>지루하지는 않은데 완전 막장임... 돈주고 보기에는....</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3D만 아니었어도 별 다섯 개 줬을텐데.. 왜 3D로 나와서 제 심기를 불편하게 하죠??</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49995</th>\n",
       "      <td>오랜만에 평점 로긴했네ㅋㅋ 킹왕짱 쌈뽕한 영화를 만났습니다 강렬하게 육쾌함</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49996</th>\n",
       "      <td>의지 박약들이나 하는거다 탈영은 일단 주인공 김대희 닮았고 이등병 찐따 OOOO</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49997</th>\n",
       "      <td>그림도 좋고 완성도도 높았지만... 보는 내내 불안하게 만든다</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>절대 봐서는 안 될 영화.. 재미도 없고 기분만 잡치고.. 한 세트장에서 다 해먹네</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49999</th>\n",
       "      <td>마무리는 또 왜이래</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>50000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   texts  labels\n",
       "0                                                    굳 ㅋ       1\n",
       "1                                   GDNTOPCLASSINTHECLUB       0\n",
       "2                 뭐야 이 평점들은.... 나쁘진 않지만 10점 짜리는 더더욱 아니잖아       0\n",
       "3                       지루하지는 않은데 완전 막장임... 돈주고 보기에는....       0\n",
       "4      3D만 아니었어도 별 다섯 개 줬을텐데.. 왜 3D로 나와서 제 심기를 불편하게 하죠??       0\n",
       "...                                                  ...     ...\n",
       "49995          오랜만에 평점 로긴했네ㅋㅋ 킹왕짱 쌈뽕한 영화를 만났습니다 강렬하게 육쾌함       1\n",
       "49996       의지 박약들이나 하는거다 탈영은 일단 주인공 김대희 닮았고 이등병 찐따 OOOO       0\n",
       "49997                 그림도 좋고 완성도도 높았지만... 보는 내내 불안하게 만든다       0\n",
       "49998     절대 봐서는 안 될 영화.. 재미도 없고 기분만 잡치고.. 한 세트장에서 다 해먹네       0\n",
       "49999                                         마무리는 또 왜이래       0\n",
       "\n",
       "[50000 rows x 2 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2584c0d2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "158"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max(len(l) for l in train_data['texts'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "36c9201e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "152"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max(len(l) for l in test_data['texts'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bfbfab7",
   "metadata": {},
   "source": [
    "여기에서 train/test 데이터가 너무 많아서 학습이 오래걸려 1/10 으로 줄여서 진행합니다.\n",
    "(이 코드는 샘플이므로)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "af26fadc",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = train_data.head(int(len(train_data)/10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f2022d87",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data = test_data.head(int(len(test_data)/10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db89741a",
   "metadata": {},
   "source": [
    "학습에 사용될 pre-trained 된 BERT 모델을 가져와서 토큰화 하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "069729f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "pretrained_model_name=\"beomi/kcbert-base\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "43b8c428",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "56135201",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 경고가 뜬다면 다음 명령으로 설치해주자 !pip install ipywidgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "907b89b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "    pretrained_model_name\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "37c7c0a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_train_sentences = tokenizer(\n",
    "    list(train_data.texts),\n",
    "    return_tensors=\"pt\",\n",
    "    padding=True,\n",
    "    truncation=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "26cadf6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_test_sentences = tokenizer(\n",
    "    list(test_data.texts),\n",
    "    return_tensors=\"pt\",\n",
    "    padding=True,\n",
    "    truncation=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bec1403d",
   "metadata": {},
   "source": [
    "출력해봅니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d59edf01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['input_ids', 'token_type_ids', 'attention_mask'])\n",
      "tensor([[    2,  2170,   832,  ...,     0,     0,     0],\n",
      "        [    2,  3521,    17,  ...,     0,     0,     0],\n",
      "        [    2,  8069,  4089,  ...,     0,     0,     0],\n",
      "        ...,\n",
      "        [    2,    43, 17697,  ...,     0,     0,     0],\n",
      "        [    2,  2477,  4116,  ...,     0,     0,     0],\n",
      "        [    2,  2170,  4565,  ...,     0,     0,     0]])\n",
      "tensor([[1, 1, 1,  ..., 0, 0, 0],\n",
      "        [1, 1, 1,  ..., 0, 0, 0],\n",
      "        [1, 1, 1,  ..., 0, 0, 0],\n",
      "        ...,\n",
      "        [1, 1, 1,  ..., 0, 0, 0],\n",
      "        [1, 1, 1,  ..., 0, 0, 0],\n",
      "        [1, 1, 1,  ..., 0, 0, 0]])\n",
      "tensor([[0, 0, 0,  ..., 0, 0, 0],\n",
      "        [0, 0, 0,  ..., 0, 0, 0],\n",
      "        [0, 0, 0,  ..., 0, 0, 0],\n",
      "        ...,\n",
      "        [0, 0, 0,  ..., 0, 0, 0],\n",
      "        [0, 0, 0,  ..., 0, 0, 0],\n",
      "        [0, 0, 0,  ..., 0, 0, 0]])\n"
     ]
    }
   ],
   "source": [
    "print(tokenized_train_sentences.keys())\n",
    "print(tokenized_train_sentences['input_ids'])\n",
    "print(tokenized_train_sentences['attention_mask'])\n",
    "print(tokenized_train_sentences['token_type_ids'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "a1978d28",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_label = train_data['labels'].values\n",
    "test_label = test_data['labels'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a43f00c2",
   "metadata": {},
   "source": [
    "데이터 로더 준비, 이게 필요한 이유는 배치 처리하는 내부에서 원소를 액세스 하기 위함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "099da9f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "class DataloaderDataset(torch.utils.data.Dataset):\n",
    "    def __init__(self, encodings, labels):\n",
    "        self.encodings = encodings\n",
    "        self.labels = labels\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}\n",
    "        item['labels'] = torch.tensor(self.labels[idx])\n",
    "        return item\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a094cde8",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = DataloaderDataset(tokenized_train_sentences, train_label)\n",
    "test_dataset = DataloaderDataset(tokenized_test_sentences, test_label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6b2afc8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BertConfig, AutoModelForSequenceClassification, Trainer, TrainingArguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6ab6e455",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda:0\n"
     ]
    }
   ],
   "source": [
    "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ce8a3eed",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at beomi/kcbert-base were not used when initializing BertForSequenceClassification: ['cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at beomi/kcbert-base and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "pretrained_model_config = BertConfig.from_pretrained(\n",
    "    pretrained_model_name,\n",
    ")\n",
    "model = AutoModelForSequenceClassification.from_pretrained(\n",
    "        pretrained_model_name,\n",
    "        config=pretrained_model_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "6884b3c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BertConfig {\n",
       "  \"_name_or_path\": \"beomi/kcbert-base\",\n",
       "  \"architectures\": [\n",
       "    \"BertForMaskedLM\"\n",
       "  ],\n",
       "  \"attention_probs_dropout_prob\": 0.1,\n",
       "  \"classifier_dropout\": null,\n",
       "  \"directionality\": \"bidi\",\n",
       "  \"gradient_checkpointing\": false,\n",
       "  \"hidden_act\": \"gelu\",\n",
       "  \"hidden_dropout_prob\": 0.1,\n",
       "  \"hidden_size\": 768,\n",
       "  \"initializer_range\": 0.02,\n",
       "  \"intermediate_size\": 3072,\n",
       "  \"layer_norm_eps\": 1e-12,\n",
       "  \"max_position_embeddings\": 300,\n",
       "  \"model_type\": \"bert\",\n",
       "  \"num_attention_heads\": 12,\n",
       "  \"num_hidden_layers\": 12,\n",
       "  \"pad_token_id\": 0,\n",
       "  \"pooler_fc_size\": 768,\n",
       "  \"pooler_num_attention_heads\": 12,\n",
       "  \"pooler_num_fc_layers\": 3,\n",
       "  \"pooler_size_per_head\": 128,\n",
       "  \"pooler_type\": \"first_token_transform\",\n",
       "  \"position_embedding_type\": \"absolute\",\n",
       "  \"transformers_version\": \"4.10.0\",\n",
       "  \"type_vocab_size\": 2,\n",
       "  \"use_cache\": true,\n",
       "  \"vocab_size\": 30000\n",
       "}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pretrained_model_config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "a7060de8",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install evaluate\n",
    "#!pip install scikit-learn\n",
    "import numpy as np\n",
    "import evaluate \n",
    "metric = evaluate.load(\"accuracy\")\n",
    "def compute_metrics(eval_pred):\n",
    "    logits, labels = eval_pred\n",
    "    predictions = np.argmax(logits, axis=-1)\n",
    "    return metric.compute(predictions=predictions, references=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "d31c3308",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "PyTorch: setting up devices\n",
      "The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).\n"
     ]
    }
   ],
   "source": [
    "training_args = TrainingArguments(\n",
    "    output_dir='./results',          # output directory\n",
    "    num_train_epochs=1,              # total number of training epochs\n",
    "    #per_device_train_batch_size=32,  # batch size per device during training\n",
    "    #per_device_eval_batch_size=64,   # batch size for evaluation\n",
    "    per_device_train_batch_size=16,  # batch size per device during training\n",
    "    per_device_eval_batch_size=16,   # batch size for evaluation\n",
    "    warmup_steps=500,                # number of warmup steps for learning rate scheduler\n",
    "    weight_decay=0.01,               # strength of weight decay\n",
    "    logging_dir='./logs',            # directory for storing logs\n",
    "    logging_steps=100,\n",
    "    save_steps=200,\n",
    "    save_total_limit=2,\n",
    "    save_on_each_node=True,\n",
    "    do_train=True,                   # Perform training\n",
    "    do_eval=True,                    # Perform evaluation\n",
    "    evaluation_strategy=\"epoch\",\n",
    "    seed=3\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "539b7e6b",
   "metadata": {},
   "source": [
    "좀 더 많은 인자는 아래 링크에서 확인"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fe80e3a",
   "metadata": {},
   "source": [
    "https://huggingface.co/docs/transformers/v4.19.2/en/main_classes/trainer#transformers.TrainingArguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "3c0e89dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer = Trainer(\n",
    "    model=model,                         # the instantiated 🤗 Transformers model to be trained\n",
    "    args=training_args,                  # training arguments, defined above\n",
    "    train_dataset=train_dataset,         # training dataset\n",
    "    eval_dataset=test_dataset,\n",
    "    compute_metrics=compute_metrics,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "cde846d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running training *****\n",
      "  Num examples = 15000\n",
      "  Num Epochs = 1\n",
      "  Instantaneous batch size per device = 16\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 16\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 938\n",
      "C:\\Users\\jun\\AppData\\Local\\Temp\\ipykernel_27736\\1263192275.py:7: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='938' max='938' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [938/938 04:42, Epoch 1/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.344700</td>\n",
       "      <td>0.314100</td>\n",
       "      <td>0.867800</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to ./results\\checkpoint-200\n",
      "Configuration saved in ./results\\checkpoint-200\\config.json\n",
      "Model weights saved in ./results\\checkpoint-200\\pytorch_model.bin\n",
      "C:\\Users\\jun\\AppData\\Local\\Temp\\ipykernel_27736\\1263192275.py:7: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}\n",
      "Saving model checkpoint to ./results\\checkpoint-400\n",
      "Configuration saved in ./results\\checkpoint-400\\config.json\n",
      "Model weights saved in ./results\\checkpoint-400\\pytorch_model.bin\n",
      "Deleting older checkpoint [results\\checkpoint-500] due to args.save_total_limit\n",
      "C:\\Users\\jun\\AppData\\Local\\Temp\\ipykernel_27736\\1263192275.py:7: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}\n",
      "Saving model checkpoint to ./results\\checkpoint-600\n",
      "Configuration saved in ./results\\checkpoint-600\\config.json\n",
      "Model weights saved in ./results\\checkpoint-600\\pytorch_model.bin\n",
      "Deleting older checkpoint [results\\checkpoint-200] due to args.save_total_limit\n",
      "C:\\Users\\jun\\AppData\\Local\\Temp\\ipykernel_27736\\1263192275.py:7: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}\n",
      "Saving model checkpoint to ./results\\checkpoint-800\n",
      "Configuration saved in ./results\\checkpoint-800\\config.json\n",
      "Model weights saved in ./results\\checkpoint-800\\pytorch_model.bin\n",
      "Deleting older checkpoint [results\\checkpoint-400] due to args.save_total_limit\n",
      "C:\\Users\\jun\\AppData\\Local\\Temp\\ipykernel_27736\\1263192275.py:7: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 5000\n",
      "  Batch size = 16\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=938, training_loss=0.3618158074075988, metrics={'train_runtime': 283.118, 'train_samples_per_second': 52.981, 'train_steps_per_second': 3.313, 'total_flos': 824791491900000.0, 'train_loss': 0.3618158074075988, 'epoch': 1.0})"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "03ad7707",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to trained_model\n",
      "Configuration saved in trained_model\\config.json\n",
      "Model weights saved in trained_model\\pytorch_model.bin\n"
     ]
    }
   ],
   "source": [
    "trainer.save_model(\"trained_model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7624ab2c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}