NarrativeKoGPT2.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"NarrativeKoGPT2.ipynb","provenance":[{"file_id":"1ChkS56shytnc2x8rXwaYwhhMJYPEQxaT","timestamp":1584444701945}],"collapsed_sections":[],"machine_shape":"hm","authorship_tag":"ABX9TyN2F5JshFIe4WPyUZzuAN8N"},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"K0p6JShGUnpZ","colab_type":"text"},"source":["# NarrativeKoGPT2 학습"]},{"cell_type":"markdown","metadata":{"id":"JV7_Ye-1UuS2","colab_type":"text"},"source":["## 1.Google Drive 연동\n","- 모델 파일과 학습 데이터가 저장 되어있는 구글 드라이브의 디렉토리와 Colab을 연동.  \n","- 좌측상단 메뉴에서 런타임-> 런타임 유형 변경 -> 하드웨어 가속기 -> GPU 선택 후 저장"]},{"cell_type":"markdown","metadata":{"id":"18LqQI0SVNX9","colab_type":"text"},"source":["### 1.1 GPU 연동 확인"]},{"cell_type":"code","metadata":{"id":"CmKD5vgYUeTa","colab_type":"code","outputId":"b0ad3aed-4402-47d1-baf0-fad7f14a6c7f","executionInfo":{"status":"ok","timestamp":1584698768724,"user_tz":-540,"elapsed":2666,"user":{"displayName":"김성환","photoUrl":"","userId":"17497395371430681608"}},"colab":{"base_uri":"https://localhost:8080/","height":306}},"source":["!nvidia-smi"],"execution_count":1,"outputs":[{"output_type":"stream","text":["Fri Mar 20 10:06:07 2020       \n","+-----------------------------------------------------------------------------+\n","| NVIDIA-SMI 440.64.00    Driver Version: 418.67       CUDA Version: 10.1     |\n","|-------------------------------+----------------------+----------------------+\n","| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n","| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n","|===============================+======================+======================|\n","|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n","| N/A   38C    P0    27W / 250W |      0MiB / 16280MiB |      0%      Default |\n","+-------------------------------+----------------------+----------------------+\n","                                                                               \n","+-----------------------------------------------------------------------------+\n","| Processes:                                                       GPU Memory |\n","|  GPU       PID   Type   Process name                             Usage      |\n","|=============================================================================|\n","|  No running processes found                                                 |\n","+-----------------------------------------------------------------------------+\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Vi2gIIroVXeS","colab_type":"text"},"source":["### 1.2 Google Drive 연동\n","아래 코드를 실행후 나오는 URL을 클릭하여 나오는 인증 코드 입력"]},{"cell_type":"code","metadata":{"id":"n2tPgkJzUmBF","colab_type":"code","outputId":"e36dff32-b78c-411c-d07e-247bcdc22715","executionInfo":{"status":"ok","timestamp":1584698791269,"user_tz":-540,"elapsed":21888,"user":{"displayName":"김성환","photoUrl":"","userId":"17497395371430681608"}},"colab":{"base_uri":"https://localhost:8080/","height":122}},"source":["from google.colab import drive\n","drive.mount('/content/drive')"],"execution_count":2,"outputs":[{"output_type":"stream","text":["Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n","\n","Enter your authorization code:\n","··········\n","Mounted at /content/drive\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"Pg07ZiFiVjJU","colab_type":"code","outputId":"31b99529-688a-498a-895c-396a96bac8d0","executionInfo":{"status":"ok","timestamp":1584682733302,"user_tz":-540,"elapsed":4377,"user":{"displayName":"김성환","photoUrl":"","userId":"17497395371430681608"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["!ls"],"execution_count":0,"outputs":[{"output_type":"stream","text":["drive  sample_data\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"KS8lBXaKR6TF","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"kpCSwfFGkRx7","colab_type":"text"},"source":["**Colab 디렉토리 아래 NarrativeKoGPT2 경로 확인**"]},{"cell_type":"code","metadata":{"id":"a7arJ4k2XLG_","colab_type":"code","outputId":"b27dd97a-5cd5-4c7a-f9f4-c9fc5ec61476","executionInfo":{"status":"ok","timestamp":1584690917331,"user_tz":-540,"elapsed":2185,"user":{"displayName":"김성환","photoUrl":"","userId":"17497395371430681608"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["!ls drive/'My Drive'/'Colab Notebooks'/"],"execution_count":3,"outputs":[{"output_type":"stream","text":[" BERT_X\t\t\t\t  KorQuAD-beginner   NarrativeKoGPT2.ipynb\n","'Copy of NarrativeKoGPT2.ipynb'   NarrativeKoGPT2\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"RVmhNd21kse2","colab_type":"text"},"source":["**필요 패키지들 설치**"]},{"cell_type":"code","metadata":{"id":"FDrIL81uXPB0","colab_type":"code","outputId":"adbcd3a7-235c-456f-ece6-04b4aa5f00b1","executionInfo":{"status":"ok","timestamp":1584698816577,"user_tz":-540,"elapsed":20771,"user":{"displayName":"김성환","photoUrl":"","userId":"17497395371430681608"}},"colab":{"base_uri":"https://localhost:8080/","height":972}},"source":["!pip install -r drive/'My Drive'/'Colab Notebooks'/NarrativeKoGPT2/requirements.txt"],"execution_count":3,"outputs":[{"output_type":"stream","text":["Collecting gluonnlp>=0.8.3\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/c6/27/07b57d22496ed6c98b247e578712122402487f5c265ec70a747900f97060/gluonnlp-0.9.1.tar.gz (252kB)\n","\u001b[K     |████████████████████████████████| 256kB 5.1MB/s \n","\u001b[?25hCollecting mxnet\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/81/f5/d79b5b40735086ff1100c680703e0f3efc830fa455e268e9e96f3c857e93/mxnet-1.6.0-py2.py3-none-any.whl (68.7MB)\n","\u001b[K     |████████████████████████████████| 68.7MB 44kB/s \n","\u001b[?25hCollecting sentencepiece>=0.1.6\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/74/f4/2d5214cbf13d06e7cb2c20d84115ca25b53ea76fa1f0ade0e3c9749de214/sentencepiece-0.1.85-cp36-cp36m-manylinux1_x86_64.whl (1.0MB)\n","\u001b[K     |████████████████████████████████| 1.0MB 54.3MB/s \n","\u001b[?25hRequirement already satisfied: torch>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from -r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 4)) (1.4.0)\n","Collecting transformers>=2.1.1\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/13/33/ffb67897a6985a7b7d8e5e7878c3628678f553634bd3836404fef06ef19b/transformers-2.5.1-py3-none-any.whl (499kB)\n","\u001b[K     |████████████████████████████████| 501kB 51.3MB/s \n","\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from -r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 6)) (4.38.0)\n","Requirement already satisfied: numpy>=1.16.0 in /usr/local/lib/python3.6/dist-packages (from gluonnlp>=0.8.3->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 1)) (1.18.2)\n","Requirement already satisfied: cython in /usr/local/lib/python3.6/dist-packages (from gluonnlp>=0.8.3->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 1)) (0.29.15)\n","Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from gluonnlp>=0.8.3->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 1)) (20.3)\n","Collecting graphviz<0.9.0,>=0.8.1\n","  Downloading https://files.pythonhosted.org/packages/53/39/4ab213673844e0c004bed8a0781a0721a3f6bb23eb8854ee75c236428892/graphviz-0.8.4-py2.py3-none-any.whl\n","Requirement already satisfied: requests<3,>=2.20.0 in /usr/local/lib/python3.6/dist-packages (from mxnet->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 2)) (2.21.0)\n","Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from transformers>=2.1.1->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 5)) (1.12.23)\n","Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers>=2.1.1->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 5)) (2019.12.20)\n","Collecting tokenizers==0.5.2\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d1/3f/73c881ea4723e43c1e9acf317cf407fab3a278daab3a69c98dcac511c04f/tokenizers-0.5.2-cp36-cp36m-manylinux1_x86_64.whl (3.7MB)\n","\u001b[K     |████████████████████████████████| 3.7MB 56.7MB/s \n","\u001b[?25hCollecting sacremoses\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a6/b4/7a41d630547a4afd58143597d5a49e07bfd4c42914d8335b2a5657efc14b/sacremoses-0.0.38.tar.gz (860kB)\n","\u001b[K     |████████████████████████████████| 870kB 57.7MB/s \n","\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers>=2.1.1->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 5)) (3.0.12)\n","Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->gluonnlp>=0.8.3->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 1)) (2.4.6)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from packaging->gluonnlp>=0.8.3->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 1)) (1.12.0)\n","Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.20.0->mxnet->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 2)) (1.24.3)\n","Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.20.0->mxnet->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 2)) (2.8)\n","Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.20.0->mxnet->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 2)) (3.0.4)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.20.0->mxnet->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 2)) (2019.11.28)\n","Requirement already satisfied: botocore<1.16.0,>=1.15.23 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers>=2.1.1->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 5)) (1.15.23)\n","Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers>=2.1.1->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 5)) (0.3.3)\n","Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->transformers>=2.1.1->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 5)) (0.9.5)\n","Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers>=2.1.1->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 5)) (7.1.1)\n","Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers>=2.1.1->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 5)) (0.14.1)\n","Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.16.0,>=1.15.23->boto3->transformers>=2.1.1->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 5)) (0.15.2)\n","Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.16.0,>=1.15.23->boto3->transformers>=2.1.1->-r drive/My Drive/Colab Notebooks/NarrativeKoGPT2/requirements.txt (line 5)) (2.8.1)\n","Building wheels for collected packages: gluonnlp, sacremoses\n","  Building wheel for gluonnlp (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for gluonnlp: filename=gluonnlp-0.9.1-cp36-cp36m-linux_x86_64.whl size=470974 sha256=1952a212eebb3e8949417908c0f399ab4d383f3dbc0acedd4f219c440260ba14\n","  Stored in directory: /root/.cache/pip/wheels/af/60/16/1f8a40e68b85bd9bd7960e91830bca5e40cd113f3220b7e231\n","  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for sacremoses: filename=sacremoses-0.0.38-cp36-none-any.whl size=884628 sha256=4cc8105063ef093f8c730b0326be3e98f77791a6f5aec4852170b12d4bcc695c\n","  Stored in directory: /root/.cache/pip/wheels/6d/ec/1a/21b8912e35e02741306f35f66c785f3afe94de754a0eaf1422\n","Successfully built gluonnlp sacremoses\n","Installing collected packages: gluonnlp, graphviz, mxnet, sentencepiece, tokenizers, sacremoses, transformers\n","  Found existing installation: graphviz 0.10.1\n","    Uninstalling graphviz-0.10.1:\n","      Successfully uninstalled graphviz-0.10.1\n","Successfully installed gluonnlp-0.9.1 graphviz-0.8.4 mxnet-1.6.0 sacremoses-0.0.38 sentencepiece-0.1.85 tokenizers-0.5.2 transformers-2.5.1\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"vSCmVmaTlZ5S","colab_type":"code","outputId":"0337cecb-5240-4c4e-8b6c-42114cb45d96","executionInfo":{"status":"ok","timestamp":1584698825809,"user_tz":-540,"elapsed":875,"user":{"displayName":"김성환","photoUrl":"","userId":"17497395371430681608"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["import os\n","\n","import sys\n","sys.path.append('drive/My Drive/Colab Notebooks/')\n","print(os.getcwd())\n"],"execution_count":4,"outputs":[{"output_type":"stream","text":["/content\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"1hnSOCChk9lU","colab_type":"text"},"source":["## 2.KoGPT2 Transfer Learning"]},{"cell_type":"markdown","metadata":{"id":"OL6xVLtHn6vK","colab_type":"text"},"source":["### 2.1.Import Package"]},{"cell_type":"code","metadata":{"id":"-IGI-Rcakhsw","colab_type":"code","outputId":"43ac441c-c395-4de8-b814-af71aa13a731","executionInfo":{"status":"ok","timestamp":1584698842737,"user_tz":-540,"elapsed":13813,"user":{"displayName":"김성환","photoUrl":"","userId":"17497395371430681608"}},"colab":{"base_uri":"https://localhost:8080/","height":80}},"source":["import random\n","import torch\n","from torch.utils.data import DataLoader # 데이터로더\n","from gluonnlp.data import SentencepieceTokenizer \n","from NarrativeKoGPT2.kogpt2.utils import get_tokenizer\n","from NarrativeKoGPT2.kogpt2.utils import download, tokenizer\n","from NarrativeKoGPT2.model.torch_gpt2 import GPT2Config, GPT2LMHeadModel\n","from NarrativeKoGPT2.util.data import NovelDataset\n","import gluonnlp\n","from tqdm import tqdm"],"execution_count":5,"outputs":[{"output_type":"display_data","data":{"text/html":["<p style=\"color: red;\">\n","The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n","We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n","or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n","<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{"tags":[]}}]},{"cell_type":"markdown","metadata":{"id":"01vGgfaNIDT_","colab_type":"text"},"source":["**torch GPU 확인**"]},{"cell_type":"code","metadata":{"id":"ztdqTt3OIBPI","colab_type":"code","outputId":"8991ed4a-104d-4ed5-f2bf-19a654fff0bb","executionInfo":{"status":"ok","timestamp":1584688418157,"user_tz":-540,"elapsed":869,"user":{"displayName":"김성환","photoUrl":"","userId":"17497395371430681608"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["print(torch.cuda.device_count())  # GPU deviec count check"],"execution_count":0,"outputs":[{"output_type":"stream","text":["1\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"G20dHg4mn5x4","colab_type":"text"},"source":["### 2.2. koGPT-2 Config"]},{"cell_type":"code","metadata":{"id":"rPoFzMKkk8eB","colab_type":"code","colab":{}},"source":["ctx= 'cuda'#'cuda' #'cpu' #학습 Device CPU or GPU. colab의 경우 GPU 사용\n","cachedir='~/kogpt2/' # KoGPT-2 모델 다운로드 경로\n","epoch =200  # 학습 epoch\n","save_path = 'drive/My Drive/Colab Notebooks/NarrativeKoGPT2/checkpoint/'\n","#use_cuda = True # Colab내 GPU 사용을 위한 값\n","\n","pytorch_kogpt2 = {\n","    'url':\n","    'https://kobert.blob.core.windows.net/models/kogpt2/pytorch/pytorch_kogpt2_676e9bcfa7.params',\n","    'fname': 'pytorch_kogpt2_676e9bcfa7.params',\n","    'chksum': '676e9bcfa7'\n","}\n","kogpt2_config = {\n","    \"initializer_range\": 0.02,\n","    \"layer_norm_epsilon\": 1e-05,\n","    \"n_ctx\": 1024,\n","    \"n_embd\": 768,\n","    \"n_head\": 12,\n","    \"n_layer\": 12,\n","    \"n_positions\": 1024,\n","    \"vocab_size\": 50000\n","}"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-xIXykCtn45d","colab_type":"text"},"source":["### 2.3 Model and Vocab Download"]},{"cell_type":"code","metadata":{"id":"kvtLJGh5o0MZ","colab_type":"code","outputId":"5ee66192-e5df-47c9-b62f-366da32cca71","executionInfo":{"status":"ok","timestamp":1584698905026,"user_tz":-540,"elapsed":57506,"user":{"displayName":"김성환","photoUrl":"","userId":"17497395371430681608"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# download model\n","model_info = pytorch_kogpt2\n","model_path = download(model_info['url'],\n","                       model_info['fname'],\n","                       model_info['chksum'],\n","                       cachedir=cachedir)\n","# download vocab\n","vocab_info = tokenizer\n","vocab_path = download(vocab_info['url'],\n","                       vocab_info['fname'],\n","                       vocab_info['chksum'],\n","                       cachedir=cachedir)"],"execution_count":7,"outputs":[{"output_type":"stream","text":["[██████████████████████████████████████████████████]\n","[██████████████████████████████████████████████████]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"fu7-2csBpLQR","colab_type":"text"},"source":["### 2.4.KoGPT-2 Model Vocab"]},{"cell_type":"code","metadata":{"id":"z5AK_S6spqwQ","colab_type":"code","colab":{}},"source":["# KoGPT-2 언어 모델 학습을 위한 GPT2LMHeadModel 선언\n","kogpt2model = GPT2LMHeadModel(config=GPT2Config.from_dict(kogpt2_config))\n","# model_path로부터 다운로드 받은 내용을 load_state_dict으로 업로드\n","kogpt2model.load_state_dict(torch.load(model_path))\n","\n","device = torch.device(ctx)\n","kogpt2model.to(device)\n","\n","# kogpt2model.eval()\n","# 추가로 학습하기 위해 .train() 사용\n","kogpt2model.train()\n","vocab_b_obj = gluonnlp.vocab.BERTVocab.from_sentencepiece(vocab_path,\n","                                                     mask_token=None,\n","                                                     sep_token=None,\n","                                                     cls_token=None,\n","                                                     unknown_token='<unk>',\n","                                                     padding_token='<pad>',\n","                                                     bos_token='<s>',\n","                                                     eos_token='</s>')\n"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"9rnV010Wq9Xw","colab_type":"text"},"source":["### 2.5. Get Batch Data using DataLoader"]},{"cell_type":"code","metadata":{"id":"Ukfj9FPHpwfk","colab_type":"code","outputId":"97d001d4-5220-4f26-fd12-c794b67c2510","executionInfo":{"status":"ok","timestamp":1584699184992,"user_tz":-540,"elapsed":2925,"user":{"displayName":"김성환","photoUrl":"","userId":"17497395371430681608"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["tok_path = get_tokenizer()\n","model, vocab = kogpt2model, vocab_b_obj\n","sentencepieceTokenizer = SentencepieceTokenizer(tok_path)\n","\n","#os.chdir(\"../\")\n","data_file_path = 'drive/My Drive/Colab Notebooks/NarrativeKoGPT2/data/backmyo_novel_1/untokenized_bm_data.txt'\n","batch_size = 2\n","novel_dataset = NovelDataset(data_file_path, vocab,sentencepieceTokenizer)\n","novel_data_loader = DataLoader(novel_dataset, batch_size=batch_size, shuffle=True, pin_memory=True)"],"execution_count":9,"outputs":[{"output_type":"stream","text":["using cached model\n","(905,)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"_RFndCOIrLS0","colab_type":"text"},"source":["### 2.6. Learning rate, Loss function, Adam Optimizer"]},{"cell_type":"code","metadata":{"id":"2pY_o_C-qBhz","colab_type":"code","colab":{}},"source":["learning_rate = 1e-5\n","criterion = torch.nn.CrossEntropyLoss()\n","optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"JMadZrwzXjbM","colab_type":"code","colab":{}},"source":["import subprocess\n","\n","def get_gpu_memory_map():\n","    \"\"\"Get the current gpu usage.\n","\n","    Returns\n","    -------\n","    usage: dict\n","        Keys are device ids as integers.\n","        Values are memory usage as integers in MB.\n","    \"\"\"\n","    result = subprocess.check_output(\n","        [\n","            'nvidia-smi', '--query-gpu=memory.used',\n","            '--format=csv,nounits,noheader'\n","        ], encoding='utf-8')\n","    # Convert lines into a dictionary\n","    gpu_memory = [int(x) for x in result.strip().split('\\n')]\n","    gpu_memory_map = dict(zip(range(len(gpu_memory)), gpu_memory))\n","    return gpu_memory_map"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Sgd28DRhthzo","colab_type":"text"},"source":["### 2.7. KoGPT-2 Transfer Laerning"]},{"cell_type":"code","metadata":{"id":"pbKCkcY63Y4a","colab_type":"code","outputId":"f81332b5-4469-42e6-cb1c-da0324161bde","executionInfo":{"status":"ok","timestamp":1584690255916,"user_tz":-540,"elapsed":1315,"user":{"displayName":"김성환","photoUrl":"","userId":"17497395371430681608"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["torch.cuda.\n"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["11330912256"]},"metadata":{"tags":[]},"execution_count":14}]},{"cell_type":"code","metadata":{"id":"yYkDU-cbrduY","colab_type":"code","outputId":"ce32766a-431f-45fc-e784-518e670e7a0f","executionInfo":{"status":"error","timestamp":1584759691454,"user_tz":-540,"elapsed":1462,"user":{"displayName":"김성환","photoUrl":"","userId":"17497395371430681608"}},"colab":{"base_uri":"https://localhost:8080/","height":269}},"source":["print('KoGPT-2 Transfer Learning Start')\n","epoch=200\n","for epoch in range(epoch):\n","  count = 0\n","  for data in novel_data_loader:\n","    optimizer.zero_grad()\n","    data = torch.stack(data) # list of Tensor로 구성되어 있기 때문에 list를 stack을 통해 변환해준다.\n","\n","    data= data.transpose(1,0)\n","    data= data.to(ctx)\n","    \n","    outputs = model(data, labels=data)\n","    loss, logits = outputs[:2]\n","    loss.backward()\n","    optimizer.step()\n","    if count %10 ==0:\n","      print('epoch no.{} train no.{}  loss = {}' . format(epoch, count+1, loss))\n","      # torch.save(model,save_path+'checkpoint_{}_{}.tar'.format(epoch,count))\n","      # 추론 및 학습 재개를 위한 일반 체크포인트 저장하기\n","    if (count >0 and count%100==0) or (len(data) < batch_size):\n","      torch.save({\n","        'epoch': epoch,\n","        'train_no': count,\n","        'model_state_dict': model.state_dict(),\n","        'optimizer_state_dict': optimizer.state_dict(),\n","        'loss':loss\n","      }, save_path+'narrativeKoGPT2_checkpoint.tar')\n","\n","    count += 1"],"execution_count":1,"outputs":[{"output_type":"stream","text":["KoGPT-2 Transfer Learning Start\n"],"name":"stdout"},{"output_type":"error","ename":"NameError","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)","\u001b[0;32m<ipython-input-1-c2a316037805>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mepoch\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mepoch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m   \u001b[0mcount\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m   \u001b[0;32mfor\u001b[0m \u001b[0mdata\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mnovel_data_loader\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m     \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m     \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstack\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# list of Tensor로 구성되어 있기 때문에 list를 stack을 통해 변환해준다.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mNameError\u001b[0m: name 'novel_data_loader' is not defined"]}]},{"cell_type":"code","metadata":{"id":"by1zYlUWudTX","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]}]}