RNN&LSTM/RNN&LSTM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RNN\n",
    "**RNN** 은 딥러닝에서 사용하는 Recurrent Neural Network으로서 시계열 데이터를 다룰때 사용하는 신경망 이다.  \n",
    "**대표적인 예**와 그에 **해당하는 이론에 대한 내용**은 아래 링크를 참조하자.  \n",
    "\n",
    "1. <a href=\"https://wjddyd66.github.io/dl/2019/09/05/%EC%9E%90%EC%97%B0%EC%96%B4%EC%99%80-%EB%8B%A8%EC%96%B4%EC%9D%98-%EB%B6%84%EC%82%B0-%ED%91%9C%ED%98%84.html\">자연어와 단어의 분산 표현</a>\n",
    "2. <a href=\"https://wjddyd66.github.io/dl/2019/09/05/word2vec.html\">word2vec</a>\n",
    "3. <a href=\"https://wjddyd66.github.io/dl/2019/09/06/Fast-word2vec.html\">Fast word2vec</a>\n",
    "4. <a href=\"https://wjddyd66.github.io/dl/2019/09/09/RNN.html\">RNN</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 필요한 라이브러리 import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VwV7Yzdz4Lla"
   },
   "outputs": [],
   "source": [
    "# 단순한 문자 RNN을 만들어보겠습니다.\n",
    "\n",
    "import torch \n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### HyperParameter 설정\n",
    "- n_hidden: 신경망 Node수\n",
    "- lr: Learning Rate\n",
    "- epochs: 반복 횟수"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jcjSLaS337wU"
   },
   "outputs": [],
   "source": [
    "# 하이퍼파라미터 설정\n",
    "\n",
    "n_hidden = 35 \n",
    "lr = 0.01\n",
    "epochs = 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Input Data생성 및 Vocab 설정\n",
    "- InputData: string\n",
    "- Vocab: chars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HE0ZIXVp37wJ"
   },
   "outputs": [],
   "source": [
    "# 사용하는 문자는 영어 소문자 및 몇가지 특수문자로 제한했습니다.\n",
    "# alphabet(0-25), space(26), ... , start(0), end(1)\n",
    "\n",
    "string = \"hello pytorch. how long can a rnn cell remember? show me your limit!\"\n",
    "chars =  \"abcdefghijklmnopqrstuvwxyz ?!.,:;01\"\n",
    "\n",
    "# 문자들을 리스트로 바꾸고 이의 길이(=문자의 개수)를 저장해놓습니다.\n",
    "char_list = [i for i in chars]\n",
    "n_letters = len(char_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DhYTZTsuGygL"
   },
   "source": [
    "#### Encoder\n",
    "\n",
    "문자를 그대로 쓰지않고 one-hot 벡터로 바꿔서 연산에 사용\n",
    "\n",
    "Start = [0 0 0 … 1 0]\n",
    "\n",
    "a =     [1 0 0 … 0 0]\n",
    "\n",
    "b =     [0 1 0 … 0 0]\n",
    "\n",
    "c =     [0 0 1 … 0 0]\n",
    "\n",
    "...\n",
    "\n",
    "end =   [0 0 0 … 0 1]\n",
    "\n",
    "\n",
    "최종적인 Output은 start + 문장 + end로서 구성된다.\n",
    "\n",
    "One-hot Vector는 Encoder에서 Input String을 변환해서 RNN Model의 Input으로 들어갈 수 있게 One-hot Vecotr로 변환하는 역할을 한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "R0oGDc_o37wN"
   },
   "outputs": [],
   "source": [
    "# 문자열을 one-hot 벡터의 스택으로 만드는 함수\n",
    "# abc -> [[1 0 0 … 0 0],\n",
    "#         [0 1 0 … 0 0],\n",
    "#         [0 0 1 … 0 0]]\n",
    "\n",
    "def string_to_onehot(string):\n",
    "    # 먼저 시작 토큰과 끝 토큰을 만들어줍니다.\n",
    "    start = np.zeros(shape=n_letters ,dtype=int)\n",
    "    end = np.zeros(shape=n_letters ,dtype=int)\n",
    "    start[-2] = 1\n",
    "    end[-1] = 1\n",
    "    # 여기서부터는 문자열의 문자들을 차례대로 받아서 진행합니다.\n",
    "    for i in string:\n",
    "        # 먼저 문자가 몇번째 문자인지 찾습니다.\n",
    "        # a:0, b:1, c:2,...\n",
    "        idx = char_list.index(i)\n",
    "        # 0으로만 구성된 배열을 만들어줍니다.\n",
    "        # [0 0 0 … 0 0]\n",
    "        zero = np.zeros(shape=n_letters ,dtype=int)\n",
    "        # 해당 문자 인데스만 1로 바꿔줍니다.\n",
    "        # b: [0 1 0 … 0 0]\n",
    "        zero[idx]=1\n",
    "        # start와 새로 생긴 zero를 붙이고 이를 start에 할당합니다.\n",
    "        # 이게 반복되면 start에는 문자를 one-hot 벡터로 바꾼 배열들이 점점 쌓여가게 됩니다.\n",
    "        start = np.vstack([start,zero])\n",
    "    # 문자열이 다 끝나면 쌓아온 start와 end를 붙여줍니다.\n",
    "    output = np.vstack([start,end])\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Decoder\n",
    "\n",
    "onehot_to_word는 Decoder에서 RNN Model의 output을 Output String 으로 바꿔주는 역할을 한다.\n",
    "\n",
    "이러한 변환을 통하여 Model의 Output과 실제 Target과의 차이를 구할 수 있다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UNSZj7Tv37wQ"
   },
   "outputs": [],
   "source": [
    "# One-hot 벡터를 문자로 바꿔주는 함수 \n",
    "# [1 0 0 ... 0 0] -> a \n",
    "# https://pytorch.org/docs/stable/tensors.html?highlight=numpy#torch.Tensor.numpy\n",
    "\n",
    "def onehot_to_word(onehot_1):\n",
    "    # 텐서를 입력으로 받아 넘파이 배열로 바꿔줍니다.\n",
    "    onehot = torch.Tensor.numpy(onehot_1)\n",
    "    # one-hot 벡터의 최대값(=1) 위치 인덱스로 문자를 찾습니다.\n",
    "    return char_list[onehot.argmax()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### RNN Model\n",
    "실직적인 RNN Model을 구성하는 Code이다.\n",
    "\n",
    "기존 Neural Network와 달리 RNN에서 중요한 점을 살펴보면 다음과 같다.\n",
    "- <code>combined = torch.cat((input, hidden), 1)</code>: Input으로 들어가는 것은 기존의 Input 뿐만이 아니라 이전 Hidden Layer의 결과또한 들어가게 된다.\n",
    "- <code>return output, hidden</code>: Input으로서 HiddenLayer의 값또한 들어가게 되므로 Return값이 output과 HiddenLayer의 output 2가지 이다.\n",
    "- <code>init_hidden(self)</code>: 처음 HiddenLayer의 output값이 없으므로 초기화를 시켜주는 작업이 필요하다.\n",
    "- ActivationFunction: ReLu\n",
    "\n",
    "**참고사항: troch.cat Parameter**  \n",
    "<code>torch.cat(tensors, dim=0, out=None)</code>\n",
    "- tensors: Input Tensor\n",
    "- dim: Tensor의 차원을 합친뒤 Output으로 내보낼 Tensor의 차원\n",
    "- out: return 값은 Tensor로서 특정 Tensor에 대입하고할 때 사용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HMI_b1MO37wR"
   },
   "outputs": [],
   "source": [
    "# RNN with 1 hidden layer\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(RNN, self).__init__()\n",
    "        \n",
    "        self.input_size = input_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.output_size = output_size\n",
    "        \n",
    "        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)\n",
    "        self.i2o = nn.Linear(input_size + hidden_size, output_size)\n",
    "        self.act_fn = nn.Tanh()\n",
    "    \n",
    "    def forward(self, input, hidden):\n",
    "        # 입력과 hidden state를 cat함수로 붙여줍니다.\n",
    "        combined = torch.cat((input, hidden), 1)\n",
    "        # 붙인 값을 i2h 및 i2o에 통과시켜 hidden state는 업데이트, 결과값은 계산해줍니다.\n",
    "        hidden = self.act_fn(self.i2h(combined))\n",
    "        output = self.i2o(combined)\n",
    "        return output, hidden\n",
    "    \n",
    "    # 아직 입력이 없을때(t=0)의 hidden state를 초기화해줍니다. \n",
    "    def init_hidden(self):\n",
    "        return torch.zeros(1, self.hidden_size)\n",
    "    \n",
    "rnn = RNN(n_letters, n_hidden, n_letters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loss Function & Optimizer Define\n",
    "- LossFunction: MSE\n",
    "- Optimizer: Adam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_func = nn.MSELoss()\n",
    "optimizer = torch.optim.Adam(rnn.parameters(), lr=lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train\n",
    "앞에서 선언한 Model을 Train하는 과정이다.\n",
    "**Char_RNN을 목표로하고 있으므로 Target Data는 Input Data에서 Index가 하나씩 추가되어진 형태**이다.\n",
    "\n",
    "RNN의 Input으로 들어가기 위하여 문자열을 One-Hot-Vector로서 변환하는 과정이 필요하고 Model에 들어가기 위해서는 Tensor형태이여야 하므로 type을 <code>torch.FloadTensor()</code>로서 정의한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1717
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 20539,
     "status": "ok",
     "timestamp": 1559517557821,
     "user": {
      "displayName": "Choi Gunho",
      "photoUrl": "",
      "userId": "04388737836176863066"
     },
     "user_tz": -540
    },
    "id": "d9x44BXX37wZ",
    "outputId": "b2643cae-1f35-4c1a-8c82-bdaf2abb4acf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(2.4497, grad_fn=<AddBackward0>)\n",
      "tensor(1.0887, grad_fn=<AddBackward0>)\n",
      "tensor(0.6670, grad_fn=<AddBackward0>)\n",
      "tensor(0.4221, grad_fn=<AddBackward0>)\n",
      "tensor(0.2892, grad_fn=<AddBackward0>)\n",
      "tensor(0.2065, grad_fn=<AddBackward0>)\n",
      "tensor(0.1621, grad_fn=<AddBackward0>)\n",
      "tensor(0.1233, grad_fn=<AddBackward0>)\n",
      "tensor(0.0983, grad_fn=<AddBackward0>)\n",
      "tensor(0.1053, grad_fn=<AddBackward0>)\n",
      "tensor(0.0735, grad_fn=<AddBackward0>)\n",
      "tensor(0.0627, grad_fn=<AddBackward0>)\n",
      "tensor(0.0537, grad_fn=<AddBackward0>)\n",
      "tensor(0.0465, grad_fn=<AddBackward0>)\n",
      "tensor(0.0453, grad_fn=<AddBackward0>)\n",
      "tensor(0.0383, grad_fn=<AddBackward0>)\n",
      "tensor(0.0332, grad_fn=<AddBackward0>)\n",
      "tensor(0.0292, grad_fn=<AddBackward0>)\n",
      "tensor(0.0320, grad_fn=<AddBackward0>)\n",
      "tensor(0.0268, grad_fn=<AddBackward0>)\n",
      "tensor(0.0234, grad_fn=<AddBackward0>)\n",
      "tensor(0.0215, grad_fn=<AddBackward0>)\n",
      "tensor(0.0214, grad_fn=<AddBackward0>)\n",
      "tensor(0.0188, grad_fn=<AddBackward0>)\n",
      "tensor(0.0190, grad_fn=<AddBackward0>)\n",
      "tensor(0.0210, grad_fn=<AddBackward0>)\n",
      "tensor(0.0174, grad_fn=<AddBackward0>)\n",
      "tensor(0.0156, grad_fn=<AddBackward0>)\n",
      "tensor(0.0160, grad_fn=<AddBackward0>)\n",
      "tensor(0.0143, grad_fn=<AddBackward0>)\n",
      "tensor(0.0130, grad_fn=<AddBackward0>)\n",
      "tensor(0.0161, grad_fn=<AddBackward0>)\n",
      "tensor(0.0118, grad_fn=<AddBackward0>)\n",
      "tensor(0.0115, grad_fn=<AddBackward0>)\n",
      "tensor(0.0105, grad_fn=<AddBackward0>)\n",
      "tensor(0.0195, grad_fn=<AddBackward0>)\n",
      "tensor(0.0127, grad_fn=<AddBackward0>)\n",
      "tensor(0.0107, grad_fn=<AddBackward0>)\n",
      "tensor(0.0093, grad_fn=<AddBackward0>)\n",
      "tensor(0.0086, grad_fn=<AddBackward0>)\n",
      "tensor(0.0080, grad_fn=<AddBackward0>)\n",
      "tensor(0.0257, grad_fn=<AddBackward0>)\n",
      "tensor(0.0161, grad_fn=<AddBackward0>)\n",
      "tensor(0.0091, grad_fn=<AddBackward0>)\n",
      "tensor(0.0081, grad_fn=<AddBackward0>)\n",
      "tensor(0.0072, grad_fn=<AddBackward0>)\n",
      "tensor(0.0067, grad_fn=<AddBackward0>)\n",
      "tensor(0.0063, grad_fn=<AddBackward0>)\n",
      "tensor(0.0060, grad_fn=<AddBackward0>)\n",
      "tensor(0.0125, grad_fn=<AddBackward0>)\n",
      "tensor(0.0081, grad_fn=<AddBackward0>)\n",
      "tensor(0.0065, grad_fn=<AddBackward0>)\n",
      "tensor(0.0055, grad_fn=<AddBackward0>)\n",
      "tensor(0.0051, grad_fn=<AddBackward0>)\n",
      "tensor(0.0151, grad_fn=<AddBackward0>)\n",
      "tensor(0.0059, grad_fn=<AddBackward0>)\n",
      "tensor(0.0050, grad_fn=<AddBackward0>)\n",
      "tensor(0.0045, grad_fn=<AddBackward0>)\n",
      "tensor(0.0043, grad_fn=<AddBackward0>)\n",
      "tensor(0.0040, grad_fn=<AddBackward0>)\n",
      "tensor(0.0039, grad_fn=<AddBackward0>)\n",
      "tensor(0.0096, grad_fn=<AddBackward0>)\n",
      "tensor(0.0073, grad_fn=<AddBackward0>)\n",
      "tensor(0.0044, grad_fn=<AddBackward0>)\n",
      "tensor(0.0038, grad_fn=<AddBackward0>)\n",
      "tensor(0.0035, grad_fn=<AddBackward0>)\n",
      "tensor(0.0033, grad_fn=<AddBackward0>)\n",
      "tensor(0.0031, grad_fn=<AddBackward0>)\n",
      "tensor(0.0039, grad_fn=<AddBackward0>)\n",
      "tensor(0.0118, grad_fn=<AddBackward0>)\n",
      "tensor(0.0054, grad_fn=<AddBackward0>)\n",
      "tensor(0.0037, grad_fn=<AddBackward0>)\n",
      "tensor(0.0032, grad_fn=<AddBackward0>)\n",
      "tensor(0.0029, grad_fn=<AddBackward0>)\n",
      "tensor(0.0027, grad_fn=<AddBackward0>)\n",
      "tensor(0.0026, grad_fn=<AddBackward0>)\n",
      "tensor(0.0026, grad_fn=<AddBackward0>)\n",
      "tensor(0.0109, grad_fn=<AddBackward0>)\n",
      "tensor(0.0050, grad_fn=<AddBackward0>)\n",
      "tensor(0.0027, grad_fn=<AddBackward0>)\n",
      "tensor(0.0025, grad_fn=<AddBackward0>)\n",
      "tensor(0.0023, grad_fn=<AddBackward0>)\n",
      "tensor(0.0022, grad_fn=<AddBackward0>)\n",
      "tensor(0.0112, grad_fn=<AddBackward0>)\n",
      "tensor(0.0053, grad_fn=<AddBackward0>)\n",
      "tensor(0.0027, grad_fn=<AddBackward0>)\n",
      "tensor(0.0021, grad_fn=<AddBackward0>)\n",
      "tensor(0.0020, grad_fn=<AddBackward0>)\n",
      "tensor(0.0019, grad_fn=<AddBackward0>)\n",
      "tensor(0.0018, grad_fn=<AddBackward0>)\n",
      "tensor(0.0033, grad_fn=<AddBackward0>)\n",
      "tensor(0.0025, grad_fn=<AddBackward0>)\n",
      "tensor(0.0022, grad_fn=<AddBackward0>)\n",
      "tensor(0.0018, grad_fn=<AddBackward0>)\n",
      "tensor(0.0016, grad_fn=<AddBackward0>)\n",
      "tensor(0.0016, grad_fn=<AddBackward0>)\n",
      "tensor(0.0099, grad_fn=<AddBackward0>)\n",
      "tensor(0.0032, grad_fn=<AddBackward0>)\n",
      "tensor(0.0021, grad_fn=<AddBackward0>)\n",
      "tensor(0.0016, grad_fn=<AddBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# train\n",
    "\n",
    "# 문자열을 onehot 벡터로 만들고 이를 토치 텐서로 바꿔줍니다.\n",
    "# 또한 데이터타입도 학습에 맞게 바꿔줍니다.\n",
    "one_hot = torch.from_numpy(string_to_onehot(string)).type_as(torch.FloatTensor())\n",
    "\n",
    "for i in range(epochs):\n",
    "    optimizer.zero_grad()\n",
    "    # 학습에 앞서 hidden state를 초기화해줍니다.\n",
    "    hidden = rnn.init_hidden()\n",
    "    \n",
    "    # 문자열 전체에 대한 손실을 구하기 위해 total_loss라는 변수를 만들어줍니다. \n",
    "    total_loss = 0\n",
    "    for j in range(one_hot.size()[0]-1):\n",
    "        # 입력은 앞에 글자 \n",
    "        # pyotrch 에서 p y t o r c\n",
    "        input_ = one_hot[j:j+1,:]\n",
    "        # 목표값은 뒤에 글자\n",
    "        # pytorch 에서 y t o r c h\n",
    "        target = one_hot[j+1]\n",
    "        output, hidden = rnn.forward(input_, hidden)\n",
    "        \n",
    "        loss = loss_func(output.view(-1),target.view(-1))\n",
    "        total_loss += loss\n",
    "\n",
    "    total_loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    if i % 10 == 0:\n",
    "        print(total_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test\n",
    "Train된 Model을 확인한다.\n",
    "start는 맨처음 선언한 One-hot-Vecotr의 첫번째로서 선언하게 된다.\n",
    "Output을 확인하기 위하여 One-hot-Vecotr => String으로 바꾸어서 확인한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 20534,
     "status": "ok",
     "timestamp": 1559517557823,
     "user": {
      "displayName": "Choi Gunho",
      "photoUrl": "",
      "userId": "04388737836176863066"
     },
     "user_tz": -540
    },
    "id": "O8suWP0r37wf",
    "outputId": "891bbd66-520f-4434-f502-d91aee67ce0a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello pytorch. h yrlrememno crnmeonmelrn ng cn noe ieiemyoe ynmlonon\n"
     ]
    }
   ],
   "source": [
    "# test \n",
    "# hidden state 는 처음 한번만 초기화해줍니다.\n",
    "\n",
    "start = torch.zeros(1,n_letters)\n",
    "start[:,-2] = 1\n",
    "\n",
    "with torch.no_grad():\n",
    "    hidden = rnn.init_hidden()\n",
    "    # 처음 입력으로 start token을 전달해줍니다.\n",
    "    input_ = start\n",
    "    # output string에 문자들을 계속 붙여줍니다.\n",
    "    output_string = \"\"\n",
    "\n",
    "    # 원래는 end token이 나올때 까지 반복하는게 맞으나 끝나지 않아서 string의 길이로 정했습니다.\n",
    "    for i in range(len(string)):\n",
    "        output, hidden = rnn.forward(input_, hidden)\n",
    "        # 결과값을 문자로 바꿔서 output_string에 붙여줍니다.\n",
    "        output_string += onehot_to_word(output.data)\n",
    "        # 또한 이번의 결과값이 다음의 입력값이 됩니다.\n",
    "        input_ = output\n",
    "\n",
    "print(output_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batch 처리(RNN, LSTM)\n",
    "\n",
    "RNN과 LSTM의 성능차이를 확인하기 위하여 Batch처리를 하여 **Character Recurrent Neural Network**를 2가지의 Model로서 구성하고 확인해본다.\n",
    "\n",
    "최종적인 Model은 세익스피어 문체를 모방하는 Model을 작성하는 것이 목표이다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "2가지의 Model에서 사용하기 위한 DataSet을 구축하는 단계이다.\n",
    "\n",
    "#### Data Download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-09-23 08:46:35--  https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/tinyshakespeare/input.txt\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.228.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.228.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1115394 (1.1M) [text/plain]\n",
      "Saving to: ‘./data/input.txt’\n",
      "\n",
      "input.txt           100%[===================>]   1.06M  4.34MB/s    in 0.2s    \n",
      "\n",
      "2019-09-23 08:46:35 (4.34 MB/s) - ‘./data/input.txt’ saved [1115394/1115394]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!rm -r data\n",
    "import os \n",
    "\n",
    "try:\n",
    "  os.mkdir(\"./data\")\n",
    "except:\n",
    "  pass\n",
    "\n",
    "!wget https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/tinyshakespeare/input.txt -P ./data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting unidecode\n",
      "  Using cached https://files.pythonhosted.org/packages/d0/42/d9edfed04228bacea2d824904cae367ee9efd05e6cce7ceaaedd0b0ad964/Unidecode-1.1.1-py2.py3-none-any.whl\n",
      "Installing collected packages: unidecode\n",
      "Successfully installed unidecode-1.1.1\n"
     ]
    }
   ],
   "source": [
    "!pip3 install unidecode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Settings\n",
    "\n",
    "**1) 필요한 라이브러리들을 불러온다**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import unidecode\n",
    "import string\n",
    "import random\n",
    "import re\n",
    "import time, math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2) HyperParameter설정**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_epochs = 2000\n",
    "print_every = 100\n",
    "plot_every = 10\n",
    "\n",
    "# chunk에 대한 설명은 아래 함수정의하면서 하겠습니다.\n",
    "chunk_len = 200\n",
    "\n",
    "hidden_size = 100\n",
    "batch_size = 1\n",
    "num_layers = 1\n",
    "embedding_size = 70\n",
    "lr = 0.002"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Data preprocessing\n",
    "\n",
    "**1) Prepare characters**  \n",
    "출력 가능한 모든 문자의 길이를 알아내는 작업이다.\n",
    "이것이 One-hot-vector로 단어를 변환할때의 Dimension의 크기가 된다.(Vocab size를 알아내는 것과 같다.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n",
      "\r",
      "\u000b",
      "\f",
      "\n",
      "num_chars =  100\n"
     ]
    }
   ],
   "source": [
    "# import 했던 string에서 출력가능한 문자들을 다 불러옵니다. \n",
    "all_characters = string.printable\n",
    "\n",
    "# 출력가능한 문자들의 개수를 저장해놓습니다.\n",
    "n_characters = len(all_characters)\n",
    "print(all_characters)\n",
    "print('num_chars = ', n_characters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2) Get text Data**  \n",
    "Data로서 사용할 File의 Character의 개수를 알아온다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file_len = 1115394\n"
     ]
    }
   ],
   "source": [
    "# 앞서 다운받은 텍스트 파일을 열어줍니다.\n",
    "\n",
    "file = unidecode.unidecode(open('./data/input.txt').read())\n",
    "file_len = len(file)\n",
    "print('file_len =', file_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3) Random chunk**  \n",
    "Train과 Test로서 사용할 Data를 일정한 Character의 개수로서 자르는 작업을 한다.  \n",
    "\n",
    "**rnadon.randint(start,end)**  \n",
    "start ~ end-1 사이의 Random한 숫자 출력  \n",
    "즉 Data의 Size를 고려하여서 end = file_len - chunk_len으로서 선언  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sir, ere long I'll visit you again.\n",
      "\n",
      "CLAUDIO:\n",
      "Most holy sir, I thank you.\n",
      "\n",
      "ISABELLA:\n",
      "My business is a word or two with Claudio.\n",
      "\n",
      "Provost:\n",
      "And very welcome. Look, signior, here's your sister.\n",
      "\n",
      "DUKE VINC\n"
     ]
    }
   ],
   "source": [
    "def random_chunk():\n",
    "    # (시작지점 < 텍스트파일 전체길이 - 불러오는 텍스트의 길이)가 되도록 시작점과 끝점을 정합니다.\n",
    "    start_index = random.randint(0, file_len - chunk_len)\n",
    "    end_index = start_index + chunk_len + 1\n",
    "    return file[start_index:end_index]\n",
    "\n",
    "print(random_chunk())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4) Character to tensor**  \n",
    "문자열을 받았을때 Tensor로서 바꿔주는 역할을 한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([36, 37, 38, 13, 14, 15])\n"
     ]
    }
   ],
   "source": [
    "# 문자열을 받았을때 이를 인덱스의 배열로 바꿔주는 함수입니다.\n",
    "def char_tensor(string):\n",
    "    tensor = torch.zeros(len(string)).long()\n",
    "    for c in range(len(string)):\n",
    "        tensor[c] = all_characters.index(string[c])\n",
    "    return tensor\n",
    "\n",
    "print(char_tensor('ABCdef'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5) Chunk into input & label**  \n",
    "Parameter로서 받은 문자열을 Input Data와 Target Data로서 분류하는 작업이다.  \n",
    "ex)\n",
    "- Parameter: pytorch\n",
    "- Input: pytorc\n",
    "- Target: ytorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_training_set():    \n",
    "    chunk = random_chunk()\n",
    "    inp = char_tensor(chunk[:-1])\n",
    "    target = char_tensor(chunk[1:])\n",
    "    return inp, target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아래의 Model에서는 하나의 Character를 입력받아 다음 Character를 출력하는 Model이다.  \n",
    "이러한 Model에 Chunk_size만큼 Character를 Input으로서 사용하겠다는 의미는 Batch처리를 하겠다는 의미와도 같다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RNN Model\n",
    "\n",
    "**1)Model 선언**  \n",
    "Model에 Input으로 들어가는 Value는 앞에서 Char_RNN에서 했던 One-hot-vector방식이 아닌 Embdeeing을 통하여 이루워 진다.  \n",
    "\n",
    "<code>torch.nn.Embedding(num_embeddings, embedding_dim, padding_idx=None, max_norm=None, norm_type=2.0, scale_grad_by_freq=False, sparse=False, _weight=None)</code>\n",
    "\n",
    "- num_embedding: Embedding 의 Size(Vocab size)\n",
    "- embedding_dim: 각각의 Embedding Vector의 Size\n",
    "- padding_idx: Output의 Vector의 크기를 맞추기 위하여 사용\n",
    "- max_norm: Embedding Vector안의 요소가 일정크기 이상이면 새롭게 Normalization을 실시\n",
    "- norm_type: p_norm or max_norm 을하여 모든 요소의 값을 0 ~ 1값으로 바꾸기 위하여 사용\n",
    "\n",
    "<code>torch.nn.RNN(*args, **kwargs)</code>  \n",
    "$h_t = tanh(W_{ih}x_t + b_{ih} + W_{hh}h_{t+1} + b_{hh})$\n",
    "- input_size: Input Size\n",
    "- output_size: Output Size\n",
    "- num_layers: Number of layers\n",
    "- nonlinearity: tanh or relu(Default: tanh)\n",
    "- bias: True or False\n",
    "- batch_first: 만약 True로 설정하면 Input Data = (batch, seq, feature)로서 설정해야 한다.\n",
    "- dropout: Dropout설정\n",
    "- bidirectrional: 양방향 RNN으로 설정하지(Default: False)\n",
    "\n",
    "참조: <a href=\"https://pytorch.org/docs/stable/nn.html\">Pytorch 정식 사이트</a>\n",
    "\n",
    "또한 위에서 Chunk_size로서 Batch_size를 정하고 Batch_size만큼 Input이 들어오게 된다.\n",
    "따라서 Encoder와 Decoder에 들어오는 Data의 Size를 -1을 통하여 각각의 Size에 맞게 변환 후 사용해야 한다.\n",
    "\n",
    "- input.view(1,-1): Input은 Character하나이므로 (1,-1)로서 Shape 변환\n",
    "- out.view(batch_size,-1): Output은 들어온 Batch_Size만큼 변환하여 결과 수행"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RNN(nn.Module):\n",
    "    def __init__(self, input_size, embedding_size, hidden_size, output_size, num_layers=1):\n",
    "        super(RNN, self).__init__()\n",
    "        self.input_size = input_size\n",
    "        self.embedding_size = embedding_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.output_size = output_size\n",
    "        self.num_layers = num_layers\n",
    "        \n",
    "        self.encoder = nn.Embedding(self.input_size, self.embedding_size)\n",
    "        self.rnn = nn.RNN(self.embedding_size,self.hidden_size,self.num_layers)\n",
    "        self.decoder = nn.Linear(self.hidden_size, self.output_size)\n",
    "        \n",
    "    \n",
    "    def forward(self, input, hidden):\n",
    "        out = self.encoder(input.view(1,-1))\n",
    "        out,hidden = self.rnn(out,hidden)\n",
    "        out = self.decoder(out.view(batch_size,-1))\n",
    "        return out,hidden\n",
    "\n",
    "    def init_hidden(self):\n",
    "        hidden = torch.zeros(self.num_layers, batch_size, self.hidden_size)\n",
    "        return hidden\n",
    "    \n",
    "model = RNN(n_characters, embedding_size, hidden_size, n_characters, num_layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2) Model에 미리 선언한 Parameter 대입**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = RNN(input_size=n_characters, \n",
    "            embedding_size=embedding_size,\n",
    "            hidden_size=hidden_size, \n",
    "            output_size=n_characters, \n",
    "            num_layers=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3) Define Loss Function & Optimizer**  \n",
    "- LossFunction: CrossEntropy\n",
    "- Optimizer: Adam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
    "loss_func = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4) Test Function**  \n",
    "중간중간 결과 확인을 위한 Function  \n",
    "아래 두가지의 Parameter를 살펴보게 되면 2가지의 과정을 거치게 된다.  \n",
    "최종적인 목적을 말하게 되면 항상 같은 문장이 생성되는 것이 아닌 매번 달라지는 문장을 생성하는 것이 목표이다.  \n",
    "이러한 확률적인 방법은 각각의 단어에 나올 확률을 기반으로 이루워진다.  \n",
    "\n",
    "**<code>output_dist = output.data.view(-1).div(0.8).exp()</code>**  \n",
    "위의 Code는 확률적인 문장 생성을 위한 Code이다.  \n",
    "0.8로서 나눔으로 인하여 문장의 확률을 조금 Normalization하는 효과가 있다.  \n",
    "\n",
    "**<code>torch.multinomial(output_dist, 1)[0]</code>**  \n",
    "위의 결과로 얻어진 확률분포를 통하여 확률적으로 단어를 선택하게 하는 Code이다.  \n",
    "EX) out_dist = [0.2, 0.8]이면 첫번째 단어가 나올확률은 20%, 두번째 단어가 나올 확률은 80%로서 단어를 선택하는 방법이다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 임의의 문자(start_str)로 시작하는 길이 200짜리 모방 글을 생성하는 코드입니다.\n",
    "def test():\n",
    "    start_str = \"b\"\n",
    "    inp = char_tensor(start_str)\n",
    "    hidden = model.init_hidden()\n",
    "    x = inp\n",
    "\n",
    "    print(start_str,end=\"\")\n",
    "    for i in range(200):\n",
    "        output,hidden = model(x,hidden)\n",
    "\n",
    "        output_dist = output.data.view(-1).div(0.8).exp()\n",
    "        top_i = torch.multinomial(output_dist, 1)[0]\n",
    "        predicted_char = all_characters[top_i]\n",
    "\n",
    "        print(predicted_char,end=\"\")\n",
    "\n",
    "        x = char_tensor(predicted_char)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5) Train**\n",
    "\n",
    "Mini_Batch처리를 하고있으므로 실질적인 Loss는 Batch_Size인 Chunk_len으로서 나눈 것 이다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " tensor([4.5817], grad_fn=<DivBackward0>) \n",
      "\n",
      "*bSl 8RX{dj3\\b['*jb3@<'Dhj6)5Uw4)mebp!A9*kZLeRiE_b0I.J8ot4 tPLg(hxH2*GwrmQph\\?kNiYU_ >g\f",
      "G(wjFyV@Vg&X.z|%\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([2.4368], grad_fn=<DivBackward0>) \n",
      "\n",
      "bund ind ye fut yit giou the bLome to ut:\n",
      "What kon se the wall sand\n",
      "\n",
      "There the wandnn wathe that aim, ins mamne som sada\n",
      "Thoul un atunct is pcisth yoam ke thin, gon sentetr tort youc torand ared hos, i\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([2.1629], grad_fn=<DivBackward0>) \n",
      "\n",
      "be mare ala\n",
      "e sues is i- lolh tis presighill, and\n",
      "An lorn fingtmis gRaat yome\n",
      "I has so souns,\n",
      "Buti/ore came s;our thelicen hormere fount,\n",
      "Pice me, hash.z\n",
      "Ly yourt wace insttathice toith mecith aed,\n",
      "Nf \n",
      " ====================================================================================================\n",
      "\n",
      " tensor([2.1459], grad_fn=<DivBackward0>) \n",
      "\n",
      "by thee for coupls:\n",
      "Whis to the not theat sind all this to fell thou tid with thene wace hat roth to duncentin, ullave at willelns,\n",
      "I heat in yeus uprits be dore Tham andes, andees resh that thil:\n",
      "To h\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([2.0884], grad_fn=<DivBackward0>) \n",
      "\n",
      "ble your stakever will shit mine but couked saltee sirsoncines?\n",
      "Bo wlit, the neal the may weaked pate some,\n",
      "The det his kord tay theat siy wike make irst he have is wath we. oe tefs my be ale sir:\n",
      "Are \n",
      " ====================================================================================================\n",
      "\n",
      " tensor([1.9862], grad_fn=<DivBackward0>) \n",
      "\n",
      "bencewers stimn, muhe,\n",
      "Good wome.\n",
      "\n",
      "CUOMES:\n",
      "O noth well sping caincls you.\n",
      "\n",
      "JARLANIS:\n",
      "What it the the haw ale in thou krow me-thes to hell to thee thee,\n",
      "Oratrempens,\n",
      "Whithe',\n",
      "Troth my eidht\n",
      "Bisto the wh\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([1.8999], grad_fn=<DivBackward0>) \n",
      "\n",
      "blow come of high aflel of sing his shall shanger inder sonould sone mic are me hirs coslestred it 'ather, wible yiursbils\n",
      "Why his thir sell sale, whis sing conte lade: sine.\n",
      "\n",
      "PROMTH:\n",
      "\n",
      "VARES:\n",
      "Do, I sii\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([2.1515], grad_fn=<DivBackward0>) \n",
      "\n",
      "by of sur'd gage cheds, the swet ferver was surusanes meseraus tconton me rakfent-our penorg, the with in with on sharding her sangness conten greace; have real me sal and underan, the thinds sup, pran\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([2.0460], grad_fn=<DivBackward0>) \n",
      "\n",
      "bore the brow it,\n",
      "Cour!\n",
      "\n",
      "DUKE MIO:\n",
      "Rtall'd now reart. Whequry sone king of my darje, as of you lize the cofd to to be purberand for thee of this will my pant fent you and would you trow, sson the sgepi\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([1.9207], grad_fn=<DivBackward0>) \n",
      "\n",
      "bed best here saizers on my wlord woud, the land, lest.\n",
      "\n",
      "CKINGELLA:\n",
      "My het she his groings in the vour other not at chip fang other so with and they the Ke here of heare than the beart mady sour sinst \n",
      " ====================================================================================================\n",
      "\n",
      " tensor([2.3151], grad_fn=<DivBackward0>) \n",
      "\n",
      "bBegher, tade.\n",
      "\n",
      "BARGILO:\n",
      "Whose blled;\n",
      "Beauch entuid her if that: to lade lame you and art him hath sit for is your is seat enstik: but shame thou sateen I makister speate, full sir him salm cvoses we h\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([1.8078], grad_fn=<DivBackward0>) \n",
      "\n",
      "be now, the enect teess in the sher is uncent pless of hence your the sprey:\n",
      "And the and the grien sbut I menewind stay then's is, I nenfort the pooth thou where it they true were you with the the say \n",
      " ====================================================================================================\n",
      "\n",
      " tensor([1.7959], grad_fn=<DivBackward0>) \n",
      "\n",
      "bat lised hath tame her,\n",
      "Hit your\n",
      "a cillo me and dust wheself, duspead cingur.\n",
      "\n",
      "ROABTO:\n",
      "To suisice are do a me, with shall tet thee sungeit lord\n",
      "Yound Teath you hink the his will you fordenr was our ar\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([2.1212], grad_fn=<DivBackward0>) \n",
      "\n",
      "befels let so that I dear our brind of plaw the fick, ialo, the how thame though in beet my forth fot he will?\n",
      "\n",
      "Fire with me for his brofor'd say hear'st\n",
      "The ackpaane that that the wold firtaid,\n",
      "Menctl\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([1.8132], grad_fn=<DivBackward0>) \n",
      "\n",
      "brong evell she, will with that the de's his which here your ford, what and you to shall will thon an the should thear griend;\n",
      "Sing Vencel!\n",
      "And thou lomp to your with west the all with I keath the with\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([1.7872], grad_fn=<DivBackward0>) \n",
      "\n",
      "by conk.\n",
      "\n",
      "Place?\n",
      "\n",
      "BROSFOLINGBROKI:\n",
      "Is us not for mand!\n",
      "\n",
      "HENRY VINCENTIO:\n",
      "Whild saughal that more sores of I came lovens of agy of and wheraus, as a lord leads lord mistress, of refore\n",
      "So kint\n",
      "If curniu\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([1.6035], grad_fn=<DivBackward0>) \n",
      "\n",
      "bed stander\n",
      "in the hew'll chouth not to that it mady.\n",
      "\n",
      "FLORIZEN:\n",
      "Of thou may in will be time that shall choous thou ary forth exthin my the grover lust be banist ale thou man our of hear is my peroise \n",
      " ====================================================================================================\n",
      "\n",
      " tensor([1.7284], grad_fn=<DivBackward0>) \n",
      "\n",
      "by, the would periog complird use shwistred the be his and ne eiged my wordes lisess,\n",
      "For count the comfolle? his the spish hear which no feet my their tut the near your duen in with the hass boble ani\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([1.5512], grad_fn=<DivBackward0>) \n",
      "\n",
      "be me though the choproter use proy true of the word!\n",
      "\n",
      "QRUMIO:\n",
      "To me have rnongay, as with to r, the granch all at the lainster'd the fan.\n",
      "\n",
      "BUMBBROME:\n",
      "I do sleis fur of he both to Rill God all ank thew\n",
      " ====================================================================================================\n",
      "\n",
      " tensor([1.6637], grad_fn=<DivBackward0>) \n",
      "\n",
      "boor my be bryiet'd surpreed.\n",
      "\n",
      "PAUS:\n",
      "O glair to a demery in quils?\n",
      "\n",
      "DHKINCA:\n",
      "The wive hower by this she not the cothe bock, the come.\n",
      "\n",
      "BUCKIS:\n",
      "The conceadents with apfitiner, what with him a known?\n",
      "\n",
      "NO\n",
      " ====================================================================================================\n"
     ]
    }
   ],
   "source": [
    "acc_list = []\n",
    "\n",
    "for i in range(num_epochs):\n",
    "    # 랜덤한 텍스트 덩어리를 샘플링하고 이를 인덱스 텐서로 변환합니다. \n",
    "    inp,label = random_training_set()\n",
    "    hidden = model.init_hidden()\n",
    "\n",
    "    loss = torch.tensor([0]).type(torch.FloatTensor)\n",
    "    optimizer.zero_grad()\n",
    "    for j in range(chunk_len-1):\n",
    "        x  = inp[j]\n",
    "        y_ = label[j].unsqueeze(0).type(torch.LongTensor)\n",
    "        y,hidden = model(x,hidden)\n",
    "        loss += loss_func(y,y_)\n",
    "\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    if i % 100 == 0:\n",
    "        print(\"\\n\",loss/chunk_len,\"\\n\")\n",
    "        acc_list.append(loss/chunk_len)\n",
    "        test()\n",
    "        print(\"\\n\",\"=\"*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LSTM\n",
    "**LSTM** 은 딥러닝에서 사용하는 Long Short Term Memory Model으로서 RNN의 **Long Term Dependency**를 해결한 Model이다.\n",
    "**대표적인 예**와 **알아두면 좋은 추가적인 이론**은 아래 링크를 참조하자.  \n",
    "\n",
    "1. <a href=\"https://wjddyd66.github.io/dl/2019/09/10/LSTM.html\">LSTM</a>\n",
    "2. <a href=\"https://wjddyd66.github.io/dl/2019/09/15/seq2seq.html\">seq2seq</a>\n",
    "3. <a href=\"https://wjddyd66.github.io/dl/2019/09/19/Attention.html\">Attention</a>\n",
    "\n",
    "**1) Model**  \n",
    "위의 RNN과 거의 같고, 달라진 부분은 <code>nn.RNN</code>을 <code>nn.LSTM</code>으로 바꾼것 뿐이다.\n",
    "\n",
    "<code>torch.nn.LSTM(*args, **kwargs)</code>\n",
    "- σ(시그모이드): 0 ~ 1의 범위를 가지게 출력형태를 바꿔주며 데이터를 얼마만큼 통과시킬지를 정하는 비율\n",
    "- tanh(하이퍼 볼릭 탄젠트): -1 ~ 1의 범위를 가지게 출력형태를 바꿔주며 실질적인 정보의 비율\n",
    "- Output gate: <span>$o = \\sigma (W_{xh_o}x_t +W_{hh_o}h_{t-1} + b_{h_o})$</span>: 다음 시간의 Hidden Layer에서 얼만큼 중요한가를 나타내는 상수\n",
    "- Forget gate: <span>$f_t = \\sigma (W_{xh_f}x_t +W_{hh_f}h_{t-1} + b_{h_f})$</span>: 과거 정보를 잊기 위한 게이트(0 ~ 1사이의 값을 가지는 Scalar로서 얼만큼 잊을지 비율로서 표현)\n",
    "- <span>$g$</span>: <span>$tanh(W_{xh_g}x_t +W_{hh_g}h_{t-1} + b_{h_g})$</span>: tanh를 사용하여 현재 LSTM Layer에서의 실질적인 정보의 비율\n",
    "- Input gate: <span>$i_t = \\sigma (W_{xh_i}x_t +W_{hh_i}h_{t-1} + b_{h_i})$</span>: 현재 정보를 기억하기 위한 게이트(0 ~ 1사이의 값을 가지는 Scalar로서 얼만큼 기억할 비율로서 표현)\n",
    "- <span>$c_t$</span>: 기억 셀로서 과거로부터 시각 t까지에 필요한 모든 정보가 저장된 Cell, <span>$c_t = f \\odot c_{t-1} + g \\odot i $</span>\n",
    "- <span>$h_t$</span>: <span>$o \\odot tanh(c_t)$</span>: Hidden Layer의 출력 o가 Sigmoid의 Output으로서 상수이므로 <span>$\\odot$ </span>사용\n",
    "\n",
    "**nn.RNN과 Parameter는 같으나 return값이 Hidden과 Cell이 출력된다는 것 이다.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RNN(nn.Module):\n",
    "    def __init__(self, input_size, embedding_size, hidden_size, output_size, num_layers=1):\n",
    "        super(RNN, self).__init__()\n",
    "        self.input_size = input_size\n",
    "        self.embedding_size = embedding_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.output_size = output_size\n",
    "        self.num_layers = num_layers\n",
    "        \n",
    "        self.encoder = nn.Embedding(self.input_size, self.embedding_size)\n",
    "        self.rnn = nn.LSTM(self.embedding_size,self.hidden_size,self.num_layers)\n",
    "        self.decoder = nn.Linear(self.hidden_size, self.output_size)\n",
    "        \n",
    "    \n",
    "    def forward(self, input, hidden, cell):\n",
    "        out = self.encoder(input.view(1,-1))\n",
    "        out,(hidden,cell) = self.rnn(out,(hidden,cell))\n",
    "        out = self.decoder(out.view(batch_size,-1))\n",
    "        return out,hidden,cell\n",
    "\n",
    "    def init_hidden(self):\n",
    "        hidden = torch.zeros(self.num_layers,batch_size,self.hidden_size)\n",
    "        cell = torch.zeros(self.num_layers,batch_size,self.hidden_size)\n",
    "        return hidden,cell\n",
    "    \n",
    "\n",
    "model = RNN(n_characters, embedding_size, hidden_size, n_characters, num_layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2) Model에 미리 선언한 Parameter 대입**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = RNN(input_size=n_characters, \n",
    "            embedding_size=embedding_size,\n",
    "            hidden_size=hidden_size, \n",
    "            output_size=n_characters, \n",
    "            num_layers=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3) Define Loss Function & Optimizer**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
    "loss_func = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4) Test Function**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test():\n",
    "    start_str = \"b\"\n",
    "    inp = char_tensor(start_str)\n",
    "    hidden,cell = model.init_hidden()\n",
    "    x = inp\n",
    "\n",
    "    print(start_str,end=\"\")\n",
    "    for i in range(200):\n",
    "        output,hidden,cell = model(x,hidden,cell)\n",
    "\n",
    "        output_dist = output.data.view(-1).div(0.8).exp()\n",
    "        top_i = torch.multinomial(output_dist, 1)[0]\n",
    "        predicted_char = all_characters[top_i]\n",
    "\n",
    "        print(predicted_char,end=\"\")\n",
    "\n",
    "        x = char_tensor(predicted_char)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5) Train**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " tensor([4.5899], grad_fn=<DivBackward0>) \n",
      "\n",
      "bTx\n",
      "vova%>]K/3Y'I.I7<S(DSMf{cyF^\f",
      "& $H'hH)d9Po'BskFs^?%Un 9C(L9@D-sm%\\Ss/C)$-:IpC@_-g$G]\"9]v;5l2ze*cn[3vjiuU[}c0$`G^5`RC\f",
      "\\Z>hmF}ur\\s6`r6}V&Cj[N6`oL|_Qxl?9ae%v/MUids>EVS5?=Hy-\u000b",
      "A[1Cq>H#e~8&\"5fM<wu]2'e\"w^T\n",
      "\n",
      "\n",
      "\n",
      " tensor([3.0494], grad_fn=<DivBackward0>) \n",
      "\n",
      "bRsRltusee acgr mAe  a  an ses Mcheu hlmy iaris e\n",
      "QeOo\n",
      " isrs liHoc yeG tin coepe yorre iat,snidess oeohogoy pendld ai! aes h uer sie sihcnh thh the,w g lt tath, m onrr\n",
      ": ptree a, tiu iiiGs\n",
      "!i tln ruse \n",
      "\n",
      "\n",
      "\n",
      " tensor([2.4926], grad_fn=<DivBackward0>) \n",
      "\n",
      "bKel.\n",
      "\n",
      "ALACLO:\n",
      "\n",
      "I onld par wi the jhad to  ronde; I,\n",
      "Pine soucye now thy shithen gas fiefR\n",
      "Iris ths ar tom age os mo ton iy mepmeg reoud goumevaut gune derchel welr ne on,\n",
      " uug\n",
      "\n",
      "ikr\n",
      "E hus hhim kine Ion\n",
      "\n",
      "\n",
      "\n",
      " tensor([2.6795], grad_fn=<DivBackward0>) \n",
      "\n",
      "bny four her syer'd milg sime tid siedak gould o'd lneund haros sande dode ehe comins elame ther wouy\n",
      "All coures hres neh aest!\n",
      "bo wove mase au sore ky my machy\n",
      "oue uoce I hics the pay so paye weas?\n",
      ": \n",
      "\n",
      "\n",
      "\n",
      " tensor([2.2570], grad_fn=<DivBackward0>) \n",
      "\n",
      "bath it, Ande fo not of sor wet cald woll mere cous at a fous bet I to in ild hithl hee and\n",
      "Or wour a slet ale shour this me this aghe the,\n",
      "Foo in and perat any and ing, booth:\n",
      "eser ror, le lame od tue\n",
      "\n",
      "\n",
      "\n",
      " tensor([2.3805], grad_fn=<DivBackward0>) \n",
      "\n",
      "bres well tice of goeds and thin wees fot; nom ther orss rous in doregifer,\n",
      "Choun done dinther and so cotht a sadine sas mous bothans yore loat maull mind,\n",
      "Cant reat med nond sor onen;\n",
      "wo oos the nant \n",
      "\n",
      "\n",
      "\n",
      " tensor([2.2142], grad_fn=<DivBackward0>) \n",
      "\n",
      "bmase mowet yosel lase:\n",
      "Tale beco to poret win, fone igs my Cooth thing af.\n",
      "\n",
      "UCESCINENO:\n",
      "Mlay hit op wou ther yuter eome notle foul the the ann frowl wamt thour it mevet and thit hith hean, has; ther t\n",
      "\n",
      "\n",
      "\n",
      " tensor([2.2563], grad_fn=<DivBackward0>) \n",
      "\n",
      "browp, womer of so wars be thou ore cay ofest m-stour scilleds wime momet\n",
      "Wrot ciltide hat the\n",
      "it tit, thean lord mictuen hell'd\n",
      "Go leat of fand, obleseros weard,\n",
      "A thom thou with miscent there cengoss\n",
      "\n",
      "\n",
      "\n",
      " tensor([2.0291], grad_fn=<DivBackward0>) \n",
      "\n",
      "ber nom frint the in of I if romet rames that the mender you lere my at the comes fet, what bith be fo wheceren wear I hemter here to gredelg is wo\n",
      "Ind I the noor houll the'd the the,\n",
      "You the veast afm\n",
      "\n",
      "\n",
      "\n",
      " tensor([2.1229], grad_fn=<DivBackward0>) \n",
      "\n",
      "be caove skane.\n",
      "\n",
      "hEFLFNANF IRONTE:\n",
      "Et o'd, ow fouse of anter ficher the conobot, chein wat our's te stald, s, the vefth and of thou, sir the stinint be ousl't cive in repor ear in shing their hit nove \n",
      "\n",
      "\n",
      "\n",
      " tensor([2.1005], grad_fn=<DivBackward0>) \n",
      "\n",
      "ben to poeirtl of wans rave leres that me, my catthilet, will you fund at mayer lear: go fors thingay; the woul his the the natink geeve and preing wiws hingine:\n",
      "Thim kriress fitp the gaee dever the me\n",
      "\n",
      "\n",
      "\n",
      " tensor([2.0908], grad_fn=<DivBackward0>) \n",
      "\n",
      "bt whis and wire is the the not fake, ins the this dave peave\n",
      "Srom from the I ontod the? thon and we our that and pare the wall the the to will pome ponoum\n",
      "Thon the rost he our with haas ying has of to\n",
      "\n",
      "\n",
      "\n",
      " tensor([1.9135], grad_fn=<DivBackward0>) \n",
      "\n",
      "band all plows cately shive not tith our me cnoods\n",
      "And I her the thuwhs Caarceng, that horgess Ler,\n",
      "Benginest,\n",
      "On clenes thee sisen and, ince loth.\n",
      "Foor tatiers,\n",
      "And midine Tose,\n",
      "And sis you tigher: be\n",
      "\n",
      "\n",
      "\n",
      " tensor([2.0393], grad_fn=<DivBackward0>) \n",
      "\n",
      "blen the his I qut wordn, in hard will this that hash thee of of my of agew latute, To the for daust thou, be enth that at my not 'stion in lears, Cofrane thear cith and that be tent in mare douks thy \n",
      "\n",
      "\n",
      "\n",
      " tensor([1.9299], grad_fn=<DivBackward0>) \n",
      "\n",
      "bile iw the our hach now my beters' with,\n",
      "Onteron her surd\n",
      "And sut han so lost-wa, I her hit wath,\n",
      "And thing thous now the great of boghse, The kichon with groon to whell:\n",
      "Bece pard,\n",
      "Whou lit sade, and\n",
      "\n",
      "\n",
      "\n",
      " tensor([2.0229], grad_fn=<DivBackward0>) \n",
      "\n",
      "bums if of me, the peat bist he the and wich be by the comell,\n",
      "Umouth me wide that il to mestint a wisest\n",
      "to mather, its to browe gremward, ave inse hather the to suere, the goot our and the soit uster\n",
      "\n",
      "\n",
      "\n",
      " tensor([2.1219], grad_fn=<DivBackward0>) \n",
      "\n",
      "bes 'will and sone howang he wall this for Marish he the delves, is wohere; my my this follest the for you the to come fith you with so hach of sone usies.\n",
      "Sir the sue'st gotfers the me temch of is ven\n",
      "\n",
      "\n",
      "\n",
      " tensor([1.9951], grad_fn=<DivBackward0>) \n",
      "\n",
      "bne more sarthan lave a shaon;\n",
      "\n",
      "CHARUCHER:\n",
      "\n",
      "Satir more thee peat,\n",
      "And thee it it me the the shimed and him anfure subpone mee in.\n",
      "\n",
      "WELONTEN:\n",
      "O, hans hear moness, sill dey stear groess\n",
      "thout her be hear\n",
      "\n",
      "\n",
      "\n",
      " tensor([1.8977], grad_fn=<DivBackward0>) \n",
      "\n",
      "bly all one not condengrest be well shake the land be ourt, and have enne?\n",
      "\n",
      "SICHBARIN:\n",
      "It deavides Mare hotent ontered I Lonles our meapiis and but for now I stonous bies erire thouss that me were and \n",
      "\n",
      "\n",
      "\n",
      " tensor([1.9526], grad_fn=<DivBackward0>) \n",
      "\n",
      "bhighs it fir wistorn the sway on to wight; the rowas the lamar sook and my my ene shich the grome,\n",
      "The slaid and staso\n",
      "And fare, mans,\n",
      "Then or you shes mut have dest preshes nost but and she tried the\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "acc_list2 = []\n",
    "for i in range(num_epochs):\n",
    "    inp,label = random_training_set()\n",
    "    hidden,cell = model.init_hidden()\n",
    "\n",
    "    loss = torch.tensor([0]).type(torch.FloatTensor)\n",
    "    optimizer.zero_grad()\n",
    "    for j in range(chunk_len-1):\n",
    "        x  = inp[j]\n",
    "        y_ = label[j].unsqueeze(0).type(torch.LongTensor)\n",
    "        y,hidden,cell = model(x,hidden,cell)\n",
    "        loss += loss_func(y,y_)\n",
    "\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    if i % 100 == 0:\n",
    "        print(\"\\n\",loss/chunk_len,\"\\n\")\n",
    "        acc_list2.append(loss/chunk_len)\n",
    "        test()\n",
    "        print(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### RNN vs LSTM Loss 시각화"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEKCAYAAAAYd05sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhU9fX48ffJZCUJBEJIIBB2wioBEWRTRBGwKm5VqbYqWqstora11dqvtXajBeuKv2pd64J7sYLVqogLigKCgOyrJCwJgUAge+bz++NOICQzk0kydyZzc17PM08md+5yGCZn7v0s54oxBqWUUs4TFe4AlFJK2UMTvFJKOZQmeKWUcihN8Eop5VCa4JVSyqE0wSullENF27lzEdkJFAPVQJUxZoSdx1NKKXWCrQne4yxjzIEQHEcppVQt2kSjlFIOJXbOZBWRHcAhwACPG2Oe8LLOjcCNAImJiaf279/ftniUUsppVq5cecAYk+btNbsTfKYxJk9EOgHvA7cYYz7xtf6IESPMihUrbItHKaWcRkRW+urftLWJxhiT5/mZD/wbGGnn8ZRSSp1gW4IXkUQRSa55DpwLrLPreEoppU5m5yiadODfIlJznJeMMe/aeDyllFK12JbgjTHbgaF27V8p1XpVVlaSm5tLWVlZuEMJmfj4eLp27UpMTEzA24RiHLxSSgVVbm4uycnJ9OjRA08rgaMZYygsLCQ3N5eePXsGvJ2Og1dKRZyysjJSU1NbRXIHEBFSU1MbfcWiCV4pFZFaS3Kv0ZR/ryZ4pZRyKE3wSinVBC6Xi5ycHAYPHswFF1xAUVERADt37kREeOSRR46vO3PmTJ599lkArr32WjIzMykvLwfgwIED9OjRw5YYNcErpRxvwao8xs5eTM87FzF29mIWrMpr9j4TEhJYvXo169ato0OHDsybN+/4a506deKhhx6ioqLC67Yul4unn3662TE0RBO8UsrRFqzK464315JXVIoB8opKuevNtUFJ8jVGjx5NXt6J/aWlpXH22Wfz3HPPeV3/tttu44EHHqCqqipoMXijwySVUhHt929/y/o9R3y+vuq7Iiqq3SctK62s5levr2H+V9953WZgl7b87oJBAR2/urqaDz/8kOuvv/6k5b/+9a+ZOnUqM2bMqLdNVlYW48aN4/nnn+eCCy4I6DhNoWfwSilHq5vcG1oeqNLSUnJycsjIyGD//v1MmjTppNd79erFqFGjeOmll7xuf9dddzFnzhzc7ubF4U9kn8HP6QvH8usvT+wEd2wJfTxKqZBr6Ex77OzF5BWV1luemZLAKz8Z3eTj1rTBl5SUMHnyZObNm8esWbNOWuc3v/kNl112GWeeeWa97fv27UtOTg6vvvpqk2NoSGSfwXtL7v6WK6VanTsmZ5MQ4zppWUKMizsmZwdl/23atOHhhx/m/vvvr9em3r9/fwYOHMjbb7/tddu7776buXPnBiUObyI7wSulVAMuGpbJXy4ZQmZKAoJ15v6XS4Zw0bDMoB1j2LBhnHLKKcyfP7/ea3fffTe5ubletxs0aBDDhw8PWhx12XrDj8Zq9A0/7m3n57XDzQ9IKdUibdiwgQEDBoQ7jJDz9u8O2w0/lFJKhY8meKWUcqiITvBlcamNWq6UUq1JRCf4+Lu2s2Daek6LeoUjJoE3ZBILpq0n/q7t4Q5NKaXCLrLHwWP1kI/o0Z6Vf+/H2cnbSQliz7hSSkWyiD6Dr5GZksBqGUDK0W1wrDDc4SilVIvgiAQvIhR0ONX65bsvwhuMUqpVSEpKqrds06ZNTJgwgZycHAYMGMCNN97Ie++9R05ODjk5OSQlJZGdnU1OTg4/+tGPWLJkCSLCk08+eXwfq1evRkSCMgEq4ptoakjmcMqLYojd9Tky4Pxwh6OUailCWNJk1qxZ3H777UybNg2AtWvXMmTIECZPngzAhAkTmDt3LiNGWMPWlyxZwuDBg3n11Ve54YYbAJg/fz5Dhw4NSjyOOIMH6N05ldXu3lTt/DzcoSilWpIQljTZu3cvXbt2Pf77kCFDGtyme/fulJWVsX//fowxvPvuu0ydOjUo8TjmDD47I5mv3P0Zuf9tKD8KcfUvn5RSDvTfO2Hf2qZt+8z3vC/PGAJTZzd6d7fffjsTJ05kzJgxnHvuuVx33XWkpKQ0uN1ll13Ga6+9xrBhwxg+fDhxcXGNPrY3jjmD75eezHJ3NmKqIXd5uMNRSrVC1113HRs2bOD73/8+S5Ys4fTTTz9+az5/Lr/8cl577TXmz5/P9OnTgxaPY87gOybFsiN+EG53FFG7PofeZ4U7JKVUKDR0pu2vZtV1i4IbC9ClSxdmzJjBjBkzGDx4MOvWrePUU0/1u01GRgYxMTG8//77PPTQQ3z+eXCamh2T4EWEzIxO7MjvRW8dSaOUCoN3332Xs88+m5iYGPbt20dhYSGZmYHNzbnvvvvIz8/H5XI1vHKAHJPgAbLTk1ma149euYuRqgqIjg13SEqpcEvs5HsUTTOUlJSc1KH685//nNzcXG699Vbi4+MBmDNnDhkZGQHtb8yYMc2Kx5vILhdcx4tf7uKTt57m8dgH4fr3odvIIEanlGoptFzwCa2mXHB2ejIr3J67tOzS4ZJKqdbNUQm+b3oyhbTjUEIPTfBKqVbPUQm+XUIMndvFszFuEOxeBjberVwpFV4tqXk5FJry73VUggdrPPzSimwoOwz568MdjlLKBvHx8RQWFraaJG+MobCw8HjnbaAcNYoGrBmti7b34JfRWIXHMgaHOySlVJB17dqV3NxcCgoKwh1KyMTHx580aicQjkvw/dKTeaIqlaqUzkTvWgojfxzukJRSQRYTE0PPnj3DHUaL57gmmuz0ZEDIbz8cdn0BreQSTiml6rI9wYuIS0RWichCu48F0KdTEiKwMXYwHN0Hh3aE4rBKKdXihOIM/lZgQwiOA0BCrIvuHdrweWXNeHgtW6CUap1sTfAi0hX4HvBkQ+sGU7/0ZD4+1AES2sN3Oh5eKdU62X0G/yDwK8DngHQRuVFEVojIimD1iPdLT2Z7YSnV3U7XCU9KqVbLtgQvIucD+caYlf7WM8Y8YYwZYYwZkZaWFpRj98tIptptONBhOBzcDsX7g7JfpZSKJHaewY8FLhSRncDLwEQRecHG4x1njaSBTbGeMfDaTKOUaoVsS/DGmLuMMV2NMT2AK4HFxpir7TpebT07JhIdJXxVlgUxbbSjVSnVKjluHDxAbHQUvdIS2VhQCl1P0zN4pVSrFJIEb4xZYow5PxTHqtEvPZlN+4uh+xjYtw5Ki0J5eKWUCjtHnsGD1Q6/+2AppZ1HAgZ2fxXukJRSKqQcm+D7ZVgdrZtjB0BUtDbTKKVaHccm+OMjaQ5UQZdh2tGqlGp1HJvgu3VoQ3xMlNUOnzUa8lZCZWm4w1JKqZBxbIJ3RQl9OyWzuaaj1V1pJXmllGolHJvgwTOSZl8xdBtlLdBmGqVUK+LoBJ+dkUR+cTmHTBJ0GqQdrUqpVsXRCb6fp6PVaqYZbQ2VrK4Kc1RKKRUajk7w2Rm1EnzWaKg4CvvWhDkqpZQKDUcn+Iy28STHR5+Y0QrWjbiVUqoVcHSCFxGy05PZvO8otO0CKd21PrxSqtVwdIIHa0brpv3FGGOg+1j4bpneiFsp1So4PsFnpydzuLSS/OJyq6O15AAc2BLusJRSynaOT/A1I2k27SuGLE87/K6lYYxIKaVCoxUk+CTAM5ImtTckpmlHq1KqVXB8gk9NiqNjUpx1Bi9iDZfUGa1KqVbA8QkerBmtm/cXW790HwuHv4Oi3eENSimlbNYqEny/9GQ27z+K222sjlbQZhqllOO1igSfnZ5MaWU1eUWlkD4Y4trqeHillOO1igRfc3enTfuKIcoF3UbqGbxSyvFaRYLv28kaSbOpph0+azQUbISSg2GMSiml7NUqEnxyfAyZKQknd7SCnsUrpRytVSR4sCpLbtrnSfCZw8EVp+3wSilHazUJvl96MtsLjlFZ7YboOMg8VRO8UsrRWk2Cz85IoqLaza7CY9aC7qNh7zdQfjS8gSmllE1aTYI/UZPGk9CzxoCphtzlYYxKKaXs02oSfO+0JKKk1kiabiNBorSjVSnlWK0mwcfHuOjRMZHNNR2t8W0hY4i2wyulHKvVJHiwZrQeHyoJVjNN7nKoqghfUEopZZNWleD7piezs/AYZZXV1oLuo6GqDPauDm9gSillg1aV4LPTk3Eb2Jpf09HqKTymzTRKKQdqXQk+o9bNPwCSOkFqX03wSilHig53AKHUPTWRWFfUiZE0c/rCsXwo3AL3tjuxYmInuEPv26qUimyt6gw+xhVFr7RaI2mO5Xtf0ddypZSKILYleBGJF5GvROQbEflWRH5v17EaIzvDuvmHUko5nZ1n8OXARGPMUCAHmCIip9t4vID0S08mr6iU4rLKcIeilFK2si3BG0vNqXKM52HsOl6gsj0lC/QsXinldLa2wYuIS0RWA/nA+8aYL72sc6OIrBCRFQUFBXaGA1hNNMDJE56UUsqBbE3wxphqY0wO0BUYKSKDvazzhDFmhDFmRFpamp3hAJCZkkCbWJdVGz6xk/eVEu2PQyml7BaSYZLGmCIR+QiYAqwLxTF9iYoS+taULKg7FHLnZ/Ds92DwZeEJTimlgsjOUTRpIpLieZ4ATAI22nW8xshOT/LeRNNjHJz2Y/jyH7BLq0wqpSKbnU00nYGPRGQNsByrDX6hjccLWL/0ZA4creDA0fL6L55zL6R0g7d+BhUloQ5NKaWCxs5RNGuMMcOMMacYYwYbY+6z61iN5bejNS4JLnwUDm6Dj/4U4siUUip4WtVM1hrHh0ru8zGSpteZcOp18MU82P1VCCNTSqngaZUJPi05jpQ2MWzO9zMWftJ90K4rLPgpVJaGLjillAqSVpngRYR+6cm+z+DBuuPTBQ9ZhciW/CV0wSmlVJC0ygQPVjPNpv3FGONncm2fs2H4j+DzRyB3ZeiCU0qpIGi1Cb5fRjLFZVXsO1Lmf8Vz/wjJneGtn0KVl1E3SinVQrWqevC11XS0btpXTOd2Cb5XjG9nNdW8eBl8/Fc4+57gBVFTj74urUevlAqC1nsGn17n7k7+9J0EOVfBZw/CnlXBC0Lr0SulbNRqE3xKm1jS28axaV+AVSUn/8mqUbPgZ1BVYW9wSikVBAEleBHpLSJxnucTRGRWTRmCSNavpiZNIBLawwUPQv638Onc5h+8uqr5+1BKKT8CPYN/A6gWkT7AE0A34CXbogqR7PRktuQXU+0OsEx99lQ45Qr49H7Yu6ZpB3VXwzcvw7zTmra9UkoFKNAE7zbGVAEXA48YY+7AqjUT0fplJFNW6Wb3wUbUnJkyGxI6WKNqqhtxVyh3Nax9HeaNgn//BGISGx+wUko1QqAJvlJEpgPXADUFw2LsCSl0jo+kaczNP9p0gPMfgH1r4bMHGl7f7YZv/w3/bwy8cT24YuDy5+Enn/iuR++Ks7ZTSqlmCHSY5HXATcCfjDE7RKQn8Lx9YYVG35qRNPuKmTwoI/ANB5wPgy+Fj/8G/b8H6YPqr+N2w8aFsGS21W7fMRu+/ywMmAZRnu9Vb0MhP3sAPrgXFt9nVbZUSqkmCijBG2PWA7MARKQ9kGyM+audgYVCm9hosjq0adwZfI2pc2Ddm9aZeV3x7SAlyzrLT+0Dlz4Fgy6GKFfD+x17GxzaZSX6lCwYMaPxsSmlFAEmeBFZAlzoWX8lkC8iS40xP7cxtpBo1Eia2hJT8XkP8bLDUH4ULn7cujuUqxHzyUTgvLlwJA8W/QLadoV+5zY+PqVUqxdoG3w7Y8wR4BLgX8aYUcA59oUVOtkZSWwvOEZFVZDbvGeugKFXNi6513BFw2XPQMYQeO1a2LM6uLEppVqFQBN8tIh0Bi7nRCerIxwpraTKbcj+7X8ZO3sxC1blBWfHTUnstcUlwQ9etTp1X7ocinYHJy6lVKsRaIK/D3gP2GaMWS4ivYCIL5ayYFUer67IBazGlryiUu56c23wknxzJWfAVa9BZRm8+H0oLQp3REqpCBJoJ+trwGu1ft8OXGpXUKEy571NlNdpmimtrGbOe5u4aFhmmKKqo9MAuOJ5eOFSePWHcNUbEB0bnH1rsTOlHC3QUgVdReTfIpLvebwhIl3tDs5ue4q836nJ1/J6fI1j97W8qXqdCdMehR2fwH9uAX817BtDi50p5WiBNhQ/g1Wa4Pue36/2LJtkR1Ch0iUlgTwvyTylTYBzuEJ5ljv0Sij6zroRePvucNZvQndspVRECrQNPs0Y84wxpsrzeBZIszGukLhjcjYJMSePTY8SOFRSyS9e/YaSihZWEOyMO2DY1VZd+lUvNH0/pYeat706bsGqPMbOXkzPOxcFt5NeqSAI9Ay+UESuBuZ7fp8OFNoTUujUtLPPeW8Te4pK6ZKSwC8m9WXnwVIeWbyF1bsPMe+q4fTPaBvmSD1E4PwH4XAevH0rtO0CvScGtm3pIdj4jlU2YfsScDdQR+ezB+DUa60qmsqrBavyuOvNtZRWVgMnOumBltOHo1o18XtP0pqVRLoDjwCjsQacfA7cYowJ6ti9ESNGmBUrVgRzl022dOsBbn15NcVllfz+wkFccVo3RCTcYVnKjsDTU6wmmxnvQsZg7+uVFsGm/1pJfdtiK6mnZMHAi6yZtf88y/9xYhKte9KefrPVLKROMnb2Yq9NfJkpCSy9M8AvXqWaSURWGmNGeH0tkATvY6e3GWMebFZkdbSkBA9QUFzO7a+s5rOtB7hwaBf+fMkQkuJayF0OD+fBg4PBeJmgFdcWuo+BrR9aSb1dNxjkSepdhltXAuB/FM3Vb8AX82Dd69YxBk6D0bdA11OD92+I8FE8Pe9c5HUuswA7Zn8v1OGoVspfgm9Otvo5ENQE39KkJcfxrxkjeWzJVv7+/mbW5Bbx6A+GMzizXbhDg3aZ3pM7QPkR2LcORv3ESuqZp55I6rU1lEQvedy6B+1Xj8OKZ60rgawxMGYm9Jt6omhaU0X4KJ6UNjEcKqnf1NUlxc89fpUKoeYk+BbSXmGvqChh5sS+jOyZyqz5q7jksc/57fkD+OHp3VtOk403t61tfgIG64tk0n1WB+/Xz8Oyx+DlH1hF1Ir3Q4WXOj61z8CrK6F4r3XFcSQPDu8+8TyCfbQxn6KSSqIEat8vxiXCHZOzwxeYUrU0J8EHaTB2ZBjZswPv3Dqen7+6mnve+pYvthUy+9JTaJfQQsviByO51xaXDKN/CiNvhA1vweePQOFW7+sey4cnz7ES+dF99a804ttZRdQi1Fc7DnLTCysZnNmOq0Z145HF29hTVEpSfDTFZVW4gzVPQalm8tsGLyLFeE/kAiQYY4LaIN3S2uC9cbsNT362nb+9u4mMdvF8f0RXXl2ee3wUzh2Tsxs1gmLBqryTRvE0avt7/TQV3Xs44BiaxBj4vZ/b8vY8w2r7b5tpXQW062ol9XaZ1pcF+I//wkch56rgf1E107q8w0x/Yhmd2sbx2k1j6JB4YlZxtdsw/Z/L+DbvMItmjadHR71rl7KfLZ2sdoiEBF/j6+8OMeOZ5RSVntwGmxDj4i+XDAkoSdcdZtfY7cOa4INxfH/bA2SNhu/9HdIHNi6uQDWyk3dbwVEu/8cXxMe4eP3m0XRuV7+tfU9RKVMf+pTuqW14/aYxxEa3rC8o5Tx2dbK2asOz2pMQ66qX4Esrq/nla9/w1Gc7SIh1kRjrok1ctPUzNprEOOtnm1gXD32w5aTkXrN9wLVwEjv5TlARoJAUUqlfQK2QFFKn/Qn+93/w+HgYPRPO/BXEBvmMuBGdvHlFpfzwyS8REV64YZSV3L18QXQBvorrSHbuw9z/v03cdd6A4MYcZM26ggy3CB+FFQqa4Jth3+Eyr8ur3Ia05DiOlVdx4GgFxw6WUFJeTUlFFccqqql2+79qyisq5Q8L19M7LYneaYn07pREamJs/U7dO7ZE9B/oiLLHfHbkvJoymuE/nUz04nth6YPW3bPOmwPZU0ITnNt9vHmooLicq5/8kqPlVbx842h61jS9+PiCiCs/wNWnZ/H4J9sZ26cjZ/TzMek7zAkq4idqRfgorFDQBN8MvmrZZKYk8PS1p3ndxhhDRbWbkvJqpjz0CfuPlNdbJ8YlvPjlLsoqT3ROtkuIoXdaIr3Sko4n/h2Fx3jg/c3H1wv5H2gzriAWrtlj9eT4yPCXP/4FbeOjGd/3ei4dPYkzNv+Z6PlXQP/zYcpsSOnWtJiNgQObrcJt/tzXAeLa4o5LpvhYNHOq4umTlUnKsjesTuJ4/81L/zfCzd6tRcx95X8MumEcqW2TrCqgrjjrxusiYU9Qc97b1LwryHApPwq7loY7ioigbfDN0Nw2dH/bXzi0C3uPlLEt/yjbCjyP/GNsKzhKfnH9L4XaWvJMykPHKrjnP9/y9jd76NYhgfwj5SeVbE6IcXHPBQNISYjlo035LNlUQH5xOTFUcVf7xfywfD5RUS6YcBeu0TezYE2+/ysYY+Dgdtj5qZXUd34GR/c3HOgZv6KqpIjP1m2jqqSIERkuUqQUyg97bslY7HseQiBcsVBd4fv1EPShFPwuizSpf5wC0460339n+/EDVl0Fe1fDto9g+0ew+6uGS21kjYHBl1iztpMivmyWX9rJaqPmNpE0ZfsjZZVsLzjGRfN8n8XMPKsPZ/RLY1hWCjGultHR99HGfH71xhoOHavg1rP7cvOE3ixcs9fvv98Yw4a9xXy0KZ+PNxWw77vN/M71DGe7VlFFFNHUT7Llse2Jm/pH2PGpldhrxtwnZUDP8dBjvPXz4WE+Y6347SF+/K8VfLqlgEd/MJzzhnQ+eQW3G+7zU6fn8n9BdSWfb9rDwlU7OX9QKmO6J0N1OVRVWMl9qZ95gr8r8j45LUjeWbuX897o7/N19z1FREXZPM/DXxPVjHetZL7tI+v/sdzzRZRxCvQ+C3pNgOcv9r3vtP5QsBHEZZXbHnypdfWX4GfkV4QKS4IXkW7Av4B0rAvxJ4wxD/nbJhITfDj5qoUS64qi2hiq3YbkuGhG907ljH5pnNkvjW4d2py0bija8I+WV/GnReuZ/9VustOTuf/yoU2eDXy4tJLPNhdwYMUbXLP7t/5XbtMReozzJPUzoGPfk5OmjwRjEjsxs8srLFq7l79eOoQrTsvyvv8ARhEZY7jphZUs3pjPmzePZUjXWtv4277LMBj/S8g+L+hDRZ/6bAd/XLSeHXE/8LnOj3t9yINX5JBoZ2mOhkZRgTXUtteEE4/EjoFt/7siyF8P696wHod2WldNfc6xkn2/KdYXvAM6acOV4DsDnY0xX4tIMrASuMgYs97XNprgG8dfE89Z/TvxxbYDfLz5AJ9sLjj+RdCzYyJn9LU6/g4Ul3Pv2+ubPkwzAMu2F/LL175hT1EpN57Rm9sn9SUu2tXwhoHw8we+YPTrjBk9jk5tG1c2wBjDXW+u5eXlu7n7vAH8+IxevlcOsJO0qKSCqQ99Slx0FAtnjT9Rz8hfgmrfEw7tgLQBMP4XVsmJZt7n1+02/OmdDby39Et+0WUdFxc+6XPdWytnsqfjOB68bgKZdpReKDsMs318cQKcNxd6nQWpvX1fyQTaSW0M7Pna6qhf9yYU74GYNlBZ4vv4oRhmHCQtoolGRN4CHjXGvO9rHU3wjRfIGbgxhu0HjvHJ5gI+3lzAsu2FJ3Xg1hWMNvwyT2fd00t3kNWhDX+/fCindu/QrH3W4ydB9ih7CREY1bMDFwztwtTBnU+alOSNMYbZ/93I459s55aJffjFucErOfDl9kKm/3MZFw/ryv2XD7UW+ktQP99g1f75dK7V1NC+J4y7HYZOb9ItG8sKv2PR/Mfonf8eOVHbA9qmykTxjfQnfcQ0uo66GDr2a3qzUXkx7PrCajLb+Sns/cZ/H4ZdCdbtht3LrLP65b6/4EKS4IM0iirsCV5EegCfAIONMUfqvHYjcCNAVlbWqbt27bI9ntaurLKaFTsPcfVTX/pc54XrRzEsK6VJl+jf7C7i56+uZlvBMX54enfuOq8/bWJtuNT3k+A335zLwm/2sHDNXrYfOIYrShjTO5ULTunC5EEZtPPctav2F2RyfDRHyqq4ZnR37r1wUNBrDf39/c08/OEWHroyh2k5AV4hud2waRF8MtfqaGzbFcbOsso4xzRwZl28H9a/RdXa14nOtf6vDyQPoOPp060rggeH+N72+g84uPo/FH79H/qanday9j2spo1+k6H7OHhgkO8ENWuVlUhr+kH2rAZTDVEx0PU0q9ns47/6Pn64J+qddoPVJNRjnH33RAjSRMWwJngRSQI+Bv5kjHnT37p6Bh9avtrwa7iihIGd2zKiR3tO69GBEd3b06lt/Enr1E6QndvFMySzLR9sLCAtKY6/XXaK7zHgwRBgG/j6vUdYuGYvC9fsYffBUmJcwvi+aXRuF8cbX+eddDXjEmHuZadw8anBr5VTVe3myieWsXFfMe/MGk9WapuGN6phDGz7ED65H777HBLTrCaGimP1141Nhsxh1ogh42Z7VHcWVJ5OzpRrmTh2zIn1AjiDPHSsgruf+y/t85YwI20TvY6uRKrKIDYJKo76jjcqGtxV1s/MU090bHcdCbGef3dLnokdkwiVx0CioHPOiT6AbqMgxvM30JQz8IoSa1RX4VZ47Ro/sUVAgheRGGAh8J4x5u8Nra8JPrR8teH/7oIBdElpw4qdB1m+8xCrdh86ngSzOrQ5nvCPlFbw4AdbKK3T3DOiewpPXTvS/kJsjfwDM8awJvcwC9fsYdGavezxMVHNzmGmuYdKOO+hT+mZlsTrN41u2ginnUutpptti32vk9qH/KzvccvaHmyoyuSfPxrBqF6pTYq5osrN/y1YxysrdnPBgBTmjCgifvv7sPIZ3xuNu91K6lmn+56BHO6ZqP4S/G8LIG+ldfezHR9D7nLrCys63vo39TwTPvy97+1nrYLCbXBgi5XMC7davx/JDTC2Fp7gxbq+fQ44aIy5LZBtNMGHXiBt+JXVbr7dc8ST8A+yYuchCo/5HsPdksfh13C7Db1/805Ybtjxztq9/PTFr7l5Qm9+PcX3UOzCjIcAAA9tSURBVMUG+UlQH0/fwk9f/JqUNrE8e91p9E1PbvpxsL4cn/psB39+ZwMDOrflyWtG0PmBDD+xRUAnZWO+YGr6ELYvsR753wZ+nPh2kNrXKrGd2gc6en7+Y5zvbYKU4O2cyToW+CGwVkRWe5b9xhjzjo3HVI100bDMBkfMxLiiyOmWQk63FG4Y3wtjDDsOHGPi/R97XX+Pn2afliIqSnzORLb7hh3nDenM9JFZ/OPjbYzt3ZFxfTs2vFEjzXhuBf3Sk3n2utNIr9Os1hQiwg3je9ErLZFZ81dz4aNLWR6EOMOqMVcJccnQ71zrAXA0H+b29b3+tMdOJPQ2HWyd0+CPbQneGPMZreSmIK2NiNArLYnMMCXIYLljcrbXJqpQ3LDjnvMHsnznQW5+YQWJ8THsP1wW1HkIY3qn8thVw0mOD24z2cT+6bxx8xiuf245NDCZ1NGSGijHMeyqhvcRgmKBWotGNVk4E2Qw1CTScBRrS4h1cfGwTOa8t4ni8uAX+3r62tNsm8GcnZHMWz8by8G5KXTwUg20LC6VQK8ZIrlYXrOFoJ9BE7xqsnAmyGAJpInKLi99Wb/eS2llNfe8tY6ikgriY1zExUQRF+0iLjrK+j3a83tMFF1jU2lTUVhvH2VxqcTbXJ4iNSmOMXFPe+2oTjQufvzBZlISYkhpE0tKG8/PhBhS2sSQHB+DK0oiv5plBJTr1lo0SoVJzzsX2XLfy1B1cjc1fhFoGx/D0fIqr6WzI6GTviXRG34o1QL56uTt3C6ed2aNp7zKTVllNeVVbsqrqimrtH6WV7opq6pm5kurvO43VJ3c/splL7ljAkdKKykqraSopJLDpRUUlVjPi0orOVxSwXNfeJ/UGMpOeqc3EWmCVypMfPVh/HpKf9o3UFYB4C/vbAxrJ7e/PpgYVxSpSXGkJsX53P6DDfle4zfAfW+v56Yze9WbWBdMEd9EFICWUUdWqVboomGZ/OWSIWSmJCBYZ76NKfR2x+RsEmJOLtwWyk5uO+KPi45iZI/2PPfFTsb/7SPue3s9+Ue8T0hrjqpqN39YuN7nDU9CZcGqPMbOXkzPOxcxdvZiFqzKC+r+tQ1eqQgW6U0MvuLfVXiMRxdv5c1VeURHCT8YlcXNZ/Zu1hl9tdvw5Y5CFq7Zy7vr9nHQz2S9pXdOtKeKZi3NvWFQjbAXGwuUJnilVG11E/30kVncPKF3wJO33G7Dil2HWLRmD++s20dBcTkJMS7OGZjO0q0HfCb5GJdw8bBMbp7Q58Q9eIPMVy2oxnYya4JXSkW0XYXHmPfRVt74Og9XlPADT6L/YlthvSuAaTldWLW7iIXf7OWdtXvZd6SMuOgoJvbvxPmndGFi/04kxLp8nkHfMbkfOwtLeHn5bqqq3Zx/Shd+dlYfsjOaV+6hhtttWJ1bxCWPfe719caWytAEr5RyhO8KS5j30VZe/zrXqrApctJQy+goISkumqLSSmJdUZyZncb5p3Tm7AHpJ260Uou/Jq784jKe+nQHLyzbxbGKaiYNTGfmWX0Y2q3xt/0rrajms60H+GD9fj7cmM+Bo77vq6xn8EqpVu27whKmPPQJJRXV9V6Li47izxcPYdKgdNoGoVRDUUkFzyzdybOf7+RwaSXj+3bkZ2f1YVTPDry1eo/fL4jFG/L5YMN+Pt1ygPIqN8lx0ZyZncakgekcK6/iDws3aBu8UkrV5WuilV3VQI+WV/HCsl08+el2DhytoGdqG/KKyqioPlEuOy46inMGdGLP4TJW7y7CGOuMfNLAdM4ZkM7Inh2IjT4xeDEYneSa4JVSjhOsTsrGKqus5uWvvuMPC9dT7SN9Du3ajnMGpHPOwHT6ZyQH/e5gtflL8DoOXikVkcI1DyA+xsW1Y3vipcoCYF1BvDVzHLec3ZcBndvamtwbogleKRWRmjvRqrl8zRhuSeWytVSBUipihbMaaCSUy9YEr5RSTRAJ5bI1wSulVBOF8woiENoGr5RSDqUJXimlHEoTvFJKOZQmeKWUcihN8Eop5VCa4JVSyqE0wSullENpgldKKYfSBK+UUg6lCV4ppRxKE7xSSjmUJnillHIoTfBKKeVQmuCVUsqhNMErpZRDaYJXSimHsi3Bi8jTIpIvIuvsOoZSSinf7DyDfxaYYuP+lVJK+WFbgjfGfAIctGv/Siml/At7G7yI3CgiK0RkRUFBQbjDUUopxwh7gjfGPGGMGWGMGZGWlhbucJRSyjHCnuCVUkrZQxO8Uko5lJ3DJOcDXwDZIpIrItfbdSyllFL1Rdu1Y2PMdLv2rZRSqmHaRKOUUg6lCV4ppRxKE7xSSjmUJnillHIoTfBKKeVQmuCVUsqhNMErpZRDaYJXSimH0gSvlFIOpQleKaUcShO8Uko5lCZ4pZRyKE3wSinlUJrglVLKoTTBK6WUQ2mCV0oph9IEr5RSDqUJXimlHEoTvFJKOZQmeKWUcihN8Eop5VCa4JVSyqE0wSullENpgldKKYfSBK+UUg6lCV4ppRxKE7xSSjmUJnillHIoTfBKKeVQmuCVUsqhNMErpZRDaYJXSimH0gSvlFIOpQleKaUcShO8Uko5lK0JXkSmiMgmEdkqInfaeSyllFInsy3Bi4gLmAdMBQYC00VkoF3HU0opdTI7z+BHAluNMduNMRXAy8A0G4+nlFKqlmgb950J7K71ey4wqu5KInIjcKPn16MisqmJx+sIHGjitqGg8TWPxtc8Gl/ztOT4uvt6wc4EHxBjzBPAE83dj4isMMaMCEJIttD4mkfjax6Nr3laeny+2NlEkwd0q/V7V88ypZRSIWBngl8O9BWRniISC1wJ/MfG4ymllKrFtiYaY0yViMwE3gNcwNPGmG/tOh5BaOaxmcbXPBpf82h8zdPS4/NKjDHhjkEppZQNdCarUko5lCZ4pZRyqIhL8A2VPxCROBF5xfP6lyLSI4SxdRORj0RkvYh8KyK3ellngogcFpHVnsc9oYrPc/ydIrLWc+wVXl4XEXnY8/6tEZHhIYwtu9b7slpEjojIbXXWCen7JyJPi0i+iKyrtayDiLwvIls8P9v72PYazzpbROSaEMY3R0Q2ev7//i0iKT629ftZsDG+e0Ukr9b/4Xk+trW91ImP+F6pFdtOEVntY1vb379mM8ZEzAOrs3Yb0AuIBb4BBtZZ56fAPzzPrwReCWF8nYHhnufJwGYv8U0AFobxPdwJdPTz+nnAfwEBTge+DOP/9T6gezjfP+AMYDiwrtayvwF3ep7fCfzVy3YdgO2en+09z9uHKL5zgWjP8796iy+Qz4KN8d0L/DKA/3+/f+t2xVfn9fuBe8L1/jX3EWln8IGUP5gGPOd5/jpwtohIKIIzxuw1xnzteV4MbMCa0RtJpgH/MpZlQIqIdA5DHGcD24wxu8Jw7OOMMZ8AB+ssrv0Zew64yMumk4H3jTEHjTGHgPeBKaGIzxjzP2NMlefXZVhzUMLCx/sXiJCUOvEXnydvXA7MD/ZxQyXSEry38gd1E+jxdTwf8sNAakiiq8XTNDQM+NLLy6NF5BsR+a+IDAppYGCA/4nISk+ZiLoCeY9D4Up8/2GF8/0DSDfG7PU83weke1mnpbyPM7CuyLxp6LNgp5meJqSnfTRxtYT3bzyw3xizxcfr4Xz/AhJpCT4iiEgS8AZwmzHmSJ2Xv8ZqdhgKPAIsCHF444wxw7GqfP5MRM4I8fEb5JkYdyHwmpeXw/3+ncRY1+otcqyxiNwNVAEv+lglXJ+F/wf0BnKAvVjNIC3RdPyfvbf4v6VIS/CBlD84vo6IRAPtgMKQRGcdMwYrub9ojHmz7uvGmCPGmKOe5+8AMSLSMVTxGWPyPD/zgX9jXQrX1hJKTEwFvjbG7K/7QrjfP4/9Nc1Wnp/5XtYJ6/soItcC5wNXeb6E6gngs2ALY8x+Y0y1McYN/NPHccP9/kUDlwCv+FonXO9fY0Ragg+k/MF/gJoRC5cBi319wIPN02b3FLDBGPN3H+tk1PQJiMhIrP+DkHwBiUiiiCTXPMfqjFtXZ7X/AD/yjKY5HThcqzkiVHyeOYXz/aul9mfsGuAtL+u8B5wrIu09TRDnepbZTkSmAL8CLjTGlPhYJ5DPgl3x1e7TudjHccNd6uQcYKMxJtfbi+F8/xol3L28jX1gjfLYjNXDfrdn2X1YH2aAeKxL+63AV0CvEMY2DutyfQ2w2vM4D7gJuMmzzkzgW6xRAcuAMSGMr5fnuN94Yqh5/2rHJ1g3atkGrAVGhPj/NxErYbertSxs7x/WF81eoBKrHfh6rD6dD4EtwAdAB8+6I4Ana207w/M53ApcF8L4tmK1X9d8BmtGlXUB3vH3WQhRfM97PltrsJJ257rxeX6v97ceivg8y5+t+czVWjfk719zH1qqQCmlHCrSmmiUUkoFSBO8Uko5lCZ4pZRyKE3wSinlUJrglVLKoTTBq4gjItV1qk4GrdKgiPSoXVnQz3oiVmXLCd5qHYlIfxH5QkTKReSXdV7zWiXRM+b7S8/yVzzjv5VqMk3wKhKVGmNyaj1mh/LgIpKANU56EDAYeNazrLaDwCxgbp1tXVjzDKYCA4HpIjLQ8/JfgQeMMX2AQ1hjxpVqMk3wyjE89bn/5qnR/ZWI9PEs7yEiiz3FrT4UkSzP8nSx6qV/43mM8ezKJSL/FKum///qJm9jTClwM9ZEpuuAmz3Laq+Tb4xZjjWBpjavVRI9VwETsSqggu8qlUoFTBO8ikQJdZporqj12mFjzBDgUeBBz7JHgOeMMadgFd562LP8YeBjYxUuG441IxGgLzDPGDMIKAIurX1wT8KfBzzjeczzcgbvi68qialAkTlR5jdc1SeVg0SHOwClmqDUGJPj47X5tX4+4Hk+GqtwFFjT5P/meT4R+BGAMaYaOOypG7PDGFNzF5+VQI/aBzDGlIrIDOBMz6J5RqeEqxZIE7xyGuPjeWOU13peDdQ7O/ck9CVN2LevKomFWDdXifacxYejiqdyGG2iUU5zRa2fX3ief45VjRDgKuBTz/MPsdrSERGXiLQLQXxeqyR6vjA+wqqACr6rVCoVMD2DV5EoQU6+EfK7xpia4YbtRWQN1ln4dM+yW4BnROQOoACrYxTgVuAJEbke60z9ZqzKgs0mIhnACqAt4Bbr5uEDjTFHRGQmVulgF/C0Maam7f/XwMsi8kdgFVbpaaWaTKtJKscQkZ1Y5Y0PhDsWpVoCbaJRSimH0jN4pZRyKD2DV0oph9IEr5RSDqUJXimlHEoTvFJKOZQmeKWUcqj/D8ddv7S78MAsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 그래프 그리기\n",
    "x = np.arange(len(acc_list))\n",
    "plt.plot(x, acc_list, marker='o',label='RNN')\n",
    "plt.plot(x, acc_list2, marker='s',label='LSTM')\n",
    "plt.legend()\n",
    "plt.xlabel('Epoch * 100')\n",
    "plt.ylabel('Loss')\n",
    "plt.ylim(0, 5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "간단한 Model구현으로서 서로 Loss차이가 거의 없는 것을 확인할 수 있다.  \n",
    "**오히려 RNN이 더욱 좋은 Model로서 Trainning된 것을 확인할 수 있다**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1.6487])\n",
      "tensor([2.7183])\n",
      "tensor([4.4817])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "c = torch.Tensor([0.4])\n",
    "print(c.div(0.8).exp())\n",
    "a = torch.Tensor([0.8])\n",
    "print(a.div(0.8).exp())\n",
    "b = torch.Tensor([1.2])\n",
    "print(b.div(0.8).exp())"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "6.3 간단한 문자 순환 신경망 실습.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}