3.4-classifying-movie-reviews.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.2.4'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 영화 리뷰 분류: 이진 분류 예제\n",
    "\n",
    "이 노트북은 [케라스 창시자에게 배우는 딥러닝](https://tensorflow.blog/케라스-창시자에게-배우는-딥러닝/) 책의 3장 4절의 코드 예제입니다. 책에는 더 많은 내용과 그림이 있습니다. 이 노트북에는 소스 코드에 관련된 설명만 포함합니다. 이 노트북의 설명은 케라스 버전 2.2.2에 맞추어져 있습니다. 케라스 최신 버전이 릴리스되면 노트북을 다시 테스트하기 때문에 설명과 코드의 결과가 조금 다를 수 있습니다.\n",
    "\n",
    "----\n",
    "\n",
    "2종 분류 또는 이진 분류는 아마도 가장 널리 적용된 머신 러닝 문제일 것입니다. 이 예제에서 리뷰 텍스트를 기반으로 영화 리뷰를 긍정과 부정로 분류하는 법을 배우겠습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IMDB 데이터셋\n",
    "\n",
    "인터넷 영화 데이터베이스로부터 가져온 양극단의 리뷰 50,000개로 이루어진 IMDB 데이터셋을 사용하겠습니다. 이 데이터셋은 훈련 데이터 25,000개와 테스트 데이터 25,000개로 나뉘어 있고 각각 50%는 부정, 50%는 긍정 리뷰로 구성되어 있습니다.\n",
    "\n",
    "왜 훈련 데이터와 테스트 데이터를 나눌까요? 같은 데이터에서 머신 러닝 모델을 훈련하고 테스트해서는 절대 안 되기 때문입니다! 모델이 훈련 데이터에서 잘 작동한다는 것이 처음 만난 데이터에서도 잘 동작한다는 것을 보장하지 않습니다. 중요한 것은 새로운 데이터에 대한 모델의 성능입니다(사실 훈련 데이터의 레이블은 이미 알고 있기 때문에 이를 예측하는 모델은 필요하지 않습니다). 예를 들어 모델이 훈련 샘플과 타깃 사이의 매핑을 모두 외워버릴 수 있습니다. 이런 모델은 처음 만나는 데이터에서 타깃을 예측하는 작업에는 쓸모가 없습니다. 다음 장에서 이에 대해 더 자세히 살펴보겠습니다.\n",
    "\n",
    "MNIST 데이터셋처럼 IMDB 데이터셋도 케라스에 포함되어 있습니다. 이 데이터는 전처리되어 있어 각 리뷰(단어 시퀀스)가 숫자 시퀀스로 변환되어 있습니다. 여기서 각 숫자는 사전에 있는 고유한 단어를 나타냅니다.\n",
    "\n",
    "다음 코드는 데이터셋을 로드합니다(처음 실행하면 17MB 정도의 데이터가 컴퓨터에 다운로드됩니다):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import imdb\n",
    "\n",
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "매개변수 `num_words=10000`은 훈련 데이터에서 가장 자주 나타나는 단어 10,000개만 사용하겠다는 의미입니다. 드물게 나타나는 단어는 무시하겠습니다. 이렇게 하면 적절한 크기의 벡터 데이터를 얻을 수 있습니다.\n",
    "\n",
    "변수 `train_data`와 `test_data`는 리뷰의 목록입니다. 각 리뷰는 단어 인덱스의 리스트입니다(단어 시퀀스가 인코딩된 것입니다). `train_labels`와 `test_labels`는 부정을 나타내는 0과 긍정을 나타내는 1의 리스트입니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 14,\n",
       " 22,\n",
       " 16,\n",
       " 43,\n",
       " 530,\n",
       " 973,\n",
       " 1622,\n",
       " 1385,\n",
       " 65,\n",
       " 458,\n",
       " 4468,\n",
       " 66,\n",
       " 3941,\n",
       " 4,\n",
       " 173,\n",
       " 36,\n",
       " 256,\n",
       " 5,\n",
       " 25,\n",
       " 100,\n",
       " 43,\n",
       " 838,\n",
       " 112,\n",
       " 50,\n",
       " 670,\n",
       " 2,\n",
       " 9,\n",
       " 35,\n",
       " 480,\n",
       " 284,\n",
       " 5,\n",
       " 150,\n",
       " 4,\n",
       " 172,\n",
       " 112,\n",
       " 167,\n",
       " 2,\n",
       " 336,\n",
       " 385,\n",
       " 39,\n",
       " 4,\n",
       " 172,\n",
       " 4536,\n",
       " 1111,\n",
       " 17,\n",
       " 546,\n",
       " 38,\n",
       " 13,\n",
       " 447,\n",
       " 4,\n",
       " 192,\n",
       " 50,\n",
       " 16,\n",
       " 6,\n",
       " 147,\n",
       " 2025,\n",
       " 19,\n",
       " 14,\n",
       " 22,\n",
       " 4,\n",
       " 1920,\n",
       " 4613,\n",
       " 469,\n",
       " 4,\n",
       " 22,\n",
       " 71,\n",
       " 87,\n",
       " 12,\n",
       " 16,\n",
       " 43,\n",
       " 530,\n",
       " 38,\n",
       " 76,\n",
       " 15,\n",
       " 13,\n",
       " 1247,\n",
       " 4,\n",
       " 22,\n",
       " 17,\n",
       " 515,\n",
       " 17,\n",
       " 12,\n",
       " 16,\n",
       " 626,\n",
       " 18,\n",
       " 2,\n",
       " 5,\n",
       " 62,\n",
       " 386,\n",
       " 12,\n",
       " 8,\n",
       " 316,\n",
       " 8,\n",
       " 106,\n",
       " 5,\n",
       " 4,\n",
       " 2223,\n",
       " 5244,\n",
       " 16,\n",
       " 480,\n",
       " 66,\n",
       " 3785,\n",
       " 33,\n",
       " 4,\n",
       " 130,\n",
       " 12,\n",
       " 16,\n",
       " 38,\n",
       " 619,\n",
       " 5,\n",
       " 25,\n",
       " 124,\n",
       " 51,\n",
       " 36,\n",
       " 135,\n",
       " 48,\n",
       " 25,\n",
       " 1415,\n",
       " 33,\n",
       " 6,\n",
       " 22,\n",
       " 12,\n",
       " 215,\n",
       " 28,\n",
       " 77,\n",
       " 52,\n",
       " 5,\n",
       " 14,\n",
       " 407,\n",
       " 16,\n",
       " 82,\n",
       " 2,\n",
       " 8,\n",
       " 4,\n",
       " 107,\n",
       " 117,\n",
       " 5952,\n",
       " 15,\n",
       " 256,\n",
       " 4,\n",
       " 2,\n",
       " 7,\n",
       " 3766,\n",
       " 5,\n",
       " 723,\n",
       " 36,\n",
       " 71,\n",
       " 43,\n",
       " 530,\n",
       " 476,\n",
       " 26,\n",
       " 400,\n",
       " 317,\n",
       " 46,\n",
       " 7,\n",
       " 4,\n",
       " 2,\n",
       " 1029,\n",
       " 13,\n",
       " 104,\n",
       " 88,\n",
       " 4,\n",
       " 381,\n",
       " 15,\n",
       " 297,\n",
       " 98,\n",
       " 32,\n",
       " 2071,\n",
       " 56,\n",
       " 26,\n",
       " 141,\n",
       " 6,\n",
       " 194,\n",
       " 7486,\n",
       " 18,\n",
       " 4,\n",
       " 226,\n",
       " 22,\n",
       " 21,\n",
       " 134,\n",
       " 476,\n",
       " 26,\n",
       " 480,\n",
       " 5,\n",
       " 144,\n",
       " 30,\n",
       " 5535,\n",
       " 18,\n",
       " 51,\n",
       " 36,\n",
       " 28,\n",
       " 224,\n",
       " 92,\n",
       " 25,\n",
       " 104,\n",
       " 4,\n",
       " 226,\n",
       " 65,\n",
       " 16,\n",
       " 38,\n",
       " 1334,\n",
       " 88,\n",
       " 12,\n",
       " 16,\n",
       " 283,\n",
       " 5,\n",
       " 16,\n",
       " 4472,\n",
       " 113,\n",
       " 103,\n",
       " 32,\n",
       " 15,\n",
       " 16,\n",
       " 5345,\n",
       " 19,\n",
       " 178,\n",
       " 32]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_labels[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "가장 자주 등장하는 단어 10,000개로 제한했기 때문에 단어 인덱스는 10,000을 넘지 않습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9999"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max([max(sequence) for sequence in train_data])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "재미 삼아 이 리뷰 데이터 하나를 원래 영어 단어로 어떻게 바꾸는지 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# word_index는 단어와 정수 인덱스를 매핑한 딕셔너리입니다\n",
    "word_index = imdb.get_word_index()\n",
    "# 정수 인덱스와 단어를 매핑하도록 뒤집습니다\n",
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
    "# 리뷰를 디코딩합니다. \n",
    "# 0, 1, 2는 '패딩', '문서 시작', '사전에 없음'을 위한 인덱스이므로 3을 뺍니다\n",
    "decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"? this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert ? is an amazing actor and now the same being director ? father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for ? and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also ? to the two little boy's that played the ? of norman and paul they were just brilliant children are often left out of the ? list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoded_review"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 데이터 준비\n",
    "\n",
    "신경망에 숫자 리스트를 주입할 수는 없습니다. 리스트를 텐서로 바꾸는 두 가지 방법이 있습니다:\n",
    "\n",
    "* 같은 길이가 되도록 리스트에 패딩을 추가하고 `(samples, sequence_length)` 크기의 정수 텐서로 변환합니다. 그다음 이 정수 텐서를 다룰 수 있는 층을 신경망의 첫 번째 층으로 사용합니다(`Embedding` 층을 말하며 나중에 자세히 다루겠습니다).\n",
    "* 리스트를 원-핫 인코딩하여 0과 1의 벡터로 변환합니다. 예를 들면 시퀀스 `[3, 5]`를 인덱스 3과 5의 위치는 1이고 그 외는 모두 0인 10,000차원의 벡터로 각각 변환합니다. 그다음 부동 소수 벡터 데이터를 다룰 수 있는 `Dense` 층을 신경망의 첫 번째 층으로 사용합니다.\n",
    "\n",
    "여기서는 두 번째 방식을 사용하고 이해를 돕기 위해 직접 데이터를 원-핫 벡터로 만들겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    # 크기가 (len(sequences), dimension))이고 모든 원소가 0인 행렬을 만듭니다\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.  # results[i]에서 특정 인덱스의 위치를 1로 만듭니다\n",
    "    return results\n",
    "\n",
    "# 훈련 데이터를 벡터로 변환합니다\n",
    "x_train = vectorize_sequences(train_data)\n",
    "# 테스트 데이터를 벡터로 변환합니다\n",
    "x_test = vectorize_sequences(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 샘플은 다음과 같이 나타납니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 1., 1., ..., 0., 0., 0.])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "레이블은 쉽게 벡터로 바꿀 수 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 레이블을 벡터로 바꿉니다\n",
    "y_train = np.asarray(train_labels).astype('float32')\n",
    "y_test = np.asarray(test_labels).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 신경망에 주입할 데이터가 준비되었습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 신경망 모델 만들기\n",
    "\n",
    "입력 데이터가 벡터이고 레이블은 스칼라(1 또는 0)입니다. 아마 앞으로 볼 수 있는 문제 중에서 가장 간단할 것입니다. 이런 문제에 잘 작동하는 네트워크 종류는 `relu` 활성화 함수를 사용한 완전 연결 층(즉, `Dense(16, activation='relu')`)을 그냥 쌓은 것입니다.\n",
    "\n",
    "`Dense` 층에 전달한 매개변수(16)는 은닉 유닛의 개수입니다. 하나의 은닉 유닛은 층이 나타내는 표현 공간에서 하나의 차원이 됩니다. 2장에서 `relu` 활성화 함수를 사용한 `Dense` 층을 다음과 같은 텐서 연산을 연결하여 구현하였습니다:\n",
    "\n",
    "`output = relu(dot(W, input) + b)`\n",
    "\n",
    "16개의 은닉 유닛이 있다는 것은 가중치 행렬 `W`의 크기가 `(input_dimension, 16)`이라는 뜻입니다. 입력 데이터와 `W`를 점곱하면 입력 데이터가 16 차원으로 표현된 공간으로 투영됩니다(그리고 편향 벡터 `b`를 더하고 `relu` 연산을 적용합니다). 표현 공간의 차원을 '신경망이 내재된 표현을 학습할 때 가질 수 있는 자유도'로 이해할 수 있습니다. 은닉 유닛을 늘리면 (표현 공간을 더 고차원으로 만들면) 신경망이 더욱 복잡한 표현을 학습할 수 있지만 계산 비용이 커지고 원치 않은 패턴을 학습할 수도 있습니다(훈련 데이터에서는 성능이 향상되지만 테스트 데이터에서는 그렇지 않은 패턴입니다).\n",
    "\n",
    "`Dense` 층을 쌓을 때 두 가진 중요한 구조상의 결정이 필요합니다:\n",
    "\n",
    "* 얼마나 많은 층을 사용할 것인가\n",
    "* 각 층에 얼마나 많은 은닉 유닛을 둘 것인가\n",
    "\n",
    "4장에서 이런 결정을 하는 데 도움이 되는 일반적인 원리를 배우겠습니다. 당분간은 저를 믿고 선택한 다음 구조를 따라 주세요.\n",
    "\n",
    "* 16개의 은닉 유닛을 가진 두 개의 은닉층\n",
    "* 현재 리뷰의 감정을 스칼라 값의 예측으로 출력하는 세 번째 층\n",
    "\n",
    "중간에 있는 은닉층은 활성화 함수로 `relu`를 사용하고 마지막 층은 확률(0과 1 사이의 점수로, 어떤 샘플이 타깃 '1'일 가능성이 높다는 것은 그 리뷰가 긍정일 가능성이 높다는 것을 의미합니다)을 출력하기 위해 시그모이드 활성화 함수를 사용합니다. `relu`는 음수를 0으로 만드는 함수입니다. 시그모이드는 임의의 값을 [0, 1] 사이로 압축하므로 출력 값을 확률처럼 해석할 수 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "다음이 이 신경망의 모습입니다:\n",
    "\n",
    "![3-layer network](https://s3.amazonaws.com/book.keras.io/img/ch3/3_layer_network.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "다음은 이 신경망의 케라스 구현입니다. 이전에 보았던 MNIST 예제와 비슷합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "\n",
    "model = models.Sequential()\n",
    "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(16, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "마지막으로 손실 함수와 옵티마이저를 선택해야 합니다. 이진 분류 문제이고 신경망의 출력이 확률이기 때문에(네트워크의 끝에 시그모이드 활성화 함수를 사용한 하나의 유닛으로 된 층을 놓았습니다), `binary_crossentropy` 손실이 적합합니다. 이 함수가 유일한 선택은 아니고 예를 들어 `mean_squared_error`를 사용할 수도 있습니다. 확률을 출력하는 모델을 사용할 때는 크로스엔트로피가 최선의 선택입니다. 크로스엔트로피는 정보 이론 분야에서 온 개념으로 확률 분포 간의 차이를 측정합니다. 여기에서는 원본 분포와 예측 분포 사이를 측정합니다.\n",
    "\n",
    "다음은 `rmsprop` 옵티마이저와 `binary_crossentropy` 손실 함수로 모델을 설정하는 단계입니다. 훈련하는 동안 정확도를 사용해 모니터링하겠습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "케라스에 `rmsprop`, `binary_crossentropy`, `accuracy`가 포함되어 있기 때문에 옵티마이저, 손실 함수, 측정 지표를 문자열로 지정하는 것이 가능합니다. 이따금 옵티마이저의 매개변수를 바꾸거나 자신만의 손실 함수, 측정 함수를 전달해야 할 경우가 있습니다. 전자의 경우에는 옵티마이저 파이썬 클래스를 사용해 객체를 직접 만들어 `optimizer` 매개변수에 전달하면 됩니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import optimizers\n",
    "\n",
    "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "후자의 경우는 `loss`와 `metrics` 매개변수에 함수 객체를 전달하면 됩니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import losses\n",
    "from keras import metrics\n",
    "\n",
    "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n",
    "              loss=losses.binary_crossentropy,\n",
    "              metrics=[metrics.binary_accuracy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 훈련 검증\n",
    "\n",
    "훈련하는 동안 처음 본 데이터에 대한 모델의 정확도를 측정하기 위해서는 원본 훈련 데이터에서 10,000의 샘플을 떼어서 검증 세트를 만들어야 합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_val = x_train[:10000]\n",
    "partial_x_train = x_train[10000:]\n",
    "\n",
    "y_val = y_train[:10000]\n",
    "partial_y_train = y_train[10000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "이제 모델을 512개 샘플씩 미니 배치를 만들어 20번의 에포크 동안 훈련시킵니다(`x_train`과 `y_train` 텐서에 있는 모든 샘플에 대해 20번 반복합니다). 동시에 따로 떼어 놓은 10,000개의 샘플에서 손실과 정확도를 측정할 것입니다. 이렇게 하려면 `validation_data` 매개변수에 검증 데이터를 전달해야 합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 15000 samples, validate on 10000 samples\n",
      "Epoch 1/20\n",
      "15000/15000 [==============================] - 2s 116us/step - loss: 0.5084 - acc: 0.7813 - val_loss: 0.3797 - val_acc: 0.8684\n",
      "Epoch 2/20\n",
      "15000/15000 [==============================] - 1s 70us/step - loss: 0.3004 - acc: 0.9047 - val_loss: 0.3004 - val_acc: 0.8897\n",
      "Epoch 3/20\n",
      "15000/15000 [==============================] - 1s 65us/step - loss: 0.2179 - acc: 0.9285 - val_loss: 0.3085 - val_acc: 0.8711\n",
      "Epoch 4/20\n",
      "15000/15000 [==============================] - 1s 69us/step - loss: 0.1750 - acc: 0.9437 - val_loss: 0.2840 - val_acc: 0.8832\n",
      "Epoch 5/20\n",
      "15000/15000 [==============================] - 1s 67us/step - loss: 0.1427 - acc: 0.9543 - val_loss: 0.2841 - val_acc: 0.8872\n",
      "Epoch 6/20\n",
      "15000/15000 [==============================] - 1s 67us/step - loss: 0.1150 - acc: 0.9650 - val_loss: 0.3166 - val_acc: 0.8772\n",
      "Epoch 7/20\n",
      "15000/15000 [==============================] - 1s 67us/step - loss: 0.0980 - acc: 0.9705 - val_loss: 0.3127 - val_acc: 0.8846\n",
      "Epoch 8/20\n",
      "15000/15000 [==============================] - 1s 66us/step - loss: 0.0807 - acc: 0.9763 - val_loss: 0.3859 - val_acc: 0.8649\n",
      "Epoch 9/20\n",
      "15000/15000 [==============================] - 1s 67us/step - loss: 0.0661 - acc: 0.9821 - val_loss: 0.3635 - val_acc: 0.8782\n",
      "Epoch 10/20\n",
      "15000/15000 [==============================] - 1s 69us/step - loss: 0.0561 - acc: 0.9853 - val_loss: 0.3843 - val_acc: 0.8792\n",
      "Epoch 11/20\n",
      "15000/15000 [==============================] - 1s 68us/step - loss: 0.0439 - acc: 0.9893 - val_loss: 0.4153 - val_acc: 0.8779\n",
      "Epoch 12/20\n",
      "15000/15000 [==============================] - 1s 59us/step - loss: 0.0381 - acc: 0.9921 - val_loss: 0.4525 - val_acc: 0.8690\n",
      "Epoch 13/20\n",
      "15000/15000 [==============================] - 1s 59us/step - loss: 0.0300 - acc: 0.9928 - val_loss: 0.4698 - val_acc: 0.8729\n",
      "Epoch 14/20\n",
      "15000/15000 [==============================] - 1s 66us/step - loss: 0.0247 - acc: 0.9945 - val_loss: 0.5023 - val_acc: 0.8726\n",
      "Epoch 15/20\n",
      "15000/15000 [==============================] - 1s 68us/step - loss: 0.0175 - acc: 0.9979 - val_loss: 0.5342 - val_acc: 0.8693\n",
      "Epoch 16/20\n",
      "15000/15000 [==============================] - 1s 68us/step - loss: 0.0149 - acc: 0.9983 - val_loss: 0.5710 - val_acc: 0.8698\n",
      "Epoch 17/20\n",
      "15000/15000 [==============================] - 1s 69us/step - loss: 0.0151 - acc: 0.9971 - val_loss: 0.6025 - val_acc: 0.8697\n",
      "Epoch 18/20\n",
      "15000/15000 [==============================] - 1s 65us/step - loss: 0.0075 - acc: 0.9996 - val_loss: 0.6782 - val_acc: 0.8633\n",
      "Epoch 19/20\n",
      "15000/15000 [==============================] - 1s 62us/step - loss: 0.0117 - acc: 0.9975 - val_loss: 0.6693 - val_acc: 0.8673\n",
      "Epoch 20/20\n",
      "15000/15000 [==============================] - 1s 68us/step - loss: 0.0041 - acc: 0.9999 - val_loss: 0.6942 - val_acc: 0.8658\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=20,\n",
    "                    batch_size=512,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CPU를 사용해도 에포크마다 2초가 걸리지 않습니다. 전체 훈련은 20초 이상 걸립니다. 에포크가 끝날 때마다 10,000개의 검증 샘플 데이터에서 손실과 정확도를 계산하기 때문에 약간씩 지연됩니다.\n",
    "\n",
    "`model.fit()` 메서드는 `History` 객체를 반환합니다. 이 객체는 훈련하는 동안 발생한 모든 정보를 담고 있는 딕셔너리인 `history` 속성을 가지고 있습니다. 한 번 확인해 보죠:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history_dict = history.history\n",
    "history_dict.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이 딕셔너리는 훈련과 검증하는 동안 모니터링할 측정 지표당 하나씩 모두 네 개의 항목을 담고 있습니다. 맷플롯립을 사용해 훈련과 검증 데이터에 대한 손실과 정확도를 그려 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xt4VNW5x/HvK6LIRUDA4wW5VbQCcjNFrCjeavEGaqmA4F1Rq61WbaVqWw+Vc1CpF5Ra0UqtRJHqQWiF0lqp1rYCARG5iCACRlADCoKgEnjPH2szDmGSTDLZmUny+zzPPJnZs2fvd/ZM9jtrrb3WMndHREQEYK9sByAiIrlDSUFERBKUFEREJEFJQUREEpQUREQkQUlBREQSlBSkSplZPTPbYmZtqnLdbDKzw82syq/dNrPTzGxV0uNlZnZCOutWYl+Pm9ltlX19Gdu9y8x+X9XblezZO9sBSHaZ2Zakhw2BL4Ed0eOr3T2/Ittz9x1A46pety5w9yOrYjtmdiUwzN1PStr2lVWxban9lBTqOHdPnJSjX6JXuvtLpa1vZnu7e3F1xCYi1U/VR1KmqHrgWTN7xsw2A8PM7Dgze93MNprZOjMba2b1o/X3NjM3s3bR44nR8zPMbLOZ/cfM2ld03ej5M8zsHTPbZGYPmdm/zOzSUuJOJ8arzWyFmX1qZmOTXlvPzO43sw1m9i7Qr4zjc4eZTSqxbJyZ3Rfdv9LMlkbv593oV3xp2yo0s5Oi+w3N7KkotsXAMSn2uzLa7mIz6x8tPxp4GDghqppbn3Rs70x6/TXRe99gZi+Y2cHpHJvymNm5UTwbzexlMzsy6bnbzGytmX1mZm8nvdfeZjY/Wv6Rmd2b7v4kBu6um264O8Aq4LQSy+4CvgLOIfyI2A/4FnAsoaTZAXgHuD5af2/AgXbR44nAeiAPqA88C0ysxLoHApuBAdFzNwHbgUtLeS/pxDgVaAq0Az7Z9d6B64HFQGugBfBq+FdJuZ8OwBagUdK2PwbyosfnROsYcAqwDegaPXcasCppW4XASdH9McA/gOZAW2BJiXUvAA6OPpMLoxj+K3ruSuAfJeKcCNwZ3T89irE70AD4DfByOscmxfu/C/h9dP+oKI5Tos/otui41wc6A6uBg6J12wMdovtzgSHR/SbAsdn+X6jLN5UUJB2vufuf3H2nu29z97nuPtvdi919JTAe6FvG659z9wJ33w7kE05GFV33bGCBu0+NnrufkEBSSjPG/3X3Te6+inAC3rWvC4D73b3Q3TcAo8vYz0pgESFZAXwH2OjuBdHzf3L3lR68DPwdSNmYXMIFwF3u/qm7ryb8+k/e72R3Xxd9Jk8TEnpeGtsFGAo87u4L3P0LYATQ18xaJ61T2rEpy2Bgmru/HH1Go4H9Ccm5mJCAOkdVkO9Fxw5Ccu9oZi3cfbO7z07zfUgMlBQkHe8nPzCzb5rZi2b2oZl9BowEWpbx+g+T7m+l7Mbl0tY9JDkOd3fCL+uU0owxrX0RfuGW5WlgSHT/QkIy2xXH2WY228w+MbONhF/pZR2rXQ4uKwYzu9TM3oyqaTYC30xzuxDeX2J77v4Z8ClwaNI6FfnMStvuTsJndKi7LwNuJnwOH0fVkQdFq14GdAKWmdkcMzszzfchMVBSkHSUvBzzUcKv48PdfX/gF4TqkTitI1TnAGBmxu4nsZIyiXEdcFjS4/IumX0WOC36pT2AkCQws/2A54D/JVTtNAP+mmYcH5YWg5l1AB4BrgVaRNt9O2m75V0+u5ZQJbVre00I1VQfpBFXRba7F+Ez+wDA3Se6+/GEqqN6hOOCuy9z98GEKsJfA8+bWYMMY5FKUlKQymgCbAI+N7OjgKurYZ9/Bnqa2TlmtjdwA9AqphgnAzea2aFm1gK4tayV3f0j4DVgArDM3ZdHT+0L7AMUATvM7Gzg1ArEcJuZNbPQj+P6pOcaE078RYT8eCWhpLDLR0DrXQ3rKTwDXGFmXc1sX8LJ+Z/uXmrJqwIx9zezk6J9/4TQDjTbzI4ys5Oj/W2LbjsIb+AiM2sZlSw2Re9tZ4axSCUpKUhl3AxcQviHf5TwSzlW0Yl3EHAfsAH4BvAGoV9FVcf4CKHu/y1CI+hzabzmaULD8dNJMW8EfgxMITTWDiQkt3T8klBiWQXMAP6QtN2FwFhgTrTON4Hkevi/AcuBj8wsuRpo1+v/QqjGmRK9vg2hnSEj7r6YcMwfISSsfkD/qH1hX+AeQjvQh4SSyR3RS88Ellq4um0MMMjdv8o0HqkcC1WzIjWLmdUjVFcMdPd/ZjsekdpCJQWpMcysn5k1jaogfk64omVOlsMSqVWUFKQm6QOsJFRB9APOdffSqo9EpBJUfSQiIgkqKYiISEKNGxCvZcuW3q5du2yHISJSo8ybN2+9u5d1GTdQA5NCu3btKCgoyHYYIiI1ipmV1zMfUPWRiIgkUVIQEZEEJQUREUmItU3BzPoBDxIGv3rc3UeXeP5+4OToYUPgwGhwrwrZvn07hYWFfPHFF5mGLNWgQYMGtG7dmvr1SxuaR0SyJbakEA1DMI4wvnwhMNfMprn7kl3ruPuPk9b/IdCjMvsqLCykSZMmtGvXjjB4puQqd2fDhg0UFhbSvn378l8gItUqzuqjXsCKaIKRr4BJfD0RSSpDCKM3VtgXX3xBixYtlBBqADOjRYsWKtWJ5Kg4k8Kh7D5JSCGljH9vZm0JY6y/XNmdKSHUHPqsRHJXnEkh1X9+aWNqDCZMw7gj5YbMhptZgZkVFBUVVVmAIiI1wYcfws9/DsuWxb+vOJNCIbvPHNWaMNRxKoMpo+rI3ce7e56757VqVW6HvGq3YcMGunfvTvfu3TnooIM49NBDE4+/+iq9YeEvu+wylpXziY8bN478/Pwy10lXnz59WLBgQZVsS0TisXAhXHYZtG0Lo0bBy5WuS0lfnFcfzSVMxt2eMB3fYML8tbsxsyMJE278J8ZYdpOfD7ffDmvWQJs24WAPzWCKkRYtWiROsHfeeSeNGzfmlltu2W0dd8fd2Wuv1Hl4woQJ5e7nuuuuq3yQIlIj7NwJM2fCfffBSy9Bw4Zw1VVwww3QsWP8+4+tpODuxYQpBGcCS4HJ7r7YzEaaWf+kVYcAk7yahmvNz4fhw2H1anAPf4cPD8ur2ooVK+jSpQvXXHMNPXv2ZN26dQwfPpy8vDw6d+7MyJEjE+vu+uVeXFxMs2bNGDFiBN26deO4447j448/BuCOO+7ggQceSKw/YsQIevXqxZFHHsm///1vAD7//HO+973v0a1bN4YMGUJeXl65JYKJEydy9NFH06VLF2677TYAiouLueiiixLLx44dC8D9999Pp06d6NatG8OGDavyYyZSV23bBo89Bl26wJlnwpIl8L//C++/Dw8/XD0JAWLup+Du04HpJZb9osTjO+OMoaTbb4etW3dftnVrWJ5JaaE0S5YsYcKECfz2t78FYPTo0RxwwAEUFxdz8sknM3DgQDp16rTbazZt2kTfvn0ZPXo0N910E0888QQjRozYY9vuzpw5c5g2bRojR47kL3/5Cw899BAHHXQQzz//PG+++SY9e/YsM77CwkLuuOMOCgoKaNq0Kaeddhp//vOfadWqFevXr+ett94CYOPGjQDcc889rF69mn322SexTEQq76OP4De/Cbf166F7d3jqKbjgAthnn+qPp871aF6zpmLLM/WNb3yDb33rW4nHzzzzDD179qRnz54sXbqUJUuW7PGa/fbbjzPOOAOAY445hlWrVqXc9vnnn7/HOq+99hqDBw8GoFu3bnTu3LnM+GbPns0pp5xCy5YtqV+/PhdeeCGvvvoqhx9+OMuWLeOGG25g5syZNG3aFIDOnTszbNgw8vPz1flM6ozly+Hww8MJe8gQGDkS/vhHWLQIvqzkNE+LFsEVV4Qq7JEj4bjjYNYsmD8fhg3LTkKAGjhKaqbatAlVRqmWx6FRo0aJ+8uXL+fBBx9kzpw5NGvWjGHDhqW8Xn+fpG9DvXr1KC4uTrntfffdd491KloLV9r6LVq0YOHChcyYMYOxY8fy/PPPM378eGbOnMkrr7zC1KlTueuuu1i0aBH16tWr0D5FapIvv4TBg+GTT+CII+D11+HZZ0P1M0C9etChAxx1FHTqFP4edRR885vQpMnu23KHv/41tBf89a+w334hMdxwAxx5ZPW/t1TqXFIYNSq0ISRXITVsGJbH7bPPPqNJkybsv//+rFu3jpkzZ9KvX78q3UefPn2YPHkyJ5xwAm+99VbKkkiy3r1785Of/IQNGzbQtGlTJk2axC233EJRURENGjTg+9//Pu3bt+eaa65hx44dFBYWcsopp9CnTx/y8/PZunUrTUp+80VqkZ/+NPx6nzoV+ketoVu3hstDly7d/TZjBmzf/vVrW7f+OkkcdFBou1y8ONwfNQquvhpatMjO+ypNnUsKu9oNqvLqo3T17NmTTp060aVLFzp06MDxxx9f5fv44Q9/yMUXX0zXrl3p2bMnXbp0SVT9pNK6dWtGjhzJSSedhLtzzjnncNZZZzF//nyuuOIK3B0z4+6776a4uJgLL7yQzZs3s3PnTm699VYlBKnVpk6FsWPDL/n+SZfHNGwIPXqEW7Lt22Hlyj2Txe9+B59/Dt26wZNPwqBBEBX0c06Nm6M5Ly/PS06ys3TpUo466qgsRZRbiouLKS4upkGDBixfvpzTTz+d5cuXs/feuZX/9ZlJrluzJrQhtG8P//53Zidx99CI3LIlZKtDv5nNc/e88tbLrTOFZGzLli2ceuqpFBcX4+48+uijOZcQRHJdcTFceGH4++yzmf+qN4Mc7Hebks4WtUyzZs2YN29etsMQqdF++Uv417/g6afDVUd1SZ27JFVEpCwvvRQ6jV1xRbj8tK5RUhARiXz0UegjcNRRoYG5LlL1kYgIYcyhiy6CTZu+HnOoLlJSEBEB7r4b/vY3GD8+jD9UV6n6qAqcdNJJzJw5c7dlDzzwAD/4wQ/KfF3jxo0BWLt2LQMHDix12yUvwS3pgQceYGtSb7wzzzyzSsYluvPOOxkzZkzG2xHJdf/6V5ivYNAguPLKbEeTXUoKVWDIkCFMmjRpt2WTJk1iSJqtVIcccgjPPfdcpfdfMilMnz6dZs2aVXp7InXJJ5+EBuW2bUMpoa5PDKikUAUGDhzIn//8Z76MRsZatWoVa9eupU+fPol+Az179uToo49m6tSpe7x+1apVdInKq9u2bWPw4MF07dqVQYMGsW3btsR61157bWLY7V/+8pcAjB07lrVr13LyySdz8sknA9CuXTvWr18PwH333UeXLl3o0qVLYtjtVatWcdRRR3HVVVfRuXNnTj/99N32k8qCBQvo3bs3Xbt25bzzzuPTTz9N7L9Tp0507do1MRDfK6+8kphkqEePHmzevLnSx1YkTu7hKqMPPwz9EfbfP9sRZV+ta1O48Uao6gnFuneH6HyaUosWLejVqxd/+ctfGDBgAJMmTWLQoEGYGQ0aNGDKlCnsv//+rF+/nt69e9O/f/9S5yl+5JFHaNiwIQsXLmThwoW7DX09atQoDjjgAHbs2MGpp57KwoUL+dGPfsR9993HrFmzaNmy5W7bmjdvHhMmTGD27Nm4O8ceeyx9+/alefPmLF++nGeeeYbHHnuMCy64gOeff77M+REuvvhiHnroIfr27csvfvEL/vu//5sHHniA0aNH895777HvvvsmqqzGjBnDuHHjOP7449myZQsNGjSowNEWqT4PPwwvvBAGqMsrt69v3aCSQhVJrkJKrjpyd2677Ta6du3KaaedxgcffMBHH31U6nZeffXVxMm5a9eudO3aNfHc5MmT6dmzJz169GDx4sXlDnb32muvcd5559GoUSMaN27M+eefzz//+U8A2rdvT/fu3YGyh+eGML/Dxo0b6du3LwCXXHIJr776aiLGoUOHMnHixETP6eOPP56bbrqJsWPHsnHjRvWolpw0fz7ccgucfXb4MSlBrftvLesXfZzOPfdcbrrpJubPn8+2bdsSv/Dz8/MpKipi3rx51K9fn3bt2qUcLjtZqlLEe++9x5gxY5g7dy7Nmzfn0ksvLXc7ZY1rtW9Sv/169eqVW31UmhdffJFXX32VadOm8atf/YrFixczYsQIzjrrLKZPn07v3r156aWX+OY3v1mp7YvEYfPm0KjcqhVMmKB2hGQqKVSRxo0bc9JJJ3H55Zfv1sC8adMmDjzwQOrXr8+sWbNYnWoyhyQnnngi+dHcoIsWLWLhwoVAGHa7UaNGNG3alI8++ogZM2YkXtOkSZOU9fYnnngiL7zwAlu3buXzzz9nypQpnHDCCRV+b02bNqV58+aJUsZTTz1F37592blzJ++//z4nn3wy99xzDxs3bmTLli28++67HH300dx6663k5eXx9ttvV3ifInFxh2uvDaOZPvNMGKROvlbrSgrZNGTIEM4///zdrkQaOnQo55xzDnl5eXTv3r3cX8zXXnstl112GV27dqV79+706tULCLOo9ejRg86dO+8x7Pbw4cM544wzOPjgg5k1a1Ziec+ePbn00ksT27jyyivp0aNHmVVFpXnyySe55ppr2Lp1Kx06dGDChAns2LGDYcOGsWnTJtydH//4xzRr1oyf//znzJo1i3r16tGpU6fELHIiueD3vw/zGowcCZX4jVTraehsyQp9ZpINS5eGBuVjjw0d1erSpIHpDp0da/WRmfUzs2VmtsLM9px5PqxzgZktMbPFZvZ0nPGISN21bRtccAE0agQTJ9athFARsVUfmVk9YBzwHaAQmGtm09x9SdI6HYGfAce7+6dmdmBc8YhI3eUOP/4xLFoUpsw85JBsR5S74iwp9AJWuPtKd/8KmAQMKLHOVcA4d/8UwN0/ruzOalo1WF2mz0qq04IFcNpp8Oij8JOfQBVPi17rxJkUDgXeT3pcGC1LdgRwhJn9y8xeN7OUH5eZDTezAjMrKCoq2uP5Bg0asGHDBp1sagB3Z8OGDerQJrFbty6MY9SzZ0gMDz4Y5kmQssV59VGqK39LnrX3BjoCJwGtgX+aWRd33200N3cfD4yH0NBccqOtW7emsLCQVAlDck+DBg1o3bp1tsOQWmrr1tBDefRo+OqrUG10xx3QvHm2I6sZ4kwKhcBhSY9bA2tTrPO6u28H3jOzZYQkMbciO6pfvz7t27fPJFYRqeF27gz9DkaMgMJCOP/8MBx2XZtOM1NxVh/NBTqaWXsz2wcYDEwrsc4LwMkAZtaSUJ20MsaYRKQWeu016N07zJp24IHwj3/A888rIVRGbEnB3YuB64GZwFJgsrsvNrORZtY/Wm0msMHMlgCzgJ+4+4a4YhKR2mXlSvj+90MntA8+gCefhLlzIRqmSyqhVnReE5G6ZdMmGDUqNB7vvTfceivcfHPogyCppdt5TcNciEiNUVwcJsL55S9hwwa45JKQHNTvoOpoQDwRqRFmzICuXeG668IcygUFYYRTJYSqpaQgIjmtqCgMc33mmbB9e5gU5+WXQ/8DqXpKCiKSk9xh0iTo1Ckkgl/9ChYvhgEDNP9BnNSmICI5Z926MOfB1KnQqxc88QR07pztqOoGlRREJGe4h8tKO3WCmTPh3nvhX/9SQqhOKimISE54/324+urQoHz88aF0cMQR2Y6q7lFJQUSyyh0eeyyUBl55JfQ9ePVVJYRsUUlBRLLmvffgqqvg73+Hk0+Gxx+HDh2yHVXdppKCiFS7nTvhoYdCf4M5c+C3v4WXXlJCyAUqKYhItVq+HC6/PAxi993vhh7KbdpkOyrZRSUFEakWO3bAr38deiUvWhR6I8+YoYSQa1RSEJHYzZsXhqeYPRv694dHHtHwFLlKJQURic3atXDppZCXF4a5zs8PvZOVEHKXSgoiUuW2bQtVRaNHh/GKfvpTuO02aNo025FJeZQURKTKuMOzz4b5DdasCVNi3nMPfOMb2Y5M0qXqIxGpEnPmQJ8+MGQIHHAAzJoVpsRUQqhZ6kRSyM+Hdu1gr73C3/z8bEckUnt88AFcfDEceyy8+27ogFZQACedlO3IpDJqffVRfj4MHw5bt4bHq1eHxwBDh2YvLpGabutWGDMG7r47XG76s5+FW5Mm2Y5MMlHrSwq33/51Qthl69awXEQqzh2efhqOPDJMi3nWWbB0KfzP/ygh1AaxJgUz62dmy8xshZmNSPH8pWZWZGYLotuVVR3DmjUVWy4ipXv9dfj2t0Mp+8ADw8B1kydD+/bZjkyqSmxJwczqAeOAM4BOwBAz65Ri1WfdvXt0e7yq4yitt6R6UYqkr7AQhg2D444LVbATJsDcuXDCCdmOTKpanCWFXsAKd1/p7l8Bk4ABMe4vpVGjoGHD3Zc1bBiWi0jZtm2Du+4KVUXPPReqXd95J3RI26vWVz7XTXF+rIcC7yc9LoyWlfQ9M1toZs+Z2WGpNmRmw82swMwKioqKKhTE0KFhwK22bcO8rm3bhsdqZBYpnXu4nLRTJ/j5z+GMM+Dtt0OCaNw429FJnOJMCqmm1vYSj/8EtHP3rsBLwJOpNuTu4909z93zWrVqVeFAhg6FVavCcL2rVikhiJTlrbfg1FNh4MDQcPzyy6GU0K5dtiOT6hBnUigEkn/5twbWJq/g7hvc/cvo4WPAMTHGIyJl2LAhDFrXvTu8+Sb85jcwf36Y/EbqjjiTwlygo5m1N7N9gMHAtOQVzOzgpIf9gaUxxiMiKRQXw8MPQ8eO8Oij8IMfhDkPrr0W9q71PZmkpNg+cncvNrPrgZlAPeAJd19sZiOBAnefBvzIzPoDxcAnwKVxxSMie3r5ZbjhhjC/wSmnhPmRu3TJdlSSTeZespo/t+Xl5XlBQUG2wxCp0d57D26+GaZMCX0Mfv1rOPfccDGG1E5mNs/d88pbTxeVidQhn38Od9wBRx0FM2eGS7OXLIHzzlNCkEA1hiJ1wJdfwh//CCNGhAHshg4NYxYdmuoicanTlBREaqmNG2H6dJg6NcyFvHkzHHNMGJbi29/OdnSSq5QURGqRNWtCEpg6FV55JVxZdOCBMGhQaDM44wz1RJayKSmI1GDusGDB14lgwYKw/MgjQ0PygAFhngMlAkmXkoJIDbN9eygFTJ0K06aF0oFZqBK65x7o3z8kBZHKUFIQqQF27AiXj06ZAi++CJs2QYMG8J3vhDkNzj47VBOJZEpJQSTHvf02XHZZmMugZctw+eiAASEhNGqU7eiktlFSEMlRO3bA/feHfgUNG8If/gAXXgj16mU7MqnNlBREclBy6WDAAPjtb+Ggg7IdldQFuiZBJIfs2AFjxoSRSt95B/LzQzuCEoJUF5UURHKESgeSC1RSEMkylQ4kl6ikIJJFKh1IrlFJQSQLduyAe+/9unTw9NMqHUhuUElBpJollw7OPRceeUTJQHKHSgoi1SRV6eD//k8JQXKLSgoi1WD+fLjuOpUOJPeppCASozVr4OKLwzwGK1aodCC5L9akYGb9zGyZma0wsxFlrDfQzNzMyp0/VKQm+OwzuO22MFrp5MlhxrMVK2DIEE17KbkttuojM6sHjAO+AxQCc81smrsvKbFeE+BHwOy4YhGpLtu3w2OPwZ13QlERXHQR3HUXtGmT7chE0hNnSaEXsMLdV7r7V8AkYECK9X4F3AN8EWMsIrFyD3MbHH10aDvo3BkKCsIgdkoIUpPEmRQOBd5PelwYLUswsx7AYe7+5xjjEIlVQQGcfHLofAYhObz8cmhHEKlp4kwKqWpOPfGk2V7A/cDN5W7IbLiZFZhZQVFRURWGKFJ5q1fDsGHwrW/BkiXwm9/AW2/BOeeo3UBqrjiTQiFwWNLj1sDapMdNgC7AP8xsFdAbmJaqsdndx7t7nrvntWrVKsaQRcq3aVNoOD7ySHj++dCgvGIFXHst1K+f7ehEMhNnUpgLdDSz9ma2DzAYmLbrSXff5O4t3b2du7cDXgf6u3tBjDFJDeUO774b/mbL9u0wbhwcfniYC3nQoNAJbdQo2H//7MUlUpViSwruXgxcD8wElgKT3X2xmY00s/5x7Vdqp1Gjwsm4U6cwG9knn1TfvouKYOxY6NIFrr8+NCYXFMCTT8Jhh5X/epGaxDybP70qIS8vzwsKKl6YKCyEZ5+Fm8ttwZBc89JLcPrpcNpp4fr/2bNh333h+9+H4cOhT5+qr8P/6it48cVw4n/xRSguhry8cKnpmWeqzUBqHjOb5+7l9gWrMz2an3wSbrkldCSSmuODD8K8xJ06hVFEX38dFiyAK64IV/mceGK4/POBBzIvPbiHEsAPfwiHHALnnx8S0I03hgbkuXPhrLOUEKR2S6ukYGbfAArd/UszOwnoCvzB3TfGHN8eKltS2L4dTjgBli0L/+CtW8cQnFSp7dvhlFPgjTfCCfmoo3Z//vPPQ+lv/PjdSw9XXw3HH5/+yXvtWpg4MfxwWLIkbOfcc+GSS+A734G9NUKY1AJVXVJ4HthhZocDvwPaA09nEF+1q18//ONv3x7+2XfuzHZEUp7bboPXXoPHH98zIQA0agSXXx5KD2+8Ee5PnRqSf5cu8OCD8Omnqbe9bRs88wz06xfaBW69FZo1g0cfhQ8/hEmT4IwzlBCkDnL3cm/A/OjvT4AfRvffSOe1VX075phjPBOPP+4O7mPGZLQZidmUKeFz+sEPKva6LVvCZ9yrV3h9gwbuF1/s/tpr7jt3uv/zn+5XXum+//7h+TZt3O+4w/2dd+J5HyK5AijwNM6x6VYfzQYeAG4HznH398xskbt3iStZlaay1Ue7uIe64unTYc4c6NatCoOTKrFyJfTsCR07hpLCvvtWbjsLFoSqpYkTYfNmaNo09DFo1AgGDgwlxr59Ya8607ImdVm61UfpJoVOwDXAf9z9GTNrDwxy99GZh1oxmSYFgPXrw2WFLVqEuur99qui4CRjX3wB3/42rFoV5iBo1y7zbW7ZEtoeZs0KVzGdfz40bpz5dkVqkipNCiU23JwwXtHCygaXiapICgAzZ4b65B/9KNQ9S264+urw6/5Pf4Kzz852NCK1R5U2NJvZP8xsfzM7AHgTmGBm92UaZDZ997shIYwdGxKEZN9TT4WEMGKEEoJItqRbm9rU3T8AHJpUAAATYUlEQVQDzgcmuPsxwGnxhVU9Ro8O17hfemmoUpLsWbwYrrkm1PH/6lfZjkak7ko3KextZgcDFwC1Zpjr/faD/PzQ6emqq7I7rk5dtnkzfO970KRJuExUl4GKZE+6SWEkYQyjd919rpl1AJbHF1b16dYtjKvzwgvwxBPZjqbucQ9DVSxfHvoGHHxwtiMSqdvqzNhHZdm5M4yrM2dOuIzx8MOrdPNSht/8JsxU9j//Az/7WbajEam9qrqhubWZTTGzj83sIzN73sxqzUARe+0VhjioXz9MmrJ9e7YjqhvmzAnjCp15ZuhRLCLZl2710QTCXAiHEKbU/FO0rNY47LAwxMHs2WGidYnXJ5/ABReEgeeeekodyERyRbr/iq3cfYK7F0e33wO1bgq0Cy6Aiy4KSeE//8l2NLXXzp1w8cVhILrJk+GAA7IdkYjskm5SWG9mw8ysXnQbBmyIM7BsefhhaNMmVCNt3pztaGqnu+8OcxTcfz/06pXtaEQkWbpJ4XLC5agfAuuAgcBlcQWVTfvvH6ozVq2CG27IdjS1zz/+AXfcAYMHww9+kO1oRKSktK4Id/c1wG5TaJrZjYRB8mqdPn3ClTCjRoVJVb73vWxHlH2ffRZmQGvYMFT37Lo1bQr16qW3jXXrQjLo2DH0XNZkNSK5p9KXpJrZGndvU8XxlCuOS1JT2b49DMy2ciUsXAiHHhr7LnPSzp3w+9+HJPnxx3s+bxbmITjgAGjefPeEkXxr3hzuuy9ccTRnTpjvQESqT7qXpGbSd7RW/86rXz/0du7RIwyDMXNmZlfIuIcTbLq/qnPBv/8dxoeaNw9694annw4lhU8+Kfu2cmX4u3HjnpMZPfmkEoJILsskKZRbxDCzfsCDQD3g8ZJDbZvZNcB1wA5gCzDc3ZdkEFOVOuKI0Bh69dXQsmU4ybVpE6qVhg5N/ZqNG0Pv3HfeCX+T73/5ZWjAvuGG3D4xfvBB6DeQnx8uGZ04McyTXNHqnp07Q7XTrmSx335hrCkRyV1lJgUz20zqk78BZc5CYGb1gHHAd4BCYK6ZTStx0n/a3X8brd8fuA/ol3748WvYMPy63zWt4+rVYZyk1atD3XjJk3/ywHpmIYkccURIIl98EU60jz8e5v698cYwfHeuXKP/xRfw61+H3sU7dsDtt4cRSys798Bee4WqpWbNoEOHqo1VROJRZlJw9yYZbLsXsMLdVwKY2SRgAJBICtHIq7s0Io3SR3W7445wgky2bVs4Ye5yyCEhQZx3XvjbsWNIBB06QIMGu7/2nntCI+vDD4dG7COPDCWHiy8OM4JlgztMmQI33xyuujr/fLj3Xp3IReqi2MY+MrOBQD93vzJ6fBFwrLtfX2K964CbgH2AU9x9j4H2zGw4MBygTZs2x6xevTqWmFPZa6/SR099440wTlJlfkl/9RU891yoniooCA2xw4eHcYAOOyyzmCvirbdCieXll0OV1gMPwKmnVt/+RaR6VOnYR5WNIcWyPU6v7j7O3b8B3ArckWpD7j7e3fPcPa9Vq+rtSN2mlOur2raF7t0rX7Wyzz6hnn7OnDAP8amnhl/n7duHyzZnz658zOnYsAGuvz68hzfeCCWXN95QQhCp6+JMCoVA8m/e1sDaMtafBJwbYzyVMmpUaFdI1rBhWF4VzOD44+GPfwxX7dx4I8yYEa72Oe64MAxEcXHV7AvCth5+OFRxPfIIXHttaA+57jrNYyAi8SaFuUBHM2tvZvsAgwmD6iWYWcekh2eRg3M0DB0a2gDatg0n8LZtw+PSrj7KRNu2MGYMFBaGaUKLimDQoFC3f8894df9V1+F25dfhobhbdvCbetW+PzzMEn95s3h9tlnsGlTuCJq48bQ+axHD/jhD8PfBQtCgmjRourfi4jUTLHOp2BmZxJ6PdcDnnD3UWY2Eihw92lm9iBhWs/twKfA9e6+uKxtVlfntVywYwdMnx7aHWbNqppttm8frjA691z1KBapS9JtU9AkOzXEm2+GQeR27gwn810n9F33k2+plkNozB48eM8rokSk9quOHs1Sjbp1CzcRkTjlSLcpERHJBUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKRQDfLzoV072Guv8Dc/P9sRiYikpvkUYpafD8OHh+kyAVavDo8hnik9RUQyoZJCzG6//euEsMvWrWG5iEiuUVKI2Zo1FVsuIpJNSgoxa9OmYstFRLIp1qRgZv3MbJmZrTCzESmev8nMlpjZQjP7u5m1jTOebBg1Cho23H1Zw4ZhuYhIroktKZhZPWAccAbQCRhiZp1KrPYGkOfuXYHngHviiidbhg6F8eOhbVswC3/Hj1cjs4jkpjivPuoFrHD3lQBmNgkYACzZtYK7z0pa/3VgWIzxZM3QoUoCIlIzxFl9dCjwftLjwmhZaa4AZqR6wsyGm1mBmRUUFRVVYYgiIpIszqRgKZZ5yhXNhgF5wL2pnnf38e6e5+55rVq1qsIQRUQkWZzVR4XAYUmPWwNrS65kZqcBtwN93f3LGOMREZFyxFlSmAt0NLP2ZrYPMBiYlryCmfUAHgX6u/vHMcYiIiJpiC0puHsxcD0wE1gKTHb3xWY20sz6R6vdCzQG/mhmC8xsWimbq9M0dpKIVJdYxz5y9+nA9BLLfpF0/7Q4918baOwkEalO6tGc4zR2kohUJyWFHKexk0SkOikp5DiNnSQi1UlJIcdp7CQRqU5KCjlOYyeJSHXSzGs1gMZOEpHqopKCiIgkKCnUAer8JiLpUvVRLafObyJSESop1HLq/CYiFaGkUMup85uIVISSQi2nzm8iUhFKCrWcOr+JSEUoKdRy6vwmIhWhq4/qAHV+E5F0qaQg5VI/B5G6QyUFKZP6OYjULSopSJnUz0GkblFSkDKpn4NI3RJrUjCzfma2zMxWmNmIFM+faGbzzazYzAbGGYtUjvo5iNQtsSUFM6sHjAPOADoBQ8ysU4nV1gCXAk/HFYdkpir6OaihWqTmiLOk0AtY4e4r3f0rYBIwIHkFd1/l7guBnTHGIRnItJ/Drobq1avB/euGaiUGkdwUZ1I4FHg/6XFhtKzCzGy4mRWYWUFRUVGVBCfpGzoUVq2CnTvD34pcdaSGapGaJc6kYCmWeWU25O7j3T3P3fNatWqVYVhSndRQLVKzxJkUCoHDkh63BtbGuD/JQWqoFqlZ4kwKc4GOZtbezPYBBgPTYtyf5CA1VIvULLElBXcvBq4HZgJLgcnuvtjMRppZfwAz+5aZFQLfBx41s8VxxSPZoYZqkZrF3CtVzZ81eXl5XlBQkO0wpJq0axcSQUlt24ZGbxFJj5nNc/e88tZTj2bJaWqoFqleSgqS06qioVptEiLpU1KQnJZpQ7XaJEQqRklBclqmDdXqPCdSMUoKkvMy6VFdFW0Sqn6SukRJQWq1TNskVP0kdY2SgtRqmbZJqPpJ6holBanVMm2TUPWT1DWao1lqvaFDKz+fdJs2qTvPVbT6SXNcS02hkoJIGXKl+kmlDakuSgoiZciV6ic1dkt1UVIQKUcml8RWRY/sqihtqKQh6VJSEIlRVQwdnmlpQyUNqQglBZEYZVr9BJmXNlTSkIpQUhCJWSbVT5B5aUMlDakIJQWRHJdpaUMlDakIJQWRGiCT0kZtKGlkmlSUlCrA3WvU7ZhjjnERqZiJE93btnU3C38nTkz/tW3buofT+e63tm2r5/UTJ7o3bLj7axs2TP89ZPr6qpDJ8a8qQIGncY7N+km+ojclBZHqlelJ1Sx1UjBL7/XZTkrumZ3UcyEpuaefFFR9JCJlynabRqbVV9mu/qpxbTLpZI7K3oB+wDJgBTAixfP7As9Gz88G2pW3TZUURGqWTH8pZ7ukkOnrMy0pVVVJg2yXFMysHjAOOAPoBAwxs04lVrsC+NTdDwfuB+6OKx4RyY5MSxqZNpRnu6E9F67+qpB0MkdlbsBxwMykxz8DflZinZnAcdH9vYH1gJW1XZUUROqeTBtqs9nQnu02mV3IdkkBOBR4P+lxYbQs5TruXgxsAlqU3JCZDTezAjMrKCoqiilcEclVmXYAzOYlvdluk6moOJOCpVjmlVgHdx/v7nnunteqVasqCU5EJB1VMVRJNpNSRcU5yU4hcFjS49bA2lLWKTSzvYGmwCcxxiQiUmGZTNRUFfuG0IawZk0oIYwaFV88cSaFuUBHM2sPfAAMBi4ssc404BLgP8BA4OWo7ktERCLVmZRiSwruXmxm1xMak+sBT7j7YjMbSWjwmAb8DnjKzFYQSgiD44pHRETKF+scze4+HZheYtkvku5/AXw/zhhERCR96tEsIiIJSgoiIpKgpCAiIglW0y72MbMiYHW24yhFS0Kv7Fyl+DKT6/FB7seo+DKTSXxt3b3cjl41LinkMjMrcPe8bMdRGsWXmVyPD3I/RsWXmeqIT9VHIiKSoKQgIiIJSgpVa3y2AyiH4stMrscHuR+j4stM7PGpTUFERBJUUhARkQQlBRERSVBSqCAzO8zMZpnZUjNbbGY3pFjnJDPbZGYLotsvUm0rxhhXmdlb0b4LUjxvZjbWzFaY2UIz61mNsR2ZdFwWmNlnZnZjiXWq/fiZ2RNm9rGZLUpadoCZ/c3Mlkd/m5fy2kuidZab2SXVFNu9ZvZ29PlNMbNmpby2zO9CzDHeaWYfJH2OZ5by2n5mtiz6Po6oxvieTYptlZktKOW1sR7D0s4pWfv+pTM9m267TSF6MNAzut8EeAfoVGKdk4A/ZzHGVUDLMp4/E5hBmOSoNzA7S3HWAz4kdKrJ6vEDTgR6AouSlt0DjIjujwDuTvG6A4CV0d/m0f3m1RDb6cDe0f27U8WWznch5hjvBG5J4zvwLtAB2Ad4s+T/U1zxlXj+18AvsnEMSzunZOv7p5JCBbn7OnefH93fDCxlz2lGc90A4A8evA40M7ODsxDHqcC77p71Huru/ip7TvA0AHgyuv8kcG6Kl34X+Ju7f+LunwJ/A/rFHZu7/9XDFLYArxMmscqaUo5fOnoBK9x9pbt/BUwiHPcqVVZ8ZmbABcAzVb3fdJRxTsnK909JIQNm1g7oAcxO8fRxZvammc0ws87VGliY0vSvZjbPzIaneD6d+bOrw2BK/0fM5vHb5b/cfR2Ef1zgwBTr5MKxvJxQ8kulvO9C3K6PqrieKKX6IxeO3wnAR+6+vJTnq+0YljinZOX7p6RQSWbWGHgeuNHdPyvx9HxClUg34CHghWoO73h37wmcAVxnZieWeD6tubHjZGb7AP2BP6Z4OtvHryKyeizN7HagGMgvZZXyvgtxegT4BtAdWEeooikp699FYAhllxKq5RiWc04p9WUplmV0/JQUKsHM6hM+vHx3/7+Sz7v7Z+6+Jbo/HahvZi2rKz53Xxv9/RiYQiiiJ0tn/uy4nQHMd/ePSj6R7eOX5KNd1WrR349TrJO1Yxk1Kp4NDPWogrmkNL4LsXH3j9x9h7vvBB4rZd9Z/S5amBv+fODZ0tapjmNYyjklK98/JYUKiuoffwcsdff7SlnnoGg9zKwX4ThvqKb4GplZk133CQ2Si0qsNg24OLoKqTewaVcxtRqV+ussm8evhF1ziBP9nZpinZnA6WbWPKoeOT1aFisz6wfcCvR3962lrJPOdyHOGJPbqc4rZd+Judyj0uNgwnGvLqcBb7t7Yaonq+MYlnFOyc73L64W9dp6A/oQimcLgQXR7UzgGuCaaJ3rgcWEKyleB75djfF1iPb7ZhTD7dHy5PgMGEe46uMtIK+aj2FDwkm+adKyrB4/QoJaB2wn/Pq6AmgB/B1YHv09IFo3D3g86bWXAyui22XVFNsKQl3yru/gb6N1DwGml/VdqMbj91T0/VpIOMEdXDLG6PGZhCtu3o0rxlTxRct/v+t7l7RutR7DMs4pWfn+aZgLERFJUPWRiIgkKCmIiEiCkoKIiCQoKYiISIKSgoiIJCgpiETMbIftPoJrlY3YaWbtkkfoFMlVe2c7AJEcss3du2c7CJFsUklBpBzRePp3m9mc6HZ4tLytmf09GvDt72bWJlr+XxbmOHgzun072lQ9M3ssGjP/r2a2X7T+j8xsSbSdSVl6myKAkoJIsv1KVB8NSnruM3fvBTwMPBAte5gwBHlXwoB0Y6PlY4FXPAzo15PQExagIzDO3TsDG4HvRctHAD2i7VwT15sTSYd6NItEzGyLuzdOsXwVcIq7r4wGLvvQ3VuY2XrC0A3bo+Xr3L2lmRUBrd39y6RttCOMe98xenwrUN/d7zKzvwBbCKPBvuDRYIAi2aCSgkh6vJT7pa2TypdJ93fwdZveWYSxqI4B5kUjd4pkhZKCSHoGJf39T3T/34RRPQGGAq9F9/8OXAtgZvXMbP/SNmpmewGHufss4KdAM2CP0opIddEvEpGv7We7T97+F3ffdVnqvmY2m/BDaki07EfAE2b2E6AIuCxafgMw3syuIJQIriWM0JlKPWCimTUljF57v7tvrLJ3JFJBalMQKUfUppDn7uuzHYtI3FR9JCIiCSopiIhIgkoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikvD/IYufLjtGRKgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "# ‘bo’는 파란색 점을 의미합니다\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "# ‘b’는 파란색 실선을 의미합니다\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8FeXZ//HPFUQCsgfcQBaXWhUBMQJW3C2CWnFrFbFV0VKtaG3r0weXFkvdHuterRWr1iVK+dVioVUUEUWLKEFJEFxARI0gRmQHgcD1++OewCGc5BxytoR836/XvM6ZmXtmrjM5mevMfc/cY+6OiIhITfJyHYCIiNR9ShYiIpKQkoWIiCSkZCEiIgkpWYiISEJKFiIikpCShSTNzBqZ2Woz65TOsrlkZvubWdqvHzezk8xsYcz4h2Z2dDJla7Gtv5rZdbVdXiQZu+Q6AMkcM1sdM9oMWA9sisZ/5u5FO7I+d98ENE932YbA3Q9Mx3rM7FLgAnc/Lmbdl6Zj3SI1UbLYibn7loN19Mv1Und/ubryZraLu1dkIzaRRPR9rFtUDdWAmdlNZvZ3M3vGzFYBF5jZkWY23cyWm9liM7vPzBpH5XcxMzezLtH4U9H8F8xslZm9aWZdd7RsNH+gmX1kZivM7E9m9l8zu6iauJOJ8WdmNt/MlpnZfTHLNjKzu81sqZl9DAyoYf/cYGZjqkx7wMzuit5fambvR5/n4+hXf3XrKjOz46L3zczsySi2OcDhcba7IFrvHDM7PZp+KHA/cHRUxfd1zL69MWb5y6LPvtTMnjOzvZLZNzuynyvjMbOXzewbM/vSzH4Ts53fRvtkpZkVm9ne8ar8zOyNyr9ztD+nRtv5BrjBzA4wsynRZ/k62m+tYpbvHH3G8mj+vWaWH8V8UEy5vcxsrZkVVPd5JQF319AABmAhcFKVaTcBG4AfEH44NAWOAPoQzjr3BT4ChkfldwEc6BKNPwV8DRQCjYG/A0/VouzuwCpgUDTvV8BG4KJqPksyMf4LaAV0Ab6p/OzAcGAO0BEoAKaGf4O429kXWA3sFrPur4DCaPwHURkDTgDWAd2jeScBC2PWVQYcF72/A3gVaAN0BuZWKfsjYK/ob3J+FMMe0bxLgVerxPkUcGP0vn8UY08gH/gz8Eoy+2YH93MrYAnwC6AJ0BLoHc27FigBDog+Q0+gLbB/1X0NvFH5d44+WwVwOdCI8H38DnAisGv0PfkvcEfM53kv2p+7ReWPiuaNBm6O2c6vgXG5/j+sz0POA9CQpT909cnilQTLXQP8v+h9vATwl5iypwPv1aLsUOD1mHkGLKaaZJFkjH1j5v8TuCZ6P5VQHVc575SqB7Aq654OnB+9Hwh8VEPZfwNXRO9rShafxf4tgJ/Hlo2z3veAU6P3iZLF48AtMfNaEtqpOibaNzu4n38MFFdT7uPKeKtMTyZZLEgQwznAjOj90cCXQKM45Y4CPgEsGp8FnJXu/6uGNKgaSj6PHTGz75rZf6JqhZXAKKBdDct/GfN+LTU3aldXdu/YODz8d5dVt5IkY0xqW8CnNcQL8DQwOHp/PrDlogAzO83M3oqqYZYTftXXtK8q7VVTDGZ2kZmVRFUpy4HvJrleCJ9vy/rcfSWwDOgQUyapv1mC/bwPML+aGPYhJIzaqPp93NPMxprZF1EMf6sSw0IPF1Nsw93/SzhL6Wdm3YBOwH9qGZOgNgsJvzRjPUT4Jbu/u7cEfkf4pZ9Jiwm/fAEwM2Pbg1tVqcS4mHCQqZTo0t6/AyeZWUdCNdnTUYxNgX8AtxKqiFoDLyUZx5fVxWBm+wIPEqpiCqL1fhCz3kSX+S4iVG1Vrq8FobrriyTiqqqm/fw5sF81y1U3b00UU7OYaXtWKVP18/0f4Sq+Q6MYLqoSQ2cza1RNHE8AFxDOgsa6+/pqykkSlCykqhbACmBN1ED4syxs899ALzP7gZntQqgHb5+hGMcCV5tZh6ix839rKuzuSwhVJY8BH7r7vGhWE0I9ejmwycxOI9StJxvDdWbW2sJ9KMNj5jUnHDDLCXnzUsKZRaUlQMfYhuYqngEuMbPuZtaEkMxed/dqz9RqUNN+Hg90MrPhZrarmbU0s97RvL8CN5nZfhb0NLO2hCT5JeFCikZmNoyYxFZDDGuAFWa2D6EqrNKbwFLgFgsXDTQ1s6Ni5j9JqLY6n5A4JAVKFlLVr4ELCQ3ODxF+WWdUdEA+F7iL8M+/H/Au4RdlumN8EJgMzAZmEM4OEnma0AbxdEzMy4FfAuMIjcTnEJJeMkYSznAWAi8QcyBz91LgPuDtqMx3gbdilp0EzAOWmFlsdVLl8hMJ1UXjouU7AUOSjKuqavezu68Avg+cTWhQ/wg4Npr9R+A5wn5eSWhszo+qF38KXEe42GH/Kp8tnpFAb0LSGg88GxNDBXAacBDhLOMzwt+hcv5Cwt95g7tP28HPLlVUNv6I1BlRtcIi4Bx3fz3X8Uj9ZWZPEBrNb8x1LPWdbsqTOsHMBhCqFb4lXHpZQfh1LVIrUfvPIODQXMeyM1A1lNQV/YAFhOqJAcAZapCU2jKzWwn3etzi7p/lOp6dgaqhREQkIZ1ZiIhIQjtNm0W7du28S5cuuQ5DRKRemTlz5tfuXtOl6sBOlCy6dOlCcXFxrsMQEalXzCxRLwaAqqFERCQJShYiIpKQkoWIiCSkZCEiIgkpWYiISEIZSxZm9qiZfWVm71Uz36LHJ843s1Iz6xUz70IzmxcNF2YqRhGRVBQVQZcukJcXXouKEi1Rf7efyUtn/0Z4XnB1XQMPJDx28QDCoxsfBPpEXRmPJDx+04GZZjbe3ZdlMFYRkR1SVATDhsHatWH800/DOMCQ2vbzW4e3n7EzC3efSui6uTqDgCc8mA60tvBg+ZOBSe7+TZQgJhH6ChKRnUyqv4xz+cv++uu3HqgrrV0bpicrlfjTsf0dkcs2iw5s+wjFsmhaddO3Y2bDzKzYzIrLy8szFqiIpF/lL+NPPwX3rb+Mkz1gprp85Tpqe7D+rJruCaubHm/bqcSf6vZ3VC6TRbzHT3oN07ef6D7a3QvdvbB9+4R3q4tIHZLqL+NUl0/1YN2pmgfyVje9qlTjT3X7OyqXyaKMbZ9D3JHwwJvqpotIHZPLX+apLp/qwfrmm6FZs22nNWsWpicj1fhT3f6OymWyGA/8JLoqqi+wwt0XAy8C/c2sjZm1AfpH00SkDsn1L/NUl0/1YD1kCIweDZ07g1l4HT06+cblVONPdfs7zN0zMhAeHL8Y2Eg4W7gEuAy4LJpvwAPAx4Tn5BbGLDsUmB8NFyezvcMPP9xFGpqnnnLv3NndLLw+9VT2lu/c2T2kiW2Hzp2T33azZtsu26xZ8jGkunyq8acq1fjTBSj2ZI7pyRSqD4OShTQ0uT7YmsU/2Jrt2GfIVbKrCwfrVD9/OiSbLHaaJ+UVFha6uiiXhqRLl1D1U1XnzrBwYd1fvi4oKgptFJ99Fqp/br45O/dI1CVmNtPdCxOVU3cfIjlUnxuIs93AmglDhoTEtnlzeG1oiWJHKFmI5Eh9byDOegOr5JSShUiO5PrSzXScGeiXecOhZCGSglxWI6X6y15nBrIj1MAtUktVO3KD8Ms82QPuztBALPWfGrhFMizX1Ugi2aRkIQ1afa5GEsmmTD7PQqROS/V5AJ06xa9G2pGO3IYMUXKQ+kFnFtJgqRpJJHlKFtJgqRpJJHmqhpIGS9VIIsnTmYU0WKpGEkmekoXUa6lczaRqJJHkqRpK6q1Ur2aqLKfkIJKYziyk3kr1aiYRSZ6ShdRbqV7NJCLJU7KQeivVLrZFJHlKFpJTqTRQ62omkexRspCcSfXhP7qaSSR71EW55Iy66BbJPXVRLnWeGqhF6g8lC8kZNVCL1B9KFpISNVCLNAxKFlJraqAWaTjUwC21pgZqkfpPDdyScWqgFmk4lCyk1tRALdJwKFlIramBWqThULKQWlMDtUjDoedZSEr0PAiRhkFnFiIikpCSRQOXyk11ItJwqBqqAUvHY0lFpGHQmUUDpseSikiylCwaMN1UJyLJymiyMLMBZvahmc03sxFx5nc2s8lmVmpmr5pZx5h5m8xsVjSMz2ScDZVuqhORZGUsWZhZI+ABYCBwMDDYzA6uUuwO4Al37w6MAm6NmbfO3XtGw+mZirMh0011IpKsTJ5Z9Abmu/sCd98AjAEGVSlzMDA5ej8lznzJIN1UJyLJymSy6AB8HjNeFk2LVQKcHb0/E2hhZgXReL6ZFZvZdDM7I94GzGxYVKa4vLw8nbE3GEOGhB5iN28Or0oUIhJPJpOFxZlWtT/0a4Bjzexd4FjgC6Aimtcp6jb3fOAeM9tvu5W5j3b3QncvbN++fRpDFxGRWJm8z6IM2CdmvCOwKLaAuy8CzgIws+bA2e6+ImYe7r7AzF4FDgM+zmC8IiJSjUyeWcwADjCzrma2K3AesM1VTWbWzswqY7gWeDSa3sbMmlSWAY4C5mYw1npLd2CLSDZk7MzC3SvMbDjwItAIeNTd55jZKKDY3ccDxwG3mpkDU4ErosUPAh4ys82EhHabuytZVKE7sEUkW/RY1XpMjzUVkVTpsaoNgO7AFpFsUbKox3QHtohki5JFPaY7sEUkW5Qs6jHdgS0i2aLnWdRzeqypiGSDzixERCQhJQsREUlIyUJERBJSshARkYSULEREJCElCxERSUjJIsfUa6yI1Ae6zyKH1GusiNQXOrPIoeuv35ooKq1dG6aLiNQlShY5pF5jRaS+ULLIIfUaKyL1hZJFDqnXWBGpL5Qscki9xopIfaGroXJMvcaKSH2gMwsREUlIyUJERBJSshARkYSULEREJCElCxERSUjJQkREElKyEBGRhJQsREQkISULERFJKGGyMLPhZtYmG8GIiEjdlMyZxZ7ADDMba2YDzMwyHZSIiNQtCZOFu98AHAA8AlwEzDOzW8xsvwzHJiIidURSbRbu7sCX0VABtAH+YWa3ZzA2ERGpI5Jps7jKzGYCtwP/BQ5198uBw4GzMxxfnVdUBF26QF5eeC0qynVEIiLpl0wX5e2As9z909iJ7r7ZzE7LTFj1Q1ERDBu29Tnan34axkHdjovIziWZaqjngW8qR8yshZn1AXD39zMVWH1w/fVbE0WltWvDdBGRnUkyyeJBYHXM+JpoWoP32Wc7Nl1EpL5KJllY1MANhOonknzCXnSp7YdmNt/MRsSZ39nMJptZqZm9amYdY+ZdaGbzouHCZLaXbZ067dh0EZH6KplksSBq5G4cDb8AFiRayMwaAQ8AA4GDgcFmdnCVYncAT7h7d2AUcGu0bFtgJNAH6A2MrIs3Bt58MzRrtu20Zs3CdBGRnUkyyeIy4HvAF0AZ4QA+LInlegPz3X2Bu28AxgCDqpQ5GJgcvZ8SM/9kYJK7f+Puy4BJwIAktplVQ4bA6NHQuTOYhdfRo9W4LSI7n4TVSe7+FXBeLdbdAfg8Zrwy0cQqIVx+ey9wJtDCzAqqWbZDLWLIuCFDlBxEZOeXMFmYWT5wCXAIkF853d2HJlo0zjSvMn4NcL+ZXQRMJZy9VCS5LGY2jOgsp5MaCkREMiaZaqgnCf1DnQy8BnQEViWxXBmwT8x4R2BRbAF3X+TuZ7n7YcD10bQVySwblR3t7oXuXti+ffskQhIRkdpIJlns7+6/Bda4++PAqcChSSw3AzjAzLqa2a6EqqzxsQXMrJ2ZVcZwLfBo9P5FoL+ZtYkatvtH00REJAeSSRYbo9flZtYNaAV0SbSQu1cAwwkH+feBse4+x8xGmdnpUbHjgA/N7CNgD+DmaNlvgD8QEs4MYFQ0TUREcsBibqGIX8DsUuBZwtnE34DmwG/d/aGMR7cDCgsLvbi4ONdhiIjUK2Y2090LE5WrsYE7qiJaGV2+OhXYN03xiYhIPVJjNVR0t/bwLMUiIiJ1VDJtFpPM7Boz28fM2lYOGY9MRETqjGSSxVDgCkI11MxoUONAZMkSOPlkGDMm15GIiGROMndwd81GIPXR+vVw1lkwbRpMngy77hrGRUR2Nsncwf2TeNPd/Yn0h1N/uIcHHU2bBo8+Cg8/DOedB+PHw4A614tVfGVl8Oab8M47cNxx4QxJRCSeZLoaPyLmfT5wIvAO0KCTxR//CE88ASNHwsUXw5lnwvHHh9eJE+HYY3Md4bY2boSSkpDcKofPo963zOC220KSu/NOOLhq38Ai0uAlvM9iuwXMWgFPuvvpCQtnUTbvs5gwAQYNgnPOCW0VeVHLT3l5SBKffx6qpXr3zko4cX39dThrqEwMM2bAunVhXqdO8L3vwZFHhteDDgq95Y4aBatWhTOm3/8estWDyrx50LEjNG2ane2JyFbJ3mdRm2TRGCh194NqG1wmZCtZzJ4dDrAHHghTp27/PIsvvoBjjoFly+DVV6F794yHxObNMHfutmcN8+aFeY0bw2GHhZgrE0THjvHXs3RpSBJ//jPsthvccANcdRU0aZL+mNevh3/8Ax54ICS1o4+Gl16C/PzEy4pI+qQtWZjZBLb2+JpHeAbFWHff7sl3uZSNZFFeHs4W1q8Pv9Q7VNNp+sKF4eC3YUNIKAcemLmYJkyAyy6DRVE3i+3bb00M3/seHH74jv9i/+ADuOYa+M9/oGtXuP12OPvsUF2Vqs8+g4ceCm085eWw//6h+uv++8PFAWPHQqNGqW9HRJKTbLLA3WscgGNjhqOAjomWycVw+OGHeyZ9+617v37u+fnub7+duPwHH7jvvrt7hw7uCxakP57ly90vusgd3Hv0cH/8cfd589w3b07fNl56yf3QQ8M2+vVL7nPHs3mz+6RJ7mec4Z6XF4bTT3efONF906ZQ5u67w3Z+/vP0fgYRqRlQ7EkcY5NJFl2B/JjxpkCXZFaezSGTyWLzZveLLw5765lnkl+upMS9TRv3ffd1LytLXzyTJ7t36hQOutdf775+ffrWXVVFhftDD4XEB+4//rH7558nt+zy5e733ut+4IFh2Xbt3EeMcP/kk/jl/+d/Qrmbbkpb+CKSQDqTRTGwa8z4rsCMZFaezSGTyeKOO8Ke+u1vd3zZt95yb97c/bvfdf/qq9TiWLPG/corQyzf+Y779OmprW9HrFjhfu217k2auDdt6v6737mvWhW/bEmJ+7Bh7s2ahVj79HF/4gn3detq3samTSEZgfsjj6T/M6SqoiIkwE8/dZ892/2NN9yff959zBj3f/7T/bXX3N97z33x4swmcJF0SjZZJNNmMcvde1aZVuLuPZKuFMuCTLVZPP88nHba1vr0vGTuea9i6tRQL3/ggTBlCrRuvePrmD4dLrwQPvooNDrfeuv2jevZsHAhjBgBf/877LUX3HIL/OQnUFEB//xnaLB+443QUD14MFxxRWg3SdbGjfCDH8DLL8Nzz4V9n0nuMG5cuHBhxQpYuTIMle9jX1ev3rF1N28OBQXbDm3bbj+toAC6dcvN31MknQ3ck4A/ufv4aHwQcJW7n5iWSNMkE8lizpxw9dD++8Prr4crhGrrxRfDQfDww2HSpHAgScaGDeEKpdtuC1cxPfYYnHBC7eNIlzffhF/+Et56Cw49NDRWf/kl7LsvXH55uPekoKB26169OtyzMmcOvPIK9O2b3tgrrVwJP/vZ1q5amjeHli3D0KpV/Nd401q0CH+npUu3Dt98s+147LRly0KSirX77vCb34R9p6Qh2ZTOBu79gOnAZ9EwjfD0vJxXPcUO6a6GKi9379rVfY893D/7LD3r/Oc/3Rs1cj/+ePe1axOXLykJjdcQ2kyWL09PHOmyaZP700+7d+vmfsop7v/5z9YG61QtWeK+337ubdu6v/9+etYZ69133Q84ILT73HKL+8aN6d9GdSoq3L/+2v2jj9zffNP92Wfdv//98HfefXf3O+8MVY4i2UC62iy2FAwPPWqRbPlsD+lMFuvXux9zTKifT3e7wJNPupu5n3pq9fXaGze633qre+PGIVmNH5/eGOqL+fPDwbNTJ/cvvkjPOjdvdv/LX8Lfdu+93adOTc960+H1191PPDH8V+65Z7hCLJkfFSKpSFuyAG4BWseMtwFuSmbl2RzSlSw2b3a/9NKwZ4qK0rLK7Tz0UFj/D3+4/S/ajz5y79s3zD/nnHCG05DNnBkuEDj0UPdly1Jb18qV7oMHh3178smpX3CQKVOnup9wwtakcc89ShqSOckmi2TaLN5198OqTHvH3XvtQLVYxqWrzeLee+Hqq+G66+Dmm9MQWDXuvht+9avQaP3oo2Han/8c6q3z80ND8XnnpedGuPpu0iQ49dRwk+HEibW7y7u0FH74Q5g/H/7wh9BIX5uLFbLptddCe9WUKeFighEjQlcs6bjL3T1cLDF9emh3WrYMdtkl3PG/yy7Vv69ufocO0KNHeNV3tn5JZ5tFKdAkZrwpMCeZTJTNIR1nFi+8EOqwzzgjfXXvNRk1Kvx6HDp0a/XDgAHpq3LZmRQVbT3bqqhIfrnNm91Hjw43U+61V7i8tb6ZMsX92GPD5997b/f77kt8GXJVy5eHmyx//3v3gQPD/T8hZbi3ahUuxd53X/d99gn7qX1799atw1ldfn5oa6ssX9NQUBC+y7/6VbhcurTUfcOGTOwVSRfSWA31G+AN4JJoeAP4TTIrz+aQarKYO9e9ZcvQoFzd/QPptnnz1hvRdtstVE/p7uXq3Xln2FfDhye3n1audD///LBM//6h0bw+e+UV96OPDp+nQwf3+++PnzQ2bQr3gTz8sPsll7gffHBoJ4Pw2q1bqGp95BH3OXOS/2G0eXOoNl27NuzbpUvDPi0rC/ec3H9/WO8RR4QEU5lAdt3V/bDDwkUa99zj/uqrqVcpSvokmyyS6kjQzAYAJwEGLAP2cvcrdvh8J4NSqYZauhT69Ak9rs6YEXplzRb3cM9Cnz6hHyap2TXXhG7Ub701VMtUJ7baadQouPbaul/tlAz3UC01cmS4n6VDh1Bl2qVLuJx5+nR4++1wWTCE+zr69g2XgPftG/o2a9ky83FWVIRqrlmzwlBSAu++Gy6xrtS5M/TsGYaCgtDnWuzw7bfJT9u8OVR/5eXFf61pXl4efOc7cMEF4bkuDa1vsrT2OmtmPYHzgR8BnwDPuvv9KUeZRrVNFhs3Qv/+oafWV18N/1RSd23eDD/+MTz9dLjn5KKLtp3vDo88AldeCW3awDPP1L1ni6SDe7gHZeRI+O9/w7S8vNDLcWViqLxHqK60IbiHe3Eqk0dlIvnoo+3vO2nSJAz5+Vvf1zSel7e1Mmzz5ppfq07btAmKi0OC7dABzj8/JI5s9BhdF6ScLMzsO8B5wGBgKfB34Bp375zOQNOltsni449Dl+K33RYOQlL3bdgQ7ux+5ZXwZMJTTgnTV68ON7U99RScdFJ43WOP3Maaae7hh87GjVBYmPzNnnXJ2rWwZs3Wg3/jxtlPcOvWwb//DU8+CS+8EM6MuncPSeP886vvYToVK1aEiwvefDO8b98+3Jy5++5b37dvH/6mmdwf6UgWm4HXgUvcfX40bYG775vWSNMklWqoVavCXbhSf6xaFaoMPvggJI1mzUK107x54Qqia69teNUJkh7l5aFrnyefDAdzs9BrwgUXhK76a3OscA9VopXPm3nzTXjvvTA9Ly88RmDNmvjL5udvm0Divd9nn9CTQm2kI1mcSTiz+B4wERgD/NXd62TNejaflCd1w5Il4XLaZcvCL8PWrUO103HH5Toy2VnMmxfOUJ96ChYsCAf1M84IieP73w9nQfGsXRuqtioTw7Rp4emVELqJ6dt36zNnKtuR1q0Lieqrr7YOseNV561fv3V7vXuHxFYb6ewbajfgDEJ11AnA48A4d3+pdqFlhpJFwzR/fqhG7NYt/BLc2audJDfcw0H/qafCBSnffBN+0Q8eHBLHnntue9bw7ruhKgtCB6KVjzCufIxxqhdbuIdq18okYhYukqmNjDxW1czaAj8EznX3OtCd3VZKFg1XRUW4MUwkGzZsCO0aTz4ZnlS5YcPWeU2bhoN2ZXLo2xfatctdrMnI2DO46yolCxHJtmXLQhf3a9aE5NC9e/VVU3VVsslCv8dERGqpTRsYOjTXUWTHTnCbkoiIZJqShYiIJKRkISIiCSlZiIhIQkoWIiKSkJKFiIgkpGQhIiIJZTRZmNkAM/vQzOab2XZPHzCzTmY2xczeNbNSMzslmt7FzNaZ2axo+Esm4xQRkZpl7KY8M2sEPAB8HygDZpjZeHefG1PsBmCsuz9oZgcDzwNdonkfu3vPTMUnIiLJy+SZRW9gvrsvcPcNhF5rB1Up40Dlc7taAYsyGI+IiNRSJpNFB+DzmPGyaFqsG4ELzKyMcFZxZcy8rlH11GtmdnS8DZjZMDMrNrPi8tjnNYqISFplMlnEe7ZT1V4LBwN/c/eOwCnAk2aWBywGOrn7YcCvgKfNbLsnB7v7aHcvdPfC9u3bpzl8ERGplMlkUQbsEzPeke2rmS4BxgK4+5tAPtDO3de7+9Jo+kzgY+A7GYxVRERqkMlkMQM4wMy6mtmuhKfuja9S5jPgRAAzO4iQLMrNrH3UQI6Z7QscACzIYKwiIlKDjF0N5e4VZjYceBFoBDzq7nPMbBRQ7O7jgV8DD5vZLwlVVBe5u5vZMcAoM6sANgGXufs3mYpVRERqpocfiYg0YMk+/Eh3cIuISEJKFiIikpCShYiIJKRkISIiCSlZiIhIQkoWIiKSkJKFiIgkpGQhIiIJKVmIiEhCShYiIpKQkoWIiCSkZCEiIgkpWYiISEJKFiIikpCShYiIJKRkISIiCSlZiIhIQkoWIiKSkJKFiIgkpGQhIiIJKVmIiEhCShYiIpKQkoWIiCSkZCEiIgkpWYiISEJKFiIikpCShYiIJKRkISIiCe2S6wBEpP7buHEjZWVlfPvtt7kORaqRn59Px44dady4ca2WV7IQkZSVlZXRokULunTpgpnlOhypwt1ZunQpZWVldO3atVbrUDWUiKTs22+/paCgQImijjIzCgoKUjrzU7IQkbRQoqjbUv37KFmIiEhCShYiknVFRdClC+TlhdeiotTWt3TpUnr27EnPnj3Zc8896dChw5bxDRs2JLWOiy++mA9N50i3AAARP0lEQVQ//LDGMg888ABFqQZbT6mBW0SyqqgIhg2DtWvD+KefhnGAIUNqt86CggJmzZoFwI033kjz5s255pprtinj7rg7eXnxfyM/9thjCbdzxRVX1C7AnYDOLEQkq66/fmuiqLR2bZiebvPnz6dbt25cdtll9OrVi8WLFzNs2DAKCws55JBDGDVq1Jay/fr1Y9asWVRUVNC6dWtGjBhBjx49OPLII/nqq68AuOGGG7jnnnu2lB8xYgS9e/fmwAMPZNq0aQCsWbOGs88+mx49ejB48GAKCwu3JLJYI0eO5IgjjtgSn7sD8NFHH3HCCSfQo0cPevXqxcKFCwG45ZZbOPTQQ+nRowfXZ2JnJZDRZGFmA8zsQzObb2Yj4szvZGZTzOxdMys1s1Ni5l0bLfehmZ2cyThFJHs++2zHpqdq7ty5XHLJJbz77rt06NCB2267jeLiYkpKSpg0aRJz587dbpkVK1Zw7LHHUlJSwpFHHsmjjz4ad93uzttvv80f//jHLYnnT3/6E3vuuSclJSWMGDGCd999N+6yv/jFL5gxYwazZ89mxYoVTJw4EYDBgwfzy1/+kpKSEqZNm8buu+/OhAkTeOGFF3j77bcpKSnh17/+dZr2TvIylizMrBHwADAQOBgYbGYHVyl2AzDW3Q8DzgP+HC17cDR+CDAA+HO0PhGp5zp12rHpqdpvv/044ogjtow/88wz9OrVi169evH+++/HTRZNmzZl4MCBABx++OFbft1XddZZZ21X5o033uC8884DoEePHhxyyCFxl508eTK9e/emR48evPbaa8yZM4dly5bx9ddf84Mf/AAIN9I1a9aMl19+maFDh9K0aVMA2rZtu+M7IkWZPLPoDcx39wXuvgEYAwyqUsaBltH7VsCi6P0gYIy7r3f3T4D50fpEpJ67+WZo1mzbac2ahemZsNtuu215P2/ePO69915eeeUVSktLGTBgQNx7D3bdddct7xs1akRFRUXcdTdp0mS7MpXVSTVZu3Ytw4cPZ9y4cZSWljJ06NAtccS7xNXdc35pciaTRQfg85jxsmharBuBC8ysDHgeuHIHlsXMhplZsZkVl5eXpytuEcmgIUNg9Gjo3BnMwuvo0bVv3N4RK1eupEWLFrRs2ZLFixfz4osvpn0b/fr1Y+zYsQDMnj077pnLunXryMvLo127dqxatYpnn30WgDZt2tCuXTsmTJgAhJsd165dS//+/XnkkUdYt24dAN98803a404kk8kiXhqsmnIHA39z947AKcCTZpaX5LK4+2h3L3T3wvbt26ccsIhkx5AhsHAhbN4cXrORKAB69erFwQcfTLdu3fjpT3/KUUcdlfZtXHnllXzxxRd0796dO++8k27dutGqVattyhQUFHDhhRfSrVs3zjzzTPr06bNlXlFREXfeeSfdu3enX79+lJeXc9pppzFgwAAKCwvp2bMnd999d9rjTsSSOWWq1YrNjgRudPeTo/FrAdz91pgyc4AB7v55NL4A6AtcElvWzF6M1vVmddsrLCz04uLijHwWEanZ+++/z0EHHZTrMOqEiooKKioqyM/PZ968efTv35958+axyy65v1Mh3t/JzGa6e2GiZTMZ/QzgADPrCnxBaLA+v0qZz4ATgb+Z2UFAPlAOjAeeNrO7gL2BA4C3MxiriEharF69mhNPPJGKigrcnYceeqhOJIpUZewTuHuFmQ0HXgQaAY+6+xwzGwUUu/t44NfAw2b2S0I100UeTnXmmNlYYC5QAVzh7psyFauISLq0bt2amTNn5jqMtMtounP35wkN17HTfhfzfi4Qt9LQ3W8GMnR9hIiI7AjdwS0iIgkpWYiISEJKFiIikpCShYjUe8cdd9x2N9jdc889/PznP69xuebNmwOwaNEizjnnnGrXneiy/HvuuYe1Mb0jnnLKKSxfvjyZ0OsNJQsRqfcGDx7MmDFjtpk2ZswYBg8enNTye++9N//4xz9qvf2qyeL555+ndevWtV5fXVT/L/4VkTrl6qshTo/cKenZE6KeweM655xzuOGGG1i/fj1NmjRh4cKFLFq0iH79+rF69WoGDRrEsmXL2LhxIzfddBODBm3bTd3ChQs57bTTeO+991i3bh0XX3wxc+fO5aCDDtrSxQbA5ZdfzowZM1i3bh3nnHMOv//977nvvvtYtGgRxx9/PO3atWPKlCl06dKF4uJi2rVrx1133bWl19pLL72Uq6++moULFzJw4ED69evHtGnT6NChA//617+2dBRYacKECdx0001s2LCBgoICioqK2GOPPVi9ejVXXnklxcXFmBkjR47k7LPPZuLEiVx33XVs2rSJdu3aMXny5LT9DZQsRKTeKygooHfv3kycOJFBgwYxZswYzj33XMyM/Px8xo0bR8uWLfn666/p27cvp59+erUd8z344IM0a9aM0tJSSktL6dWr15Z5N998M23btmXTpk2ceOKJlJaWctVVV3HXXXcxZcoU2rVrt826Zs6cyWOPPcZbb72Fu9OnTx+OPfZY2rRpw7x583jmmWd4+OGH+dGPfsSzzz7LBRdcsM3y/fr1Y/r06ZgZf/3rX7n99tu58847+cMf/kCrVq2YPXs2AMuWLaO8vJyf/vSnTJ06la5du6a9/yglCxFJq5rOADKpsiqqMllU/pp3d6677jqmTp1KXl4eX3zxBUuWLGHPPfeMu56pU6dy1VVXAdC9e3e6d+++Zd7YsWMZPXo0FRUVLF68mLlz524zv6o33niDM888c0vPt2eddRavv/46p59+Ol27dqVnz55A9d2gl5WVce6557J48WI2bNhA165dAXj55Ze3qXZr06YNEyZM4JhjjtlSJt3dmDf4Not0PwtYRHLjjDPOYPLkybzzzjusW7duyxlBUVER5eXlzJw5k1mzZrHHHnvE7ZY8Vryzjk8++YQ77riDyZMnU1payqmnnppwPTX1vVfZvTlU3w36lVdeyfDhw5k9ezYPPfTQlu3F67I8092YN+hkUfks4E8/BfetzwJWwhCpf5o3b85xxx3H0KFDt2nYXrFiBbvvvjuNGzdmypQpfPrppzWu55hjjqEoOgi89957lJaWAqF78912241WrVqxZMkSXnjhhS3LtGjRglWrVsVd13PPPcfatWtZs2YN48aN4+ijj076M61YsYIOHcLTGR5//PEt0/v378/999+/ZXzZsmUceeSRvPbaa3zyySdA+rsxb9DJIpvPAhaRzBs8eDAlJSVbnlQHMGTIEIqLiyksLKSoqIjvfve7Na7j8ssvZ/Xq1XTv3p3bb7+d3r3Dc9d69OjBYYcdxiGHHMLQoUO36d582LBhDBw4kOOPP36bdfXq1YuLLrqI3r1706dPHy699FIOO+ywpD/PjTfeyA9/+EOOPvrobdpDbrjhBpYtW0a3bt3o0aMHU6ZMoX379owePZqzzjqLHj16cO655ya9nWRkrIvybKtNF+V5eeGMoiqz0M++iCRHXZTXD6l0Ud6gzyyy/SxgEZH6qkEni2w/C1hEpL5q0Mkil88CFtnZ7CxV2jurVP8+Df4+iyFDlBxEUpWfn8/SpUspKCjI6OWbUjvuztKlS8nPz6/1Ohp8shCR1HXs2JGysjLKy8tzHYpUIz8/n44dO9Z6eSULEUlZ48aNt9w5LDunBt1mISIiyVGyEBGRhJQsREQkoZ3mDm4zKwdq7vQlt9oBX+c6iBoovtQovtQovtSkEl9nd2+fqNBOkyzqOjMrTuaW+lxRfKlRfKlRfKnJRnyqhhIRkYSULEREJCEli+wZnesAElB8qVF8qVF8qcl4fGqzEBGRhHRmISIiCSlZiIhIQkoWaWJm+5jZFDN738zmmNkv4pQ5zsxWmNmsaPhdDuJcaGazo+1v92hBC+4zs/lmVmpmvbIY24Ex+2aWma00s6urlMnqPjSzR83sKzN7L2ZaWzObZGbzotc21Sx7YVRmnpldmMX4/mhmH0R/v3Fm1rqaZWv8LmQwvhvN7IuYv+Ep1Sw7wMw+jL6LI7IY399jYltoZrOqWTYb+y/ucSUn30F315CGAdgL6BW9bwF8BBxcpcxxwL9zHOdCoF0N808BXgAM6Au8laM4GwFfEm4Yytk+BI4BegHvxUy7HRgRvR8B/F+c5doCC6LXNtH7NlmKrz+wS/T+/+LFl8x3IYPx3Qhck8Tf/2NgX2BXoKTq/1Om4qsy/07gdzncf3GPK7n4DurMIk3cfbG7vxO9XwW8D3TIbVS1Mgh4woPpQGsz2ysHcZwIfOzuOb0r392nAt9UmTwIeDx6/zhwRpxFTwYmufs37r4MmAQMyEZ87v6Su1dEo9OB2vdLnaJq9l8yegPz3X2Bu28AxhD2e1rVFJ+FB3P8CHgm3dtNVg3Hlax/B5UsMsDMugCHAW/FmX2kmZWY2QtmdkhWAwsceMnMZprZsDjzOwCfx4yXkZukdx7V/5Pmeh/u4e6LIfwzA7vHKVNX9uNQwpliPIm+C5k0PKome7SaKpS6sP+OBpa4+7xq5md1/1U5rmT9O6hkkWZm1hx4Frja3VdWmf0OoVqlB/An4Llsxwcc5e69gIHAFWZ2TJX58R5zltXrq81sV+B04P/FmV0X9mEy6sJ+vB6oAIqqKZLou5ApDwL7AT2BxYSqnqpyvv+AwdR8VpG1/ZfguFLtYnGm1XofKlmkkZk1JvxBi9z9n1Xnu/tKd18dvX8eaGxm7bIZo7svil6/AsYRTvdjlQH7xIx3BBZlJ7otBgLvuPuSqjPqwj4EllRWzUWvX8Upk9P9GDVmngYM8agCu6okvgsZ4e5L3H2Tu28GHq5mu7nef7sAZwF/r65MtvZfNceVrH8HlSzSJKrffAR4393vqqbMnlE5zKw3Yf8vzWKMu5lZi8r3hIbQ96oUGw/8JLoqqi+wovJ0N4uq/UWX630YGQ9UXllyIfCvOGVeBPqbWZuomqV/NC3jzGwA8L/A6e6+tpoyyXwXMhVfbBvYmdVsdwZwgJl1jc40zyPs92w5CfjA3cvizczW/qvhuJL972AmW/Ib0gD0I5zilQKzouEU4DLgsqjMcGAO4cqO6cD3shzjvtG2S6I4ro+mx8ZowAOEK1FmA4VZjrEZ4eDfKmZazvYhIWktBjYSfqldAhQAk4F50WvbqGwh8NeYZYcC86Ph4izGN59QV135PfxLVHZv4PmavgtZiu/J6LtVSjjo7VU1vmj8FMLVPx9nM75o+t8qv3MxZXOx/6o7rmT9O6juPkREJCFVQ4mISEJKFiIikpCShYiIJKRkISIiCSlZiIhIQkoWIgmY2SbbtjfctPWAamZdYns8Famrdsl1ACL1wDp375nrIERySWcWIrUUPc/g/8zs7WjYP5re2cwmRx3lTTazTtH0PSw8X6IkGr4XraqRmT0cPa/gJTNrGpW/yszmRusZk6OPKQIoWYgko2mVaqhzY+atdPfewP3APdG0+wndvHcndOJ3XzT9PuA1D50g9iLc+QtwAPCAux8CLAfOjqaPAA6L1nNZpj6cSDJ0B7dIAma22t2bx5m+EDjB3RdEnb196e4FZvY1oQuLjdH0xe7ezszKgY7uvj5mHV0Izxw4IBr/X6Cxu99kZhOB1YSedZ/zqANFkVzQmYVIarya99WViWd9zPtNbG1LPJXQT9fhwMyoJ1SRnFCyEEnNuTGvb0bvpxF6SQUYArwRvZ8MXA5gZo3MrGV1KzWzPGAfd58C/AZoDWx3diOSLfqlIpJYUzObFTM+0d0rL59tYmZvEX54DY6mXQU8amb/A5QDF0fTfwGMNrNLCGcQlxN6PI2nEfCUmbUi9AR8t7svT9snEtlBarMQqaWozaLQ3b/OdSwimaZqKBERSUhnFiIikpDOLEREJCElCxERSUjJQkREElKyEBGRhJQsREQkof8P/tfw5tN8DFEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.clf()   # 그래프를 초기화합니다\n",
    "acc = history_dict['acc']\n",
    "val_acc = history_dict['val_acc']\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "점선은 훈련 손실과 정확도이고 실선은 검증 손실과 정확도입니다. 신경망의 무작위한 초기화 때문에 사람마다 결과거 조금 다를 수 있습니다.\n",
    "\n",
    "여기에서 볼 수 있듯이 훈련 손실이 에포크마다 감소하고 훈련 정확도는 에포크마다 증가합니다. 경사 하강법 최적화를 사용했을 때 반복마다 최소화되는 것이 손실이므로 기대했던 대로입니다. 검증 손실과 정확도는 이와 같지 않습니다. 4번째 에포크에서 그래프가 역전되는 것 같습니다. 이것이 훈련 세트에서 잘 작동하는 모델이 처음 보는 데이터에 잘 작동하지 않을 수 있다고 앞서 언급한 경고의 한 사례입니다. 정확한 용어로 말하면 과대적합되었다고 합니다. 2번째 에포크 이후부터 훈련 데이터에 과도하게 최적화되어 훈련 데이터에 특화된 표현을 학습하므로 훈련 세트 이외의 데이터에는 일반화되지 못합니다.\n",
    "\n",
    "이런 경우에 과대적합을 방지하기 위해서 3번째 에포크 이후에 훈련을 중지할 수 있습니다. 일반적으로 4장에서 보게 될 과대적합을 완화하는 다양한 종류의 기술을 사용할 수 있습니다.\n",
    "\n",
    "처음부터 다시 새로운 신경망을 4번의 에포크 동안만 훈련하고 테스트 데이터에서 평가해 보겠습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/4\n",
      "25000/25000 [==============================] - 1s 47us/step - loss: 0.4749 - acc: 0.8217\n",
      "Epoch 2/4\n",
      "25000/25000 [==============================] - 1s 42us/step - loss: 0.2658 - acc: 0.9097\n",
      "Epoch 3/4\n",
      "25000/25000 [==============================] - 1s 42us/step - loss: 0.1982 - acc: 0.9299\n",
      "Epoch 4/4\n",
      "25000/25000 [==============================] - 1s 42us/step - loss: 0.1679 - acc: 0.9404\n",
      "25000/25000 [==============================] - 1s 45us/step\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(16, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, epochs=4, batch_size=512)\n",
    "results = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.3231545869159698, 0.87348]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "아주 단순한 방식으로도 87%의 정확도를 달성했습니다. 최고 수준의 기법을 사용하면 95%에 가까운 성능을 얻을 수 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 훈련된 모델로 새로운 데이터에 대해 예측하기\n",
    "\n",
    "모델을 훈련시킨 후에 이를 실전 환경에서 사용하고 싶을 것입니다. `predict` 메서드를 사용해서 어떤 리뷰가 긍정일 확률을 예측할 수 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.1402615 ],\n",
       "       [0.9997029 ],\n",
       "       [0.29552558],\n",
       "       ...,\n",
       "       [0.07234979],\n",
       "       [0.04342841],\n",
       "       [0.48153415]], dtype=float32)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "여기에서처럼 이 모델은 어떤 샘플에 대해 확신을 가지고 있지만(0.99 또는 그 이상, 0.01 또는 그 이하) 어떤 샘플에 대해서는 확신이 부족합니다(0.6, 0.4). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 추가 실험\n",
    "\n",
    "* 여기에서는 두 개의 은닉층을 사용했습니다. 한 개 또는 세 개의 은닉층을 사용하고 검증과 테스트 정확도에 어떤 영향을 미치는지 확인해 보세요.\n",
    "* 층의 은닉 유닛을 추가하거나 줄여 보세요: 32개 유닛, 64개 유닛 등\n",
    "* `binary_crossentropy` 대신에 `mse` 손실 함수를 사용해 보세요.\n",
    "* `relu` 대신에 `tanh` 활성화 함수(초창기 신경망에서 인기 있었던 함수입니다)를 사용해 보세요.\n",
    "\n",
    "다음 실험을 진행하면 여기에서 선택한 구조가 향상의 여지는 있지만 어느 정도 납득할 만한 수준이라는 것을 알게 것입니다!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 정리\n",
    "\n",
    "다음은 이 예제에서 배운 것들입니다:\n",
    "\n",
    "* 원본 데이터를 신경망에 텐서로 주입하기 위해서는 꽤 많은 전처리가 필요합니다. 단어 시퀀스는 이진 벡터로 인코딩될 수 있고 다른 인코딩 방식도 있습니다.\n",
    "* `relu` 활성화 함수와 함께 `Dense` 층을 쌓은 네트워크는 (감성 분류를 포함하여) 여러 종류의 문제에 적용할 수 있어서 앞으로 자주 사용하게 될 것입니다.\n",
    "* (출력 클래스가 두 개인) 이진 분류 문제에서 네트워크는 하나의 유닛과 `sigmoid` 활성화 함수를 가진 `Dense` 층으로 끝나야 합니다. 이 신경망의 출력은 확률을 나타내는 0과 1 사이의 스칼라 값입니다.\n",
    "* 이진 분류 문제에서 이런 스칼라 시그모이드 출력에 대해 사용할 손실 함수는 `binary_crossentropy`입니다.\n",
    "* `rmsprop` 옵티마이저는 문제에 상관없이 일반적으로 충분히 좋은 선택입니다. 걱정할 거리가 하나 줄은 셈입니다.\n",
    "* 훈련 데이터에 대해 성능이 향상됨에 따라 신경망은 과대적합되기 시작하고 이전에 본적 없는 데이터에서는 결과가 점점 나빠지게 됩니다. 항상 훈련 세트 이외의 데이터에서 성능을 모니터링해야 합니다."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}