Elmo Embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras sentiment analysis with Elmo Embeddings\n",
    "\n",
    "One of the recent trends in Natural Language Processing is transfer learning. Transfer learning allows NLP models to learn more from fewer examples. In this notebook, we experiment with so-called [ELMo Embeddings](https://allennlp.org/elmo), a new approach to word embeddings that relies on a large unlabelled text corpus to understand word meaning in context. ELMo Embeddings are available from [Tensorflow Hub](https://alpha.tfhub.dev/google/elmo/2)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparation\n",
    "\n",
    "Let's first install and import all the required libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting tensorflow_hub\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/5f/22/64f246ef80e64b1a13b2f463cefa44f397a51c49a303294f5f3d04ac39ac/tensorflow_hub-0.1.1-py2.py3-none-any.whl (52kB)\n",
      "\u001b[K    100% |████████████████████████████████| 61kB 2.6MB/s ta 0:00:011\n",
      "\u001b[?25hRequirement already satisfied: protobuf>=3.4.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow_hub) (3.6.0)\n",
      "Requirement already satisfied: six>=1.10.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow_hub) (1.11.0)\n",
      "Requirement already satisfied: numpy>=1.12.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow_hub) (1.15.1)\n",
      "Requirement already satisfied: setuptools in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from protobuf>=3.4.0->tensorflow_hub) (39.1.0)\n",
      "\u001b[31mdistributed 1.21.8 requires msgpack, which is not installed.\u001b[0m\n",
      "Installing collected packages: tensorflow-hub\n",
      "Successfully installed tensorflow-hub-0.1.1\n",
      "\u001b[33mYou are using pip version 10.0.1, however version 18.0 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install tensorflow_hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras import backend\n",
    "\n",
    "sess = tf.Session()\n",
    "backend.set_session(sess)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using /tmp/tfhub_modules to cache modules.\n",
      "INFO:tensorflow:Downloading TF-Hub Module 'https://alpha.tfhub.dev/google/elmo/2'.\n",
      "INFO:tensorflow:Downloaded TF-Hub Module 'https://alpha.tfhub.dev/google/elmo/2'.\n",
      "INFO:tensorflow:Initialize variable module/aggregation/scaling:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with aggregation/scaling\n",
      "INFO:tensorflow:Initialize variable module/aggregation/weights:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with aggregation/weights\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/W_cnn_0:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/W_cnn_0\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/W_cnn_1:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/W_cnn_1\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/W_cnn_2:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/W_cnn_2\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/W_cnn_3:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/W_cnn_3\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/W_cnn_4:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/W_cnn_4\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/W_cnn_5:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/W_cnn_5\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/W_cnn_6:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/W_cnn_6\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/b_cnn_0:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/b_cnn_0\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/b_cnn_1:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/b_cnn_1\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/b_cnn_2:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/b_cnn_2\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/b_cnn_3:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/b_cnn_3\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/b_cnn_4:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/b_cnn_4\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/b_cnn_5:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/b_cnn_5\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN/b_cnn_6:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN/b_cnn_6\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN_high_0/W_carry:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN_high_0/W_carry\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN_high_0/W_transform:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN_high_0/W_transform\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN_high_0/b_carry:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN_high_0/b_carry\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN_high_0/b_transform:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN_high_0/b_transform\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN_high_1/W_carry:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN_high_1/W_carry\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN_high_1/W_transform:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN_high_1/W_transform\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN_high_1/b_carry:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN_high_1/b_carry\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN_high_1/b_transform:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN_high_1/b_transform\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN_proj/W_proj:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN_proj/W_proj\n",
      "INFO:tensorflow:Initialize variable module/bilm/CNN_proj/b_proj:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/CNN_proj/b_proj\n",
      "INFO:tensorflow:Initialize variable module/bilm/RNN_0/RNN/MultiRNNCell/Cell0/rnn/lstm_cell/bias:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/RNN_0/RNN/MultiRNNCell/Cell0/rnn/lstm_cell/bias\n",
      "INFO:tensorflow:Initialize variable module/bilm/RNN_0/RNN/MultiRNNCell/Cell0/rnn/lstm_cell/kernel:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/RNN_0/RNN/MultiRNNCell/Cell0/rnn/lstm_cell/kernel\n",
      "INFO:tensorflow:Initialize variable module/bilm/RNN_0/RNN/MultiRNNCell/Cell0/rnn/lstm_cell/projection/kernel:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/RNN_0/RNN/MultiRNNCell/Cell0/rnn/lstm_cell/projection/kernel\n",
      "INFO:tensorflow:Initialize variable module/bilm/RNN_0/RNN/MultiRNNCell/Cell1/rnn/lstm_cell/bias:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/RNN_0/RNN/MultiRNNCell/Cell1/rnn/lstm_cell/bias\n",
      "INFO:tensorflow:Initialize variable module/bilm/RNN_0/RNN/MultiRNNCell/Cell1/rnn/lstm_cell/kernel:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/RNN_0/RNN/MultiRNNCell/Cell1/rnn/lstm_cell/kernel\n",
      "INFO:tensorflow:Initialize variable module/bilm/RNN_0/RNN/MultiRNNCell/Cell1/rnn/lstm_cell/projection/kernel:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/RNN_0/RNN/MultiRNNCell/Cell1/rnn/lstm_cell/projection/kernel\n",
      "INFO:tensorflow:Initialize variable module/bilm/RNN_1/RNN/MultiRNNCell/Cell0/rnn/lstm_cell/bias:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/RNN_1/RNN/MultiRNNCell/Cell0/rnn/lstm_cell/bias\n",
      "INFO:tensorflow:Initialize variable module/bilm/RNN_1/RNN/MultiRNNCell/Cell0/rnn/lstm_cell/kernel:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/RNN_1/RNN/MultiRNNCell/Cell0/rnn/lstm_cell/kernel\n",
      "INFO:tensorflow:Initialize variable module/bilm/RNN_1/RNN/MultiRNNCell/Cell0/rnn/lstm_cell/projection/kernel:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/RNN_1/RNN/MultiRNNCell/Cell0/rnn/lstm_cell/projection/kernel\n",
      "INFO:tensorflow:Initialize variable module/bilm/RNN_1/RNN/MultiRNNCell/Cell1/rnn/lstm_cell/bias:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/RNN_1/RNN/MultiRNNCell/Cell1/rnn/lstm_cell/bias\n",
      "INFO:tensorflow:Initialize variable module/bilm/RNN_1/RNN/MultiRNNCell/Cell1/rnn/lstm_cell/kernel:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/RNN_1/RNN/MultiRNNCell/Cell1/rnn/lstm_cell/kernel\n",
      "INFO:tensorflow:Initialize variable module/bilm/RNN_1/RNN/MultiRNNCell/Cell1/rnn/lstm_cell/projection/kernel:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/RNN_1/RNN/MultiRNNCell/Cell1/rnn/lstm_cell/projection/kernel\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Initialize variable module/bilm/char_embed:0 from checkpoint b'/tmp/tfhub_modules/147211ae67f6cee45196ab9725932f31e8151552/variables/variables' with bilm/char_embed\n"
     ]
    }
   ],
   "source": [
    "elmo_model = hub.Module(\"https://alpha.tfhub.dev/google/elmo/2\", trainable=True)\n",
    "sess.run(tf.global_variables_initializer())\n",
    "sess.run(tf.tables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ELMo Embeddings\n",
    "\n",
    "A quick example will illustrate how ELMo Embeddings work. When we pass to our model a list of sentences (either as strings or as lists of tokens), we get back a list of 1024-dimensional embeddings for every sentence. These are the ELMo embeddings of the tokens in the sentence. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = elmo_model(\n",
    "    [\"the cat is on the mat\", \"dogs are in the fog\"],\n",
    "    signature=\"default\",\n",
    "    as_dict=True)[\"elmo\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment analysis\n",
    "\n",
    "In this experiment, we're going to build a simple neural network for sentiment analysis. As our training and test data, we use the IMDB movie reviews that come pre-packaged with Keras. We shuffle the reviews and pad all texts to a maximum length of 500."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/text-datasets/imdb.npz\n",
      "17465344/17464789 [==============================] - 1s 0us/step\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "from keras.datasets import imdb\n",
    "from keras.preprocessing import sequence\n",
    "\n",
    "VOCABULARY_SIZE = 50000\n",
    "INDEX_FROM = 3\n",
    "START_INDEX = 1\n",
    "OOV_INDEX = 2\n",
    "EMBEDDING_DIM = 300\n",
    "SEQ_LENGTH = 500\n",
    "\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(path=\"imdb.npz\",\n",
    "                                                      num_words=VOCABULARY_SIZE,\n",
    "                                                      skip_top=0,\n",
    "                                                      maxlen=None,\n",
    "                                                      seed=113,\n",
    "                                                      start_char=START_INDEX,\n",
    "                                                      oov_char=OOV_INDEX,\n",
    "                                                      index_from=INDEX_FROM)\n",
    "\n",
    "train = list(zip(X_train, y_train))\n",
    "random.shuffle(train)\n",
    "X_train, y_train = zip(*train)\n",
    "y_train = np.array(y_train)\n",
    "\n",
    "X_train = sequence.pad_sequences(X_train, maxlen=SEQ_LENGTH)\n",
    "X_test = sequence.pad_sequences(X_test, maxlen=SEQ_LENGTH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple embeddings\n",
    "\n",
    "For our baseline, we're going to work with standard word embeddings. These map every token to a 300-dimensional embedding, irrespective of the context in which the token occurs. We'll use the English word embeddings from Facebook Research's [MUSE project](https://github.com/facebookresearch/MUSE)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2018-09-30 07:08:02--  https://s3.amazonaws.com/arrival/embeddings/wiki.multi.en.vec\n",
      "Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.102.29\n",
      "Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.102.29|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 628614720 (599M) [text/plain]\n",
      "Saving to: ‘/tmp/wiki.multi.en.vec’\n",
      "\n",
      "/tmp/wiki.multi.en. 100%[===================>] 599.49M  22.7MB/s    in 27s     \n",
      "\n",
      "2018-09-30 07:08:29 (22.1 MB/s) - ‘/tmp/wiki.multi.en.vec’ saved [628614720/628614720]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://s3.amazonaws.com/arrival/embeddings/wiki.multi.en.vec -O /tmp/wiki.multi.en.vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load these embeddings and put the ones we need in an embedding matrix, where their row indices correspond to the token indices that Keras has assigned to the tokens in the IMDB corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading vectors from /tmp/wiki.multi.en.vec\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "def load_vectors(embedding_file_path):\n",
    "    print(\"Loading vectors from\", embedding_file_path)\n",
    "    embeddings = []\n",
    "    word2id = {}\n",
    "    with open(embedding_file_path, 'r', encoding='utf-8') as f:\n",
    "        next(f)\n",
    "        for i, line in enumerate(f):\n",
    "            word, emb = line.rstrip().split(' ', 1)\n",
    "            emb = np.fromstring(emb, sep=' ')\n",
    "            assert word not in word2id, 'word found twice'\n",
    "            embeddings.append(emb)\n",
    "            word2id[word] = len(word2id)\n",
    "\n",
    "    embeddings = np.vstack(embeddings)\n",
    "    return embeddings, word2id\n",
    "\n",
    "embeddings_en, embedding_word2id_en = load_vectors(\"/tmp/wiki.multi.en.vec\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/text-datasets/imdb_word_index.json\n",
      "1646592/1641221 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "def create_embedding_matrix(target_word2id, embedding_word2id, embeddings, num_rows, num_columns):\n",
    "    embedding_matrix = np.zeros((num_rows, num_columns))\n",
    "    for word, i in target_word2id.items():\n",
    "        if i >= num_rows:\n",
    "            continue\n",
    "        if word in embedding_word2id: \n",
    "            embedding_matrix[i] = embeddings[embedding_word2id[word]]\n",
    "    return embedding_matrix\n",
    "\n",
    "word2id_en = imdb.get_word_index()\n",
    "word2id_en = {k:(v+INDEX_FROM) for k,v in word2id_en.items()}\n",
    "word2id_en[\"<PAD>\"] = 0\n",
    "word2id_en[\"<START>\"] = START_INDEX\n",
    "word2id_en[\"<UNK>\"] = OOV_INDEX\n",
    "\n",
    "embedding_matrix_en = create_embedding_matrix(word2id_en, embedding_word2id_en, \n",
    "                                              embeddings_en, VOCABULARY_SIZE+INDEX_FROM-1, EMBEDDING_DIM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models\n",
    "\n",
    "We build two models for text classification, which are identical apart from the first layer. Our basic model has a simple embedding layer, were tokens are mapped to their static embeddings. Our ELMo model has a more complex first layer, where the static embedding for each token is concatenated to the ELMo embedding for that token in the relevant context. This results in a 1,324-dimensional embedding for each token in context. In both models, this embedding layer is followed by a simple convolution with kernel size 3, a maximum pooling operation, a dense layer, and finally a final layer that predicts the sentiment of each text as a number between 0 and 1.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Lambda, Input\n",
    "from keras.layers import Flatten, Concatenate\n",
    "from keras.layers.convolutional import Conv1D\n",
    "from keras.layers.convolutional import MaxPooling1D\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint\n",
    "from keras.models import Model\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "ELMO_EMBEDDING_DIM = 1024\n",
    "\n",
    "def ElmoEmbedding(x):\n",
    "    y = elmo_model(tf.squeeze(x), signature=\"default\", as_dict=True)[\"elmo\"]\n",
    "    return y\n",
    "\n",
    "def create_basic_model():\n",
    "    sequence = Input(shape=(500,))\n",
    "    embedding = Embedding(VOCABULARY_SIZE+INDEX_FROM-1, EMBEDDING_DIM, input_length=SEQ_LENGTH, \n",
    "                              weights=[embedding_matrix_en], trainable=False)(sequence)\n",
    "        \n",
    "    conv = Conv1D(filters=64, kernel_size=3, padding='same', activation='relu')(embedding)\n",
    "    pool = MaxPooling1D(pool_size=SEQ_LENGTH)(conv)\n",
    "    flat = Flatten()(pool)\n",
    "    dense = Dense(250, activation='relu')(flat)\n",
    "    prediction = Dense(1, activation='sigmoid')(dense)\n",
    "\n",
    "    model = Model(inputs=sequence, outputs=prediction)\n",
    "    optimizer = Adam(lr=0.0001, decay=1e-3)\n",
    "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "    #print(model.summary())\n",
    "    \n",
    "    return model\n",
    "    \n",
    "    \n",
    "def create_elmo_model(): \n",
    "    token_sequence = Input(shape=(1,), dtype=\"string\", name=\"elmo_input\")\n",
    "    index_sequence = Input(shape=(SEQ_LENGTH,), name=\"standard_input\")\n",
    "\n",
    "    embedding1 = Lambda(ElmoEmbedding, output_shape=(SEQ_LENGTH, ELMO_EMBEDDING_DIM,))(token_sequence)\n",
    "    embedding2 = Embedding(VOCABULARY_SIZE+INDEX_FROM-1, EMBEDDING_DIM, input_length=SEQ_LENGTH, \n",
    "                          weights=[embedding_matrix_en], trainable=False)(index_sequence)\n",
    "    embedding = Concatenate()([embedding1, embedding2])\n",
    "        \n",
    "    conv = Conv1D(filters=64, kernel_size=3, padding='same', activation='relu')(embedding)\n",
    "    pool = MaxPooling1D(pool_size=SEQ_LENGTH)(conv)\n",
    "    flat = Flatten()(pool)\n",
    "    dense = Dense(250, activation='relu')(flat)\n",
    "    prediction = Dense(1, activation='sigmoid')(dense)\n",
    "\n",
    "    model = Model(inputs=[index_sequence, token_sequence], outputs=prediction)\n",
    "    optimizer = Adam(lr=0.00001, decay=1e-3)\n",
    "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "    #print(model.summary())\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "We train both models for a maximum of 100 epochs, but we stop earlier when the validation loss hasn't improved for two epochs. We save and evaluate the model with the lowest validation loss. Although we didn't make a big effort to tune the learning rate, we did find that the ELMo model benefits from having a much smaller initial learning rate than the basic model. We use the same decay rate for both models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "def train_basic_model(model, X_train, y_train, X_val, y_val, X_test, y_test): \n",
    "    batch_size = 16\n",
    "    earlystop = EarlyStopping(monitor='val_loss', patience=2) \n",
    "    checkpoint = ModelCheckpoint('basic_model.hdf5', save_best_only=True, monitor='val_loss', mode='min')\n",
    "\n",
    "    model.fit(X_train, y_train, validation_data=(X_val, y_val), \n",
    "              epochs=100, batch_size=batch_size, callbacks=[earlystop, checkpoint])\n",
    "    model.load_weights(filepath='basic_model.hdf5')\n",
    "    scores = model.evaluate(X_test, y_test, batch_size=batch_size)\n",
    "    print(\"Accuracy: %.2f%%\" % (scores[1]*100))\n",
    "    return scores[1]*100\n",
    "\n",
    "\n",
    "def train_elmo_model(model, X_train, E_train, y_train, X_val, E_val, y_val, X_test, E_test, y_test): \n",
    "    batch_size = 16\n",
    "    earlystop = EarlyStopping(monitor='val_loss', patience=2)        \n",
    "    checkpoint = ModelCheckpoint('elmo_model.hdf5', save_best_only=True, monitor='val_loss', mode='min')\n",
    "\n",
    "    model.fit([X_train, E_train], y_train, validation_data=([X_val, E_val], y_val), \n",
    "              epochs=100, batch_size=batch_size, callbacks=[earlystop, checkpoint])\n",
    "    model.load_weights(filepath='elmo_model.hdf5')\n",
    "    scores = model.evaluate([X_test, E_test], y_test, batch_size=batch_size)\n",
    "    print(\"Accuracy: %.2f%%\" % (scores[1]*100))\n",
    "    return scores[1]*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the ELMo model is quite slow, we chose to work with relatively small datasets. We train on just 200 training examples, validate the model on another 200 examples after each epoch, and test its final performance on 500 test examples. We repeat this process 10 times, and choose the training and validation examples randomly from the larger IMDB training set on each iteration. The ELMo model is trained, validated and tested on exactly the same examples as the basic model.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 14s 68ms/step - loss: 0.6963 - acc: 0.5100 - val_loss: 0.6905 - val_acc: 0.5300\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6863 - acc: 0.5050 - val_loss: 0.6901 - val_acc: 0.4850\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6654 - acc: 0.5650 - val_loss: 0.6827 - val_acc: 0.5250\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6248 - acc: 0.7800 - val_loss: 0.6739 - val_acc: 0.6800\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.5689 - acc: 0.9250 - val_loss: 0.6625 - val_acc: 0.6050\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.4656 - acc: 1.0000 - val_loss: 0.6334 - val_acc: 0.6950\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.3338 - acc: 1.0000 - val_loss: 0.6357 - val_acc: 0.5800\n",
      "Epoch 8/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.2338 - acc: 0.9950 - val_loss: 0.7009 - val_acc: 0.5400\n",
      "500/500 [==============================] - 0s 427us/step\n",
      "Accuracy: 73.20%\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 133s 665ms/step - loss: 0.7547 - acc: 0.4900 - val_loss: 0.7710 - val_acc: 0.4850\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.5166 - acc: 0.7300 - val_loss: 0.5827 - val_acc: 0.7150\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.2454 - acc: 0.9800 - val_loss: 0.5219 - val_acc: 0.7750\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0807 - acc: 1.0000 - val_loss: 0.5127 - val_acc: 0.7550\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0217 - acc: 1.0000 - val_loss: 0.4873 - val_acc: 0.7850\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0078 - acc: 1.0000 - val_loss: 0.5004 - val_acc: 0.7800\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0040 - acc: 1.0000 - val_loss: 0.5067 - val_acc: 0.7800\n",
      "500/500 [==============================] - 147s 294ms/step\n",
      "Accuracy: 78.00%\n",
      "[73.2]\n",
      "[78.0]\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 14s 71ms/step - loss: 0.6985 - acc: 0.5350 - val_loss: 0.7128 - val_acc: 0.4850\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6816 - acc: 0.6250 - val_loss: 0.6906 - val_acc: 0.5000\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6621 - acc: 0.7950 - val_loss: 0.6962 - val_acc: 0.4850\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6239 - acc: 0.7550 - val_loss: 0.6771 - val_acc: 0.5900\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.5709 - acc: 0.8600 - val_loss: 0.7070 - val_acc: 0.4850\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.4731 - acc: 0.9450 - val_loss: 0.6414 - val_acc: 0.6700\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.3689 - acc: 0.9850 - val_loss: 0.6167 - val_acc: 0.6950\n",
      "Epoch 8/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.2319 - acc: 1.0000 - val_loss: 0.5872 - val_acc: 0.6850\n",
      "Epoch 9/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.1378 - acc: 1.0000 - val_loss: 0.5707 - val_acc: 0.6950\n",
      "Epoch 10/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0772 - acc: 1.0000 - val_loss: 0.5570 - val_acc: 0.6900\n",
      "Epoch 11/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0440 - acc: 1.0000 - val_loss: 0.5659 - val_acc: 0.6950\n",
      "Epoch 12/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0278 - acc: 1.0000 - val_loss: 0.5477 - val_acc: 0.7050\n",
      "Epoch 13/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0169 - acc: 1.0000 - val_loss: 0.5372 - val_acc: 0.7100\n",
      "Epoch 14/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0121 - acc: 1.0000 - val_loss: 0.5376 - val_acc: 0.7050\n",
      "Epoch 15/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0092 - acc: 1.0000 - val_loss: 0.5314 - val_acc: 0.6950\n",
      "Epoch 16/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0073 - acc: 1.0000 - val_loss: 0.5433 - val_acc: 0.7050\n",
      "Epoch 17/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0059 - acc: 1.0000 - val_loss: 0.5301 - val_acc: 0.6900\n",
      "Epoch 18/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0049 - acc: 1.0000 - val_loss: 0.5385 - val_acc: 0.7050\n",
      "Epoch 19/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0042 - acc: 1.0000 - val_loss: 0.5346 - val_acc: 0.7000\n",
      "500/500 [==============================] - 0s 465us/step\n",
      "Accuracy: 73.60%\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 134s 668ms/step - loss: 0.7519 - acc: 0.4850 - val_loss: 0.6618 - val_acc: 0.5650\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.5526 - acc: 0.6900 - val_loss: 0.6052 - val_acc: 0.7450\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.3498 - acc: 0.9450 - val_loss: 0.6074 - val_acc: 0.6200\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.1717 - acc: 0.9950 - val_loss: 0.5556 - val_acc: 0.7050\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 118s 591ms/step - loss: 0.0557 - acc: 1.0000 - val_loss: 0.5040 - val_acc: 0.7850\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 118s 592ms/step - loss: 0.0215 - acc: 1.0000 - val_loss: 0.5164 - val_acc: 0.7500\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 118s 591ms/step - loss: 0.0093 - acc: 1.0000 - val_loss: 0.4812 - val_acc: 0.7900\n",
      "Epoch 8/100\n",
      "200/200 [==============================] - 118s 591ms/step - loss: 0.0053 - acc: 1.0000 - val_loss: 0.4932 - val_acc: 0.7850\n",
      "Epoch 9/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0036 - acc: 1.0000 - val_loss: 0.4812 - val_acc: 0.7900\n",
      "Epoch 10/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0026 - acc: 1.0000 - val_loss: 0.4881 - val_acc: 0.8000\n",
      "Epoch 11/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.0019 - acc: 1.0000 - val_loss: 0.4843 - val_acc: 0.8000\n",
      "500/500 [==============================] - 146s 293ms/step\n",
      "Accuracy: 75.40%\n",
      "[73.2, 73.6]\n",
      "[78.0, 75.4]\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 15s 76ms/step - loss: 0.6898 - acc: 0.5700 - val_loss: 0.7066 - val_acc: 0.5150\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6690 - acc: 0.5700 - val_loss: 0.6906 - val_acc: 0.5150\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6454 - acc: 0.5750 - val_loss: 0.6917 - val_acc: 0.5150\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.5979 - acc: 0.8550 - val_loss: 0.7017 - val_acc: 0.5150\n",
      "500/500 [==============================] - 0s 442us/step\n",
      "Accuracy: 52.40%\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 134s 672ms/step - loss: 0.8133 - acc: 0.5100 - val_loss: 0.6674 - val_acc: 0.5600\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 119s 593ms/step - loss: 0.5297 - acc: 0.7950 - val_loss: 0.6653 - val_acc: 0.5350\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.3082 - acc: 0.9400 - val_loss: 0.6049 - val_acc: 0.6450\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.1210 - acc: 0.9950 - val_loss: 0.5358 - val_acc: 0.7300\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 119s 596ms/step - loss: 0.0348 - acc: 1.0000 - val_loss: 0.5247 - val_acc: 0.7200\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0129 - acc: 1.0000 - val_loss: 0.4654 - val_acc: 0.7600\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0063 - acc: 1.0000 - val_loss: 0.5201 - val_acc: 0.7350\n",
      "Epoch 8/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0041 - acc: 1.0000 - val_loss: 0.4939 - val_acc: 0.7400\n",
      "500/500 [==============================] - 147s 294ms/step\n",
      "Accuracy: 75.40%\n",
      "[73.2, 73.6, 52.400000000000006]\n",
      "[78.0, 75.4, 75.4]\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 16s 79ms/step - loss: 0.6912 - acc: 0.5600 - val_loss: 0.6994 - val_acc: 0.4850\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6717 - acc: 0.5900 - val_loss: 0.7004 - val_acc: 0.4850\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6487 - acc: 0.5600 - val_loss: 0.6997 - val_acc: 0.4850\n",
      "500/500 [==============================] - 0s 430us/step\n",
      "Accuracy: 47.60%\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 136s 678ms/step - loss: 0.6857 - acc: 0.5900 - val_loss: 1.0049 - val_acc: 0.4850\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.5205 - acc: 0.7100 - val_loss: 0.6605 - val_acc: 0.5700\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.2823 - acc: 0.9400 - val_loss: 0.5442 - val_acc: 0.7350\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.0988 - acc: 1.0000 - val_loss: 0.5241 - val_acc: 0.7350\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0274 - acc: 1.0000 - val_loss: 0.5129 - val_acc: 0.7450\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.0115 - acc: 1.0000 - val_loss: 0.6625 - val_acc: 0.6550\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.0048 - acc: 1.0000 - val_loss: 0.5059 - val_acc: 0.7650\n",
      "Epoch 8/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.0027 - acc: 1.0000 - val_loss: 0.5728 - val_acc: 0.7450\n",
      "Epoch 9/100\n",
      "200/200 [==============================] - 119s 593ms/step - loss: 0.0019 - acc: 1.0000 - val_loss: 0.5233 - val_acc: 0.7650\n",
      "500/500 [==============================] - 146s 292ms/step\n",
      "Accuracy: 77.80%\n",
      "[73.2, 73.6, 52.400000000000006, 47.599999999999994]\n",
      "[78.0, 75.4, 75.4, 77.8]\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 17s 84ms/step - loss: 0.6958 - acc: 0.5250 - val_loss: 0.6964 - val_acc: 0.4850\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6791 - acc: 0.5800 - val_loss: 0.6888 - val_acc: 0.4850\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6732 - acc: 0.5550 - val_loss: 0.6851 - val_acc: 0.5050\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6187 - acc: 0.7900 - val_loss: 0.6811 - val_acc: 0.5000\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.5489 - acc: 0.7850 - val_loss: 0.6617 - val_acc: 0.5750\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.4567 - acc: 0.9550 - val_loss: 0.6384 - val_acc: 0.6450\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.3256 - acc: 0.9950 - val_loss: 0.6303 - val_acc: 0.5900\n",
      "Epoch 8/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.1966 - acc: 1.0000 - val_loss: 0.6442 - val_acc: 0.6650\n",
      "Epoch 9/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.1098 - acc: 1.0000 - val_loss: 0.6561 - val_acc: 0.6550\n",
      "500/500 [==============================] - 0s 386us/step\n",
      "Accuracy: 63.60%\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 136s 682ms/step - loss: 0.7216 - acc: 0.5250 - val_loss: 0.6614 - val_acc: 0.5900\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.4357 - acc: 0.9100 - val_loss: 0.6055 - val_acc: 0.6550\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.1969 - acc: 0.9850 - val_loss: 0.5397 - val_acc: 0.7150\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 119s 593ms/step - loss: 0.0495 - acc: 1.0000 - val_loss: 0.5921 - val_acc: 0.6850\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 119s 593ms/step - loss: 0.0136 - acc: 1.0000 - val_loss: 0.5667 - val_acc: 0.7100\n",
      "500/500 [==============================] - 146s 293ms/step\n",
      "Accuracy: 75.60%\n",
      "[73.2, 73.6, 52.400000000000006, 47.599999999999994, 63.6]\n",
      "[78.0, 75.4, 75.4, 77.8, 75.6]\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 18s 89ms/step - loss: 0.7006 - acc: 0.5250 - val_loss: 0.6925 - val_acc: 0.5200\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6802 - acc: 0.6950 - val_loss: 0.6952 - val_acc: 0.4850\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6579 - acc: 0.6550 - val_loss: 0.6955 - val_acc: 0.4850\n",
      "500/500 [==============================] - 0s 446us/step\n",
      "Accuracy: 52.60%\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 136s 682ms/step - loss: 0.6876 - acc: 0.5900 - val_loss: 0.6666 - val_acc: 0.5600\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.4204 - acc: 0.8000 - val_loss: 0.6653 - val_acc: 0.5700\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.2060 - acc: 0.9700 - val_loss: 0.5187 - val_acc: 0.7350\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.0899 - acc: 0.9950 - val_loss: 0.5245 - val_acc: 0.7200\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.0289 - acc: 1.0000 - val_loss: 0.5782 - val_acc: 0.7350\n",
      "500/500 [==============================] - 146s 292ms/step\n",
      "Accuracy: 72.60%\n",
      "[73.2, 73.6, 52.400000000000006, 47.599999999999994, 63.6, 52.6]\n",
      "[78.0, 75.4, 75.4, 77.8, 75.6, 72.6]\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 18s 92ms/step - loss: 0.7024 - acc: 0.4550 - val_loss: 0.6921 - val_acc: 0.5150\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6773 - acc: 0.5450 - val_loss: 0.7007 - val_acc: 0.5150\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6488 - acc: 0.5600 - val_loss: 0.6866 - val_acc: 0.5750\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6356 - acc: 0.6750 - val_loss: 0.7016 - val_acc: 0.5150\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.5857 - acc: 0.7050 - val_loss: 0.6808 - val_acc: 0.4900\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.5019 - acc: 0.9500 - val_loss: 0.6593 - val_acc: 0.5500\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.3816 - acc: 1.0000 - val_loss: 0.6627 - val_acc: 0.5350\n",
      "Epoch 8/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.2653 - acc: 1.0000 - val_loss: 0.6558 - val_acc: 0.5350\n",
      "Epoch 9/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.1623 - acc: 1.0000 - val_loss: 0.6869 - val_acc: 0.5550\n",
      "Epoch 10/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0873 - acc: 1.0000 - val_loss: 0.5763 - val_acc: 0.6650\n",
      "Epoch 11/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0467 - acc: 1.0000 - val_loss: 0.5194 - val_acc: 0.7200\n",
      "Epoch 12/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0281 - acc: 1.0000 - val_loss: 0.4980 - val_acc: 0.7750\n",
      "Epoch 13/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0185 - acc: 1.0000 - val_loss: 0.5184 - val_acc: 0.7500\n",
      "Epoch 14/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0127 - acc: 1.0000 - val_loss: 0.4863 - val_acc: 0.7800\n",
      "Epoch 15/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0100 - acc: 1.0000 - val_loss: 0.4854 - val_acc: 0.7800\n",
      "Epoch 16/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0077 - acc: 1.0000 - val_loss: 0.4892 - val_acc: 0.7500\n",
      "Epoch 17/100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0062 - acc: 1.0000 - val_loss: 0.4814 - val_acc: 0.7750\n",
      "Epoch 18/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0052 - acc: 1.0000 - val_loss: 0.4898 - val_acc: 0.7500\n",
      "Epoch 19/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0044 - acc: 1.0000 - val_loss: 0.4761 - val_acc: 0.7800\n",
      "Epoch 20/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0039 - acc: 1.0000 - val_loss: 0.4827 - val_acc: 0.7550\n",
      "Epoch 21/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0033 - acc: 1.0000 - val_loss: 0.4768 - val_acc: 0.7750\n",
      "500/500 [==============================] - 0s 463us/step\n",
      "Accuracy: 77.60%\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 137s 686ms/step - loss: 0.7365 - acc: 0.5700 - val_loss: 0.7008 - val_acc: 0.5450\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.4214 - acc: 0.9000 - val_loss: 0.6105 - val_acc: 0.6400\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 119s 596ms/step - loss: 0.2360 - acc: 0.9500 - val_loss: 0.5315 - val_acc: 0.7400\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 119s 596ms/step - loss: 0.0874 - acc: 1.0000 - val_loss: 0.5177 - val_acc: 0.7650\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 119s 597ms/step - loss: 0.0292 - acc: 1.0000 - val_loss: 0.4674 - val_acc: 0.7700\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 119s 597ms/step - loss: 0.0104 - acc: 1.0000 - val_loss: 0.4516 - val_acc: 0.7850\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0060 - acc: 1.0000 - val_loss: 0.4527 - val_acc: 0.7950\n",
      "Epoch 8/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.0033 - acc: 1.0000 - val_loss: 0.4443 - val_acc: 0.7850\n",
      "Epoch 9/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0022 - acc: 1.0000 - val_loss: 0.4411 - val_acc: 0.7850\n",
      "Epoch 10/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.0016 - acc: 1.0000 - val_loss: 0.4484 - val_acc: 0.7850\n",
      "Epoch 11/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0011 - acc: 1.0000 - val_loss: 0.4512 - val_acc: 0.7800\n",
      "500/500 [==============================] - 147s 294ms/step\n",
      "Accuracy: 76.80%\n",
      "[73.2, 73.6, 52.400000000000006, 47.599999999999994, 63.6, 52.6, 77.60000000000001]\n",
      "[78.0, 75.4, 75.4, 77.8, 75.6, 72.6, 76.8]\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 19s 93ms/step - loss: 0.6901 - acc: 0.5300 - val_loss: 0.7155 - val_acc: 0.4850\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6818 - acc: 0.5450 - val_loss: 0.7007 - val_acc: 0.4850\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6619 - acc: 0.8350 - val_loss: 0.6900 - val_acc: 0.5050\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6344 - acc: 0.5600 - val_loss: 0.6899 - val_acc: 0.4800\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.5892 - acc: 0.9050 - val_loss: 0.6838 - val_acc: 0.5000\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.5019 - acc: 0.9600 - val_loss: 0.6536 - val_acc: 0.6100\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.4113 - acc: 0.9500 - val_loss: 0.6431 - val_acc: 0.6050\n",
      "Epoch 8/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.2858 - acc: 0.9750 - val_loss: 0.6098 - val_acc: 0.6850\n",
      "Epoch 9/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.1752 - acc: 0.9950 - val_loss: 0.5943 - val_acc: 0.7050\n",
      "Epoch 10/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.1000 - acc: 1.0000 - val_loss: 0.5566 - val_acc: 0.7550\n",
      "Epoch 11/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0691 - acc: 1.0000 - val_loss: 0.5453 - val_acc: 0.7450\n",
      "Epoch 12/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0398 - acc: 1.0000 - val_loss: 0.5580 - val_acc: 0.6700\n",
      "Epoch 13/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0243 - acc: 1.0000 - val_loss: 0.5394 - val_acc: 0.7600\n",
      "Epoch 14/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0151 - acc: 1.0000 - val_loss: 0.5572 - val_acc: 0.7400\n",
      "Epoch 15/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0111 - acc: 1.0000 - val_loss: 0.5395 - val_acc: 0.7600\n",
      "500/500 [==============================] - 0s 445us/step\n",
      "Accuracy: 70.40%\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 138s 691ms/step - loss: 0.7729 - acc: 0.5050 - val_loss: 0.7021 - val_acc: 0.5200\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.5565 - acc: 0.7350 - val_loss: 0.6076 - val_acc: 0.7000\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.3504 - acc: 0.9100 - val_loss: 0.7633 - val_acc: 0.5450\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.1997 - acc: 0.9550 - val_loss: 0.5112 - val_acc: 0.7300\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 119s 596ms/step - loss: 0.0722 - acc: 1.0000 - val_loss: 0.5036 - val_acc: 0.7650\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0237 - acc: 1.0000 - val_loss: 0.6465 - val_acc: 0.6500\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.0121 - acc: 1.0000 - val_loss: 0.5000 - val_acc: 0.7550\n",
      "Epoch 8/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.0046 - acc: 1.0000 - val_loss: 0.5470 - val_acc: 0.7550\n",
      "Epoch 9/100\n",
      "200/200 [==============================] - 119s 594ms/step - loss: 0.0029 - acc: 1.0000 - val_loss: 0.5003 - val_acc: 0.7350\n",
      "500/500 [==============================] - 146s 292ms/step\n",
      "Accuracy: 75.20%\n",
      "[73.2, 73.6, 52.400000000000006, 47.599999999999994, 63.6, 52.6, 77.60000000000001, 70.39999999999999]\n",
      "[78.0, 75.4, 75.4, 77.8, 75.6, 72.6, 76.8, 75.2]\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 19s 97ms/step - loss: 0.7058 - acc: 0.4350 - val_loss: 0.6990 - val_acc: 0.4850\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6823 - acc: 0.5500 - val_loss: 0.6917 - val_acc: 0.4850\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6617 - acc: 0.5900 - val_loss: 0.6862 - val_acc: 0.5200\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6235 - acc: 0.8850 - val_loss: 0.6760 - val_acc: 0.6650\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.5606 - acc: 0.9450 - val_loss: 0.6644 - val_acc: 0.6350\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.4543 - acc: 0.9950 - val_loss: 0.6354 - val_acc: 0.7300\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.3182 - acc: 0.9900 - val_loss: 0.6208 - val_acc: 0.7100\n",
      "Epoch 8/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.1841 - acc: 1.0000 - val_loss: 0.5786 - val_acc: 0.7300\n",
      "Epoch 9/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0977 - acc: 1.0000 - val_loss: 0.5662 - val_acc: 0.7300\n",
      "Epoch 10/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0485 - acc: 1.0000 - val_loss: 0.5511 - val_acc: 0.7450\n",
      "Epoch 11/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0291 - acc: 1.0000 - val_loss: 0.5544 - val_acc: 0.7350\n",
      "Epoch 12/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0187 - acc: 1.0000 - val_loss: 0.5564 - val_acc: 0.7250\n",
      "500/500 [==============================] - 0s 450us/step\n",
      "Accuracy: 73.00%\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 139s 696ms/step - loss: 0.7269 - acc: 0.5750 - val_loss: 0.6576 - val_acc: 0.5750\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 119s 595ms/step - loss: 0.4709 - acc: 0.8800 - val_loss: 0.6180 - val_acc: 0.6500\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 119s 596ms/step - loss: 0.2449 - acc: 0.9650 - val_loss: 0.6223 - val_acc: 0.6700\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 119s 596ms/step - loss: 0.0785 - acc: 1.0000 - val_loss: 0.5302 - val_acc: 0.7650\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 119s 597ms/step - loss: 0.0257 - acc: 1.0000 - val_loss: 0.5341 - val_acc: 0.7400\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 119s 596ms/step - loss: 0.0100 - acc: 1.0000 - val_loss: 0.5540 - val_acc: 0.7150\n",
      "500/500 [==============================] - 147s 294ms/step\n",
      "Accuracy: 75.40%\n",
      "[73.2, 73.6, 52.400000000000006, 47.599999999999994, 63.6, 52.6, 77.60000000000001, 70.39999999999999, 73.0]\n",
      "[78.0, 75.4, 75.4, 77.8, 75.6, 72.6, 76.8, 75.2, 75.4]\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 21s 103ms/step - loss: 0.6945 - acc: 0.5300 - val_loss: 0.6905 - val_acc: 0.5200\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6844 - acc: 0.5300 - val_loss: 0.6885 - val_acc: 0.6350\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6541 - acc: 0.6250 - val_loss: 0.6828 - val_acc: 0.6850\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.6148 - acc: 0.7900 - val_loss: 0.6729 - val_acc: 0.6900\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.5352 - acc: 0.9950 - val_loss: 0.6590 - val_acc: 0.5650\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.4273 - acc: 0.9900 - val_loss: 0.6433 - val_acc: 0.5750\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.3053 - acc: 0.9950 - val_loss: 0.6134 - val_acc: 0.6800\n",
      "Epoch 8/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.1787 - acc: 1.0000 - val_loss: 0.5902 - val_acc: 0.7100\n",
      "Epoch 9/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0959 - acc: 1.0000 - val_loss: 0.5949 - val_acc: 0.6650\n",
      "Epoch 10/100\n",
      "200/200 [==============================] - 0s 1ms/step - loss: 0.0558 - acc: 1.0000 - val_loss: 0.6092 - val_acc: 0.6300\n",
      "500/500 [==============================] - 0s 449us/step\n",
      "Accuracy: 68.00%\n",
      "Train on 200 samples, validate on 200 samples\n",
      "Epoch 1/100\n",
      "200/200 [==============================] - 140s 701ms/step - loss: 0.7259 - acc: 0.5050 - val_loss: 0.6911 - val_acc: 0.4950\n",
      "Epoch 2/100\n",
      "200/200 [==============================] - 119s 597ms/step - loss: 0.5202 - acc: 0.8050 - val_loss: 0.6503 - val_acc: 0.6300\n",
      "Epoch 3/100\n",
      "200/200 [==============================] - 119s 597ms/step - loss: 0.2861 - acc: 0.9300 - val_loss: 0.6291 - val_acc: 0.6450\n",
      "Epoch 4/100\n",
      "200/200 [==============================] - 119s 597ms/step - loss: 0.1120 - acc: 0.9950 - val_loss: 0.5759 - val_acc: 0.7050\n",
      "Epoch 5/100\n",
      "200/200 [==============================] - 119s 597ms/step - loss: 0.0415 - acc: 1.0000 - val_loss: 0.5750 - val_acc: 0.7100\n",
      "Epoch 6/100\n",
      "200/200 [==============================] - 120s 598ms/step - loss: 0.0128 - acc: 1.0000 - val_loss: 0.5392 - val_acc: 0.7450\n",
      "Epoch 7/100\n",
      "200/200 [==============================] - 119s 596ms/step - loss: 0.0061 - acc: 1.0000 - val_loss: 0.5476 - val_acc: 0.7350\n",
      "Epoch 8/100\n",
      "200/200 [==============================] - 119s 596ms/step - loss: 0.0037 - acc: 1.0000 - val_loss: 0.5535 - val_acc: 0.7350\n",
      "500/500 [==============================] - 146s 293ms/step\n",
      "Accuracy: 74.00%\n",
      "[73.2, 73.6, 52.400000000000006, 47.599999999999994, 63.6, 52.6, 77.60000000000001, 70.39999999999999, 73.0, 68.0]\n",
      "[78.0, 75.4, 75.4, 77.8, 75.6, 72.6, 76.8, 75.2, 75.4, 74.0]\n",
      "65.2\n",
      "75.61999999999999\n"
     ]
    }
   ],
   "source": [
    "elmo_accuracies = []\n",
    "basic_accuracies = []\n",
    "\n",
    "test_size = 500\n",
    "validation_size = 200\n",
    "training_size = 200\n",
    "id2word_en = {v:k for k,v in word2id_en.items()}\n",
    "\n",
    "for i in range(10):     \n",
    "    \n",
    "    train = list(zip(X_train, y_train))\n",
    "    test = list(zip(X_test, y_test))\n",
    "    random.shuffle(train)\n",
    "    X_train, y_train = zip(*train)\n",
    "    X_test, y_test = zip(*test)\n",
    "    \n",
    "    X_train = np.array(X_train)\n",
    "    y_train = np.array(y_train)\n",
    "    \n",
    "    X_test = np.array(X_test)\n",
    "    y_test = np.array(y_test)\n",
    "            \n",
    "    train_texts = [\" \".join([id2word_en[idx] for idx in seq]) for seq in X_train[:training_size]]\n",
    "    test_texts = [\" \".join([id2word_en[idx] for idx in seq]) for seq in X_test[:test_size]]\n",
    "    val_texts = [\" \".join([id2word_en[idx] for idx in seq]) for seq in X_test[test_size:test_size+validation_size]]\n",
    "\n",
    "    E_train = np.array(train_texts)\n",
    "    E_test = np.array(test_texts)\n",
    "    E_val = np.array(val_texts)\n",
    "        \n",
    "    model_baseline = create_basic_model()\n",
    "    basic_acc = train_basic_model(model_baseline, X_train[:training_size], y_train[:training_size], \n",
    "                                  X_test[test_size:test_size+validation_size],\n",
    "                                  y_test[test_size:test_size+validation_size], \n",
    "                                  X_test[:test_size], y_test[:test_size])\n",
    "    basic_accuracies.append(basic_acc)\n",
    "\n",
    "    model_elmo = create_elmo_model()\n",
    "    elmo_acc = train_elmo_model(model_elmo, X_train[:training_size], E_train, y_train[:training_size],\n",
    "                                X_test[test_size:test_size+validation_size], \n",
    "                                E_val, y_test[test_size:test_size+validation_size], \n",
    "                                X_test[:test_size], E_test, y_test[:test_size])\n",
    "    elmo_accuracies.append(elmo_acc)\n",
    "        \n",
    "    print(basic_accuracies)\n",
    "    print(elmo_accuracies)\n",
    "    \n",
    "print(np.mean(basic_accuracies))\n",
    "print(np.mean(elmo_accuracies))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "When we train a basic sentiment analysis model on just 200 training examples, the results are hit and miss: the accuracies on unseen texts range from just 48% to 78%. When we replace the simple embedding layer by an ELMo embedding layer, however, the model performs about 10% better on average. Its accuracies were also much more consistent, between 73% and 78%. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f0d71fd8160>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAFpCAYAAAC4SK2+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAErJJREFUeJzt3X+s3Xd93/HXuzFoIQ0hkOYqf2xYaCi1GohH7yoxouq6HkxdJn5IdK23ohSsmnUj2/6YNhdPLQhZSicQSsWGajDIbanXKlpGhCsryPPZlGrLSEoIGU4VlZlCSUNSrQEDGkn47A+ftK7lzufmfc79+XhIV+ec7/me731b9+irp7/f7z23xhgBAOCF+YH1HgAAYDMTUwAADWIKAKBBTAEANIgpAIAGMQUA0CCmAAAaxBQAQIOYAgBoEFMAAA071vKbXXfddWPnzp1r+S3ZpL797W/nqquuWu8xgC3GvoXVePDBB58aY/zQ5dZb05jauXNnHnjggbX8lmxSk8kkKysr6z0GsMXYt7AaVfWVWdZzmg8AoEFMAQA0iCkAgAYxBQDQIKYAABrEFABAg5gCAGgQUwAADWIKAKBBTAEANIgpAIAGMQUA0LCmf+gYABahqhay3THGQrbL1uLIFACb3hhjpq9X/pvPzLyukGJWYgoAoEFMAQA0uGaKNeW6BgC2GkemWFOuawBgqxFTAAANYgoAoEFMAQA0iCkAgAYxBQDQIKYAABrEFABAg5gCAGgQUwAADf6cDAAb1s3vvzdPf/eZuW5z58ETc93eNVe+KF/45TfNdZtsLmIKgA3r6e8+k7N33Dq37U0mk6ysrMxte8n844zNx2k+AIAGMQUA0CCmAAAaxBQAQIOYAgBoEFMAAA1iCgCgQUwBADSIKQCAhst+AnpV3Zjkty9Y9Kokv5TkZUl+PsmT0+XvHWP87twnBGDbunrXwbzm2MH5bvTYfDd39a4kmd+ntLP5XDamxhh/kGR3klTVFUn+OMndSd6Z5MNjjA8udEIAtq1vnbljvUe4rGuufNF6j8A6W+3f5tub5A/HGF+pqkXMAwB/bp5/ly85/3f05r1NWO01Uz+T5PgFj99TVQ9X1Seq6to5zgUAsCnMfGSqql6c5M1JfnG66KNJPpBkTG8/lORdl3jdgSQHkmRpaSmTyaQ3MduG9wqwCPYtzNtqTvP9ZJLfH2M8kSTP3yZJVX0syWcu9aIxxpEkR5JkeXl5rKysvOBh2UZOnoj3CjB39i0swGpO8+3LBaf4quqGC557W5JH5jUUAMBmMdORqap6SZI3Jnn3BYv/XVXtzvnTfGcveg4AYFuYKabGGN9J8oqLlr1jIRMBAGwiPgEdAKBBTAEANIgpAIAGMQUA0LDaPycDABvOav7EWf3K7NsdY7yAadhuHJkCYNMbY8z0dfr06ZnXFVLMSkwBADSIKQCABjEFANAgpgAAGsQUAECDmAIAaBBTAAANPrSTubj5/ffm6e8+M9dt7jx4Yq7bu+bKF+ULv/ymuW4TAMQUc/H0d5/J2Ttundv2JpNJVlZW5ra9ZP5xBgCJ03wAbAPHjx/PTTfdlL179+amm27K8ePH13skthBHpgDY0o4fP55Dhw7l6NGjee6553LFFVdk//79SZJ9+/at83RsBY5MAbClHT58OEePHs2ePXuyY8eO7NmzJ0ePHs3hw4fXezS2CDEFwJZ25syZ3HLLLX9p2S233JIzZ86s00RsNWIKgC1t165due+++/7Ssvvuuy+7du1ap4nYasQUAFvaoUOHsn///pw+fTrPPvtsTp8+nf379+fQoUPrPRpbhAvQAdjSnr/I/Pbbb8+ZM2eya9euHD582MXnzI2YAmDL27dvX/bt27eQz7ADp/kAABrEFABAg5gCAGgQUwAADWIKAKBBTAEANIgpAIAGMQUA0CCmAAAaxBQAQIOYAgBoEFMAAA1iCgCgQUwBADSIKQCABjEFANAgpgAAGi4bU1V1Y1U9dMHXN6vqX1bVy6vqs1X12PT22rUYGABgI7lsTI0x/mCMsXuMsTvJjyb5TpK7kxxMcmqM8eokp6aPAQC2ldWe5tub5A/HGF9J8pYkx6bLjyV56zwHAwDYDHascv2fSXJ8en9pjPF4kowxHq+q6y/1gqo6kORAkiwtLWUymbzAUdno5vmzPXfu3ELeK95/sL0tat/C9jZzTFXVi5O8OckvruYbjDGOJDmSJMvLy2NlZWU1L2ezOHki8/zZTiaTuW4vydxnBDafhexb2PZWc5rvJ5P8/hjjienjJ6rqhiSZ3n5j3sMBAGx0q4mpffmLU3xJck+S26b3b0vy6XkNBQCwWcwUU1X1kiRvTPKfLlh8R5I3VtVj0+fumP94AAAb20zXTI0xvpPkFRct+9Oc/+0+AIBtyyegAwA0iCkAgIbVfs4UXNLVuw7mNcfm/CH4xy6/ympcvStJbp3vRgHY9sQUc/GtM3fk7B3zC5VFfBbMzoMn5ro9AEic5gMAaBFTAAANYgoAoEFMAQA0iCkAgAYxBQDQIKYAABrEFABAg5gCAGgQUwAADWIKAKBBTAEANIgpAIAGMQUA0CCmAAAaxBQAQIOYAgBoEFMAAA1iCgCgQUwBADSIKQCABjEFANAgpgAAGsQUAECDmAIAaBBTAAANYgoAoEFMAQA0iCkAgAYxBQDQIKYAABrEFABAg5gCAGgQUwAADWIKAKBhppiqqpdV1V1V9WhVnamq11fV+6rqj6vqoenX31/0sAAAG82OGde7M8nJMcbbq+rFSV6S5O8l+fAY44MLmw4AYIO7bExV1UuT/HiSn0uSMcb3knyvqhY7GQDAJjDLab5XJXkyySer6vNV9fGqumr63Huq6uGq+kRVXbu4MQEANqZZTvPtSPK6JLePMe6vqjuTHEzykSQfSDKmtx9K8q6LX1xVB5IcSJKlpaVMJpP5TM6GM8+f7blz5xbyXvH+g+1tUfsWtrdZYuprSb42xrh/+viuJAfHGE88v0JVfSzJZy714jHGkSRHkmR5eXmsrKy0BmaDOnki8/zZTiaTuW4vydxnBDafhexb2PYue5pvjPEnSb5aVTdOF+1N8qWquuGC1d6W5JEFzAcAsKHN+tt8tyf51PQ3+b6c5J1JfrWqduf8ab6zSd69kAkBADawmWJqjPFQkuWLFr9j/uMAAGwuPgEdAKBBTAEANIgpAIAGMQUA0CCmAAAaxBQAQIOYAgBoEFMAAA1iCgCgQUwBADSIKQCABjEFANAgpgAAGsQUAECDmAIAaBBTAAANYgoAoEFMAQA0iCkAgAYxBQDQIKYAABrEFABAg5gCAGgQUwAADWIKAKBBTAEANIgpAIAGMQUA0CCmAAAaxBQAQIOYAgBoEFMAAA1iCgCgQUwBADSIKQCABjEFANAgpgAAGsQUAECDmAIAaBBTAAANM8VUVb2squ6qqker6kxVvb6qXl5Vn62qx6a31y56WACAjWbWI1N3Jjk5xvjhJDcnOZPkYJJTY4xXJzk1fQwAsK1cNqaq6qVJfjzJ0SQZY3xvjPFnSd6S5Nh0tWNJ3rqoIQEANqpZjky9KsmTST5ZVZ+vqo9X1VVJlsYYjyfJ9Pb6Bc4JALAh7ZhxndcluX2McX9V3ZlVnNKrqgNJDiTJ0tJSJpPJC5mTTWCeP9tz584t5L3i/Qfb26L2LWxvs8TU15J8bYxx//TxXTkfU09U1Q1jjMer6oYk37jUi8cYR5IcSZLl5eWxsrLSn5qN5+SJzPNnO5lM5rq9JHOfEdh8FrJvYdu77Gm+McafJPlqVd04XbQ3yZeS3JPktumy25J8eiETAgBsYLMcmUqS25N8qqpenOTLSd6Z8yH2O1W1P8kfJfmpxYwIALBxzRRTY4yHkixf4qm98x0HAGBz8QnoAAANYgoAoEFMAQA0iCkAgAYxBQDQIKYAABrEFABAg5gCAGgQUwAADWIKAKBBTAEANIgpAIAGMQUA0CCmAAAaxBQAQIOYAgBoEFMAAA1iCgCgQUwBADSIKQCABjEFANAgpgAAGsQUAECDmAIAaBBTAAANYgoAoEFMAQA0iCkAgAYxBQDQIKYAABrEFABAg5gCAGgQUwAADWIKAKBBTAEANIgpAIAGMQUA0CCmAAAaxBQAQIOYAgBo2DHLSlV1Nsm3kjyX5NkxxnJVvS/Jzyd5crrae8cYv7uIIdkcdh48Md8Nnpzv9q658kVz3R4AJDPG1NSeMcZTFy378Bjjg/MciM3p7B23znV7Ow+emPs2AWARnOYDAGiYNaZGknur6sGqOnDB8vdU1cNV9YmqunYB8wEAbGiznuZ7wxjj61V1fZLPVtWjST6a5AM5H1ofSPKhJO+6+IXT+DqQJEtLS5lMJvOYm23AewWYt3Pnztm3MHc1xljdC85feH7uwmulqmpnks+MMW76/712eXl5PPDAA6ufkm3HNVPAIkwmk6ysrKz3GGwSVfXgGGP5cutd9jRfVV1VVVc/fz/Jm5I8UlU3XLDa25I88kKHBQDYrGY5zbeU5O6qen793xpjnKyq36iq3Tl/mu9skncvbEoAgA3qsjE1xvhykpsvsfwdC5kIAGAT8dEIAAANYgoAoEFMAQA0iCkAgAYxBQDQIKYAABrEFABAg5gCAGgQUwAADWIKAKBBTAEANIgpAIAGMQUA0CCmAAAaxBQAQIOYAgBoEFMAAA1iCgCgQUwBADSIKQCABjEFANAgpgAAGsQUAECDmAIAaBBTAAANYgoAoEFMAQA0iCkAgAYxBQDQIKYAABrEFABAg5gCAGgQUwAADWIKAKBBTAEANIgpAIAGMQUA0CCmAAAaxBQAQIOYAgBo2DHLSlV1Nsm3kjyX5NkxxnJVvTzJbyfZmeRskn84xvg/ixkTAGBjWs2RqT1jjN1jjOXp44NJTo0xXp3k1PQxAMC20jnN95Ykx6b3jyV5a38cAIDNZdaYGknuraoHq+rAdNnSGOPxJJneXr+IAQEANrKZrplK8oYxxter6vokn62qR2f9BtP4OpAkS0tLmUwmq5+Sbcl7BZi3c+fO2bcwdzPF1Bjj69Pbb1TV3Ul+LMkTVXXDGOPxqrohyTf+itceSXIkSZaXl8fKyspcBmeLO3ki3ivAvE0mE/sW5u6yp/mq6qqquvr5+0nelOSRJPckuW262m1JPr2oIQEANqpZjkwtJbm7qp5f/7fGGCer6nNJfqeq9if5oyQ/tbgxAQA2psvG1Bjjy0luvsTyP02ydxFDAQBsFj4BHQCgQUwBADSIKQCABjEFANAgpgAAGsQUAECDmAIAaBBTAAANYgoAoEFMAQA0iCkAgAYxBQDQIKYAABrEFABAg5gCAGgQUwAADWIKAKBBTAEANOxY7wHYXqpq9nV/ZfbtjjFewDQA0OfIFGtqjDHT1+nTp2deV0gBsJ7EFABAg5gCAGgQUwAADWIKAKBBTAEANIgpAIAGMQUA0CCmAAAaxBQAQIOYAgBoEFMAAA1iCgCgQUwBADTUGGPtvlnVk0m+smbfkM3suiRPrfcQwJZj38JqvHKM8UOXW2lNYwpmVVUPjDGW13sOYGuxb2ERnOYDAGgQUwAADWKKjerIeg8AbEn2Lcyda6YAABocmQIAaBBTrImq2llVjzS38eaqOjivmYCtrarOVtV16z0HW9+O9R4AZjXGuCfJPes9BwBcyJEp1tKOqjpWVQ9X1V1V9ZKq+qWq+lxVPVJVR6qqkqSq/nlVfWm67n+cLvu5qvrI9P5SVd1dVV+Yfv2d9fyHAeurqn62qv5nVT1UVb9WVVdc8NzOqnq0qj4+3dd8qqr+blX9XlU9VlU/Nl3v5VX1n6f7nf9RVa9dv38Rm4mYYi3dmOTIGOO1Sb6Z5J8m+cgY42+PMW5KcmWSfzBd92CSvzVd959cYlu/muS/jjFuTvK6JP9r4dMDG1JV7Ury00neMMbYneS5JP/4otX+ZpI7k7w2yQ8n+UdJbknyr5K8d7rO+5N8frrfeW+SX1/89GwFYoq19NUxxu9N7/9mzu/I9lTV/VX1xSQ/keRHps8/nORTVfWzSZ69xLZ+IslHk2SM8dwY4+nFjg5sYHuT/GiSz1XVQ9PHr7ponf89xvjiGOP7Of+fr1Pj/K+zfzHJzuk6tyT5jSQZY/yXJK+oqmvWYH42OTHFWrr4czhGkv+Q5O1jjNck+ViSvzZ97tYk/z7nd5APVpXr+4C/SiU5NsbYPf26cYzxvovW+b8X3P/+BY+/n7+4frgusW2fH8RliSnW0t+oqtdP7+9Lct/0/lNV9YNJ3p4kVfUDSf76GON0kn+d5GVJfvCibZ1K8gvT9a+oqpcuenhgwzqV5O1VdX3y59c+vfIFbOe/ZXp6sKpWkjw1xvjm3KZky/K/fdbSmSS3VdWvJXks50/TXZvzh9nPJvncdL0rkvzm9PB6JfnwGOPPptemP+9fJDlSVftz/vqIX0jy39fiHwFsLGOML1XVv01y7/Q/Y88k+WcvYFPvS/LJqno4yXeS3Da/KdnKfAI6AECD03wAAA1iCgCgQUwBADSIKQCABjEFANAgpgAAGsQUAECDmAIAaPh/+ReqfeBo4KgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "accuracies = pd.DataFrame({'basic' : basic_accuracies, 'elmo': elmo_accuracies})\n",
    "plt.rcParams['figure.figsize'] = (10,6)\n",
    "accuracies.boxplot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAFpCAYAAAC4SK2+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsnXtc0/X+x1/buN/H/TqRVFRERbkIy3uamacsTc3ywk55fnYRsczMk5XdswI0PdlleM3U8lKZeczSkosionjNzAswkPsdxm7f3x+072FswIDBLryfj8ceY2Pf7z4b47vX9315vTkMw4AgCIIgCILoGlxjL4AgCIIgCMKcITFFEARBEATRDUhMEQRBEARBdAMSUwRBEARBEN2AxBRBEARBEEQ3IDFFEARBEATRDUhMEQRBEARBdAMSUwRBEARBEN2AxBRBEARBEEQ3IDFFEARBEATRDax688k8PT2Z4ODg3nxKgiAIgiCILpGdnV3GMIxXR4/rVTEVHByMs2fP9uZTEgRBEARBdAkOh3NHn8dRmo8gCIIgCKIbkJgiCIIgCILoBiSmCIIgCIIgugGJKYIgCIIgiG5AYoogCIIgCKIbkJgiCIIgCILoBiSmCIIgCIIgugGJKYIgCIIgiG5AYoogCIIgCKIbkJgiCIIgCILoBiSmCIIgCIIgugGJKYIgCIIgiG5AYoogCIIgCPMhdy+QNAx43a35OnevsVcEK2MvgCAIgiAIQi9y9wLfLwPkjc23q/ObbwPA8DlGWxZFpgiCIAiCMGlUKhXKyspwedcrOP5HLb66KIekRtX8S3kjcHydUddHkSmCIAiCIHodhmFQVVWFu3fvori4uN3rkpISKBQKje33z7HHIy5/x4Sq843wCv6HRYqpgzkSrD/6BwqrGuHvZo+V94diZkSAsZdFEARBEBYNwzCoqanpUBwVFxejuLgYMplMax/W1tbw8fGBj48P/Pz8MHLkSPj6+sLHxwe+v78CFxsV/ntTgXsFvP9txOFp7ac3sTgxdTBHgtX7L6JRrgQASKoasXr/RQAgQUUQBEEQXaCurk6vCFJxcTGkUqnW9jweD97e3qwoGjZsWLM4UoukFtd8Ph8cDkdrHwzD4KsTL+Op76WQ1DKICbDC3GF/R6YYZU+/Be1icWJq/dE/0ChXou7yr1BU3QWHy0M1l4fEnO+Qf/8QWFlZaVx4PJ7WfR1d9N2Gy+Xq/EAQBEEQhLFpaGhgI0QtBZEukdTQ0KC1PYfDgZeXFyuEBg0apCWM1D97eHiAy+16mXZWVhYSEhKQkdGI0X5c7JltD6GghYRxDeryvg2BxYmpwqrmCv/6i8chvXOevb8KwLLDvb+eroq2nhR53dmmK48nQUkQBNE7NDU16RU9unv3Lmpra3Xuw9PTkxVDY8aM0Rk98vHxgaenJ6yselZGFBUV4ZVXXsHWrVvh4+MD8Rv/h0Wc/eAqW0S/rO2ByWt7dB0dYXFiyt/NHpKqRnjPfRNgVIBKBUalhJ+LNX58XgiFQqHzolQq2/ydobbpyuNlMhkaGhq69BxyudzYfw4AAJfLNTlR2NsCtztnZARB9G1kMhlKSkr0EklVVVU698Hn81kRNHr06DZTbF5eXrC2tu7lV6hNU1MTkpOT8dZbb0Emk2HVqlV45ZVX4OLiAuRObO7eqy4AXAObhZQRbREACxRTK+8P/V/NFIcHcHlwsLbD6ofD4eHhYezl9ToqlcqogtBQzyGVSru8f5VKZew/AzgcjkmKvN58Dkp7E8T/UCgUKC0t1SuCVFFRoXMfLi4urBAaPny4VmpNfe3t7Q1bW9tefoVdg2EYfPfdd3jhhRfw119/4eGHH8aHH36IAQMG/O9Bw+cYXTy1xuLElLrInLr5muFyubCxsYGNjY2xl2I0VCqVluDqjaiioR/fOkLZmedQKo1bnKnGFEVeb4pPSntbNkqlEuXl5XoJpLKyMjAMo7UPJycnVggNHjwYEyZM0BlF8vHxgb29vRFeZc9x6dIlJCYm4ueff8bQoUPx3//+F1OmTDH2svSCo+uP2VNERkYyZ8+e7bXnIwiiGYZhOoxSmkoUsqfWZCpp7/aElqmJwp5alzkJSpVKhYqKCr1SbCUlJToj4fb29m3WHbW+dnR0NMKrNC4VFRV47bXX8J///AcuLi5Yt24d/u///g9WVsaP93A4nGyGYSI7epzxV0oQRI/D4XDA4/HA4/HMJtzfE+gjKE099S2Xy9HY2Njl/ZtC2rt1HWVvC0kejweFQoHGxkY0Njaivr4eDQ0NqKurQ11dHWpqalBTU4Pq6mrU1NTofM+srKzg4eEBd3d3eHl54Z577oGXlxe8vb01xJG/vz9cXV1hbW2tsR6qo2xOdW7ZsgVr165FVVUVli5dijfeeKPjkpzcvVQzRRAEYSwo7W2YtLcpRCF1NebI5XI0NTVBJpNBJpNBLpez+1IqlVCpVDpTa11BoVCwtgJXr17t9PbqOsq+FJFseTl58iQSExNx5coVTJo0CcnJyQgPD+/4jTPR2XyU5iMIgiBMlrq6Or1b/dszi2xdb6SOInl4eMDT0xPu7u5wcnJqN3ppjqKzrW1MJe3N5XK1onbtCraK67BiZLDicvD+fba4V+015RoEJF4y+PoozUcQBEGYJG2ZReq6rq+v19pebRapFkgDBw7U2cXm6+vbbbNIS0bfOkpDCLy6ujr89NNPOHHiBLhcLiZMmIDo6GgA6Nz+pVegUHGgVAG8lqV31QXGeRP/hsQUQRAE0W0MYRbp4eHBCqExY8a0WajdG2aRfYHeSHurVCrs2LEDb7zxBu7evYtFixbhnXfegb+/f9d2mDRM91Bj18DuLbSb0KeRIAiC0IlcLkdJSYleAqk9s0i1CBo1alSbXWze3t4mYRZJGI6MjAwkJCQgKysLY8aMwaFDh9hoVJeZvFazZgogB3SCIAiid1Eoms0i9YkilZeX69yHi4sLK4SGDRuG++67T6dIMiezSMJwSCQSvPzyy9i5cyf8/f2xc+dOPP7444ZJt6qLzE2sm48K0AmCIMwcQ5hFOjo66u2FZGlmkYRhaGxsxEcffYR3330XSqUSK1euxKpVq+Dk5GTspXUZKkAnCIIwY1QqFSorK3H37t0um0Xa2dmxhdkhISGIi4tr003bnL/wCOPCMAy+/fZbvPjii7hz5w5mz56NDz74AP379zf20noNElMEQRC9BMMwqKqq0ivFVlxcDIVCobUPGxsbVggJBAJERUW1GUFydnY2K7dxwvy4cOECEhIScPLkSQwfPhy//vorJkyYYOxl9TokpgiCILoBwzCora3VK8VWXFwMmUymtQ8rKys2QuTr64sRI0a0KZDc3NxIIBFGp7S0FK+++io+//xz8Pl8fPrpp3jqqafA4/GMvTSjQGKKIAhCB901i+RyufD29mZF0NChQ9usQ+Lz+eSFRJgFcrkcmzZtwuuvv476+nosW7YMa9euBZ/PN/bSjAqJKYIg+gyNjY1tCiJ9zSI9PT3ZOiS1WaQukeTh4dFnz9IJy+Snn35CYmIirl27hvvvvx9JSUkYMmRI7y+EZvMRBEEYFrVZpD5RpJqaGp378PDwYIVQTExMmwLJy8uLzCKJPsf169exYsUKHD58GAMHDsQPP/yA6dOnGyfdnLsXim+eRm6xEgPduXCuzgf2L2n+nREFFR0VCIIwOQxhFunm5saKoFGjRrWZYvPy8urTg48Joi2qq6vx5ptvIiUlBQ4ODvjwww/x/PPP9/r/S3V1NTIzM5Geno60ne/idIEcdTLg4Fx7PDzYGgADHHqWxBRBEJaPIcwinZ2d2RSb2iyyLTdtOzu7Xn6FBGEZKJVKiMVirFmzBmVlZXjqqafw1ltvwdvbu8efm2EY3L59G+np6Th16hROnTqFS5eaBxhzOBwEuwKR/lyAAdxa+sEqtRs7epMOxRSHwwkFsKfFXSEA1gJwA/A0gNK/73+FYZgfDb5CgiBMFpVKhbKysg4FUntmkQ4ODqxACg0Nxfjx49v0QnJwcDDCqySIvsNvv/2GhIQEnD9/HmPHjkVKSgoiIiLYYcWNjY1aF6lUisbGRtTV1aGqqgpVVVWora1FTU0NampqUFdXh/r6etTX12ts19TUhKamJkilUjQ1NUGhUOj0S1PDMAxuVQG3qpqPIx9nyjC+v2mMIOpQTDEM8weAkQDA4XB4ACQADgCIB5DEMMyHPbpCgiB6FYZhUFFRoVeKrbS0FEqlUmsfarNIHx8fhISEIDY2ts06JDKLJIjOoVAotISMLnHT1v0NDQ2ora1FbW0t6uvrUVdXh8rKSuTl5aG2thZcLheOjo44d+4cxowZA7lcrvNEyNBwuVw4ODjA0dERrq6u4PP5cHJyAofDgUQiwc2bNyGXy+FpDwz15mLFGNMQUkDn03yTAfzFMMwd8jkhCPPB0GaRgYGBiIyMbLMOicwiib5CS2Gjj5DprOhRX9T3NTU16TyBMSQqlUqjm9XGxgY2NjawtbWFnZ0d7O3tYW9vDwcHBzg5ObEXFxcXuLq6wtXVFW5ubnBzc4NUKsXNmzdx5coVXLhwATdu3ADQLJxGjRqF6OhoDB06FEFBQVCpVBrHoitXriA3N1cr7V/WCPx2R4UKqekcYzorpuYB2N3i9nMcDmchgLMAXmAYptJgKyMIol3UZpFttfbraxbZ0gtpxIgRbQokMoskTJ3WwqY7Qkbfx+o68dAXLpfLXlr+b6lUKqhUqk6LJltbW1boODo6wsnJCc7OzhpCx9nZGY6OjnB0dMS1a9dw8OBBVFZWYsKECVi2bBnuueceODk5sdvb29vr7YHW1NSEc+fO4eTJkzhw4ACysrJQWdksC+zs7ODt7Y2wsDBwuVw0Njbijz/+QFvzeq2srKBQKMDj8TBkyBDExcU1W5Fc2AgfbiV8nTgY4N5iXa5BnXqvDI3eYorD4dgAeAjA6r/v+g+ANwEwf19/BECkY7slAJYAgEAg6OZyCcLyqa+v19tNu7GxUWt7tVmkWgQNGTKkzRSbu7s7mUUSPYJcLu9xIdP6vu5EbKysrGBtbQ1ra2tYWVmBx+OBx+OBw+GwQodhGKhUKjAMA2tra3A4HMjlcshksnZrfVpjY2OjEdFRC5fu3HZ0dNTb1yw7OxsJCQlIS0vDqFGj8N133+Hee+9tdxuZTIaSkhKN489ff/2F3Nxc/PnnnygqKkJtbW2b6UCpVIqamho4ODiwXm2t0/1paWnYt28fioqKEBwcjISEBCxevFizFCC3P/D9MkDe4thnbd/sNWVEOPrmQTkczsMAnmUYZqqO3wUD+IFhmGHt7SMyMpJpS4UShCXTnllk6+uOzCLbihypr8kskmiNWtj0ZKSm9X3dETbqdJKtrS1sbW3ZNFNLkaOO5qiFTstojkKhgEKhgEwmg1wuZ4ucGxsbIZfL9V4Hj8fTEC2GEj7W1sap9bl79y7WrFmD1NRUeHl54a233sL06dNRWlraoZFtRUVFu/t2dnaGn58f7rnnHgwfPhz33HMP21jSXoft9evXsWHDBmzduhX19fWYOHEiEhMT8eCDD7Z9oteLpp0cDiebYZjIDh/XCTH1NYCjDMOk/n3bj2GYor9/TgQQwzDMvPb2QWKKsCQMYRbp7u6ul0Ais0jLgGEYnamonorUqO/rrrBRp47s7e01ambUt9XRHCsrK420lfr7RS1yVCoV5HI5e5HJZJBKpRprVxdE64q6tgWHwzGo2FH/bGNjY3apbaVSifLyco1jkEQiwZEjR5Ceng6lUgkPDw8AQHl5eZsdtj4+PnB0dATDMKirq0NJSQn7N3FxcUFUVBTGjx+PiRMnIjIyslNWJAzD4Ndff0VSUhIOHz4Ma2trzJ8/H8uXL8eIESMM80YYCIOKKQ6H4wAgH0AIwzDVf9+3A81dfgyA2wD+pRZXbUFiijB11GaR+owcac8ssiNxpD5TI7NI48EwDORyeY8LmdaXzqSDWtNa2OgSN+2JHisrKzaa01LoKBQKNqLTUuQ0NTWx4kbd3q7+uWW7e2c6vVoWLRtK+Njb25ud6OkMKpUKFRUVep24lZSUtPkZs7e3x6BBgxAcHKx1TLK2tkZ+fj6uXr2Ks2fPIjs7m43ihYaGQigUspdBgwZ16f2WSqXYvXs3kpOTkZubCy8vLyxduhRLly6Fr69vt96jnsLgkSlDQGKKMAZKpbLNMHbr647MIjsSSWQW2TXUwqY3CoYNJWzs7Oz0FjKdET329vbg8XhsfY46ZSWXy9HQ0NCusOnodmciVLa2tu0Kma6IHwcHB6rR+xt1h60+x6WSkpIOO2xbX8vlcuzevRtZWVkYNGgQNmzYgPvvvx9Aszi7cuUK0tLSkJaWhvT0dPz1118Amv/uUVFRrHCKjY2Fp6dnt15rcXExPv30U2zevBklJSUIDw/H8uXLMX/+fJM/XpKYIiwafc0ii4uLUVpa2q5ZpD5RpL5kFtla2PRGGsoQwsZQQkafx9ra2oLL5UKhULDRmfaETGfFT2fqeqysrNgOLUOltxwdHSmt3AXUHbb6NpDo02Hb3rWuDtuKigq8/vrr2Lx5M5ydnbFu3TosWLAAOTk5rHDKyMhgI+ve3t6scIqLi8OoUaNga2urta6ukJubi+TkZOzatQsymQwPPvggEhMTMWnSJLOJJJKYIswOQ5lF6iOOfH19zcIsUpew6Y3ojaGETU8Imdb32dnZdXhgVqlU3Y7q6LotlUr1fl+4XK7ewqYz4odSxT1PXV2d3g0kuj4TrTts27vuaoetQqHAZ599hldffRWVlZWYMmUK+vXrh5ycHOTk5LDHy7CwMA3xdM899xhU2KhUKvz4449ISkrCL7/8AgcHByxevBgJCQkYNGiQwZ6ntyAxRZgEDMOgurpa7zM1XaFsa2trvSNILi4uPXbGwzAMZDJZr0VqekLY9HT0xtbWtlvvP8MwkEqlnRI2+jymoaGhU+swZJRHfVsf0Uf0Ho2NjXofl0y5w1apVOLLL7/E66+/jqKiItjZ2bGCzt7eHjExMaxwio2NBZ/P75F11NXVYdu2bUhJScGff/6JwMBAPPfcc3j66afh7u5u2CczwW4+iuMSnaa1WWRH17pC2Twej5235uvri/DwcJ0HobZC2S2FjfoikUhw48aNHk1Pdefkoz0h4ubmBj8/P4OKnu4Km/ZQv/9q4VJUVGSQqE99fX2nxKO9vb1O4eLt7d3lqI+9vf4mhYRp0VaHra7jUm1trc59tOywjYmJaVMkGavDtra2FpmZmUhLS8Px48eRmZnJnoS6u7tj8uTJrHgaOXJkj9sw5Ofn45NPPsFnn32GqqoqREdHY/fu3Zg1a1bPPHfuXk2fqer85ttAjwkqfaDIFMHSXbNIDocDDw8PeHp6wt3dHXw+Hy4uLmw9h4ODA1u0y+VyNURKV0SPIYRNb9XZ9KSw6Qi5XM4Klu6mt1re1xnn55YmhYaK+nTGpJAwX9Qdtvocmyytw5ZhGOTl5bG1TmlpacjNzWVPODgcDng8Hh599FGsW7euy112XeH06dNISkrCN998A4ZhMGvWLCQmJiI2Ntagz1NbW4v8/Hzk5+cjLy8P+fvWIL+0CnnVKrwzyRYxgX8LWtcgIPGSQZ8boMhUn4VhGA2DusrKShQUFKCwsBCFhYUoKSlBSUkJysrKUFFRgcrKSlRXV6OmpqbNoteWrsBcLpc10WtpjMcwDBiGQVlZGcrKyjq15vZECJ/Ph7+/v0FFjzGFTXsolUqNuh5D1fY0NTXpvYbWJoVq4eLj44N77rmny7U+xjIpJEwThUKB0tJSvU7c9OmwHTZsGCZPnmz2HbYKhQLnz59nhVNaWhokEgkAwMnJCTExMXjooYfw22+/oaKiAgsWLMC7774Lf3//Xlvf/v37kZSUhMzMTLi6uiIxMRHPPfcc+vXr1+n9yWQySCSSZpGkFksthVN+vpZA5gDwd+YgyJULacvzueqC7r24bmKRYupgjgTrj/6BwqpG+LvZY+X9oZgZEdDr61ALG0PX09TX16O2tpatBWlsbB5+KZfLuzUnisPhwNramvWyUY8ocHBw6LHojTma4jEMg4aGBoMVMasvuqJ9baE2KWwtXPh8PoKCgroc9THHvwdhGugyi2zruqysrN0OW19fX4SGhmL8+PE6I0iW0mFbVVWFjIwMVjydPn2are8TCAQYO3YsWyxeX1+PF154AcePH0dMTAxSUlIQExPTa+v8/PPPsXHjRuTn52PAgAHYuHEjFi1aBGdnZ53bqIcWtyWS8vLyUFxcrPU5cHd3h0AgQHBwMMaOHQuBQICgoCD22n/fdFjX6RBOroE98dL1xuLE1MEcCVbvv4hGeXPngqSqEav3XwTDMHhgqGevtntLpdIup6I4HA47OqHluAS1uZ4ubGxs4OHhAWdnZ7i6uoLP57NpN3UniZ+fH/z9/eHi4qIlcCzti1QtZg3Rqt7ydldNClsKF/VZdVfTXfb2lm1SSJgGKpUKlZWV7QqjjswiW3bY9u/fH7GxsTprI9Xz2SwVhmFw8+ZNjajT5cuXwTAMeDweRo4ciX/+85+seAoMbBYHEokEq1evxo4dO+Dn54ft27fjiSee6JW6vj///BMbNmxAamoq6uvrMWHCBHzyySeYPn066urqcOvWLZ0iKT8/HwUFBVrZDgcHB1YYTZ8+XUMkqS+Ojo7tL2rqa+Y9m88Q9EbNlPC9XyCpakTZDx9BeucCGIWMvXQVDofTpYiMra0tVCoVZDIZWyzdMo1TVVWFqqoqlJeXo7q6WudzOzs765Xr9/HxMZtQti7kcrnB01t1dXUGNSnsym0HBweq6yFMCrVZpD4pNlPvsDVlZDIZzp07pyGeiouLATSPY4mNjWWFU3R0tJaQlEql+Pjjj/HOO+9AoVDgxRdfxMsvv9zjgpNhGBw9ehQffPABfv31V/B4PISHh6N///5oaGhgxVJdXZ3GdjweD4GBgawoah1RCgoKgru7u2E+CybYzWdxYqr/y4fBAKg+/S0UFRJwrKzBsbIFx8oGq2YM71LtjXo6ONB8ptaZULauMzVzNotUKpUGjfKoL501KeyqJ097t8mkkDBXTMEssq9TXl6OjIwMVjhlZWWxFgUhISGIi4tjxdPQoUPbPMliGAb79+/Hiy++iNu3b2PWrFlYv349+vfvb5B1KhQKFBUVaUWUbt++jdzcXEgkEp0C2tvbW0MYtf7Z19fXIk8c+2wBur+bPSRVjXCNmaVxf4CbPVa/PEnnNmqzSH3nHumKdtja2rIHm+Dg4HZbap2cnHr8QNTSpNCQtT1dMSlsLVw8PT0RHBzcZfFjSt02BNGT1NUZ1ixyyJAhBjeL7IswDIPr169rRJ2uXbsGoFmUjho1CkuXLmUtCvz8/PTab25uLhISEnDixAmEh4fjl19+wcSJEzu1rvLycp1pN/XPhYWFWt9hNjY2UCqVUCqV4PP5GD9+PGbMmIGQkBAEBQUhMDDQrDMfvYHFiamJg72wMzOvubusqR7K+koo66vgo+Bhw4ZLbc490hUZsba2Zg82AQEBGD16dJtnal0NZbc2KTTUWApdJnPt0XqUhJOTE1xdXREQENDlqA+ZFBKENo2NjXoLpPbMItXHngEDBrQpkDw9PS0yWtDbSKVSZGdna8yyU3ct8/l8xMXFYeHChRAKhYiKioK9vX2n9l9WVoZXX30Vn332Gfh8Pv7zn//gqaee0oqW19XVtVmjpL60bmSxsbFhI0gTJkxgI0pKpRI///wzDh8+DJlMhunTpyMxMRGTJ0+m43YXsLg0n7pmqvjrf0N657zW71ubRbZ3zefz2Q+V2qTQ0B1cnTUptLOz65Sw0ecxZFJIEN2jLbNIXdc1NTU699HSLLK9a2OZRfYlSkpKNKJO2dnZbGp04MCBbLpOKBQiNDS0y8dPuVyOzZs34/XXX0dNTQ0WLVqEWbNmoaqqSmcHXGVlpcb2HA4Hfn5+bdYoCQQCeHl5setTqVQ4cuQIkpKScPz4cTg4OGDRokVISEhAaGho9940C6XP1kwFv3wYAFB74SiUNSXg2DiAY20HjpUNPlkQAw6Ho3f6q/V9XTEpNPTwUTrLJIjeoS+bRfYlVCoVrl27xgqntLQ03LhxA0DzcTwyMlJjlp2Xl1eXnqOkpERDGP3+++84duwYamtrYWtry/r1tYTP57cpkoKCguDv76/X56a+vp4d9XL9+nUEBATg+eef75lRLxZGn62Z4nE4UDIMGq7+BumdCxq/m39Ex+NbmBS2FC4+Pj4ICQnpUtTH0dGRDowEYYIoFAqUlZV1OGqkI7NItRAKCwuzCLPIvkRDQwOysrJY4ZSRkcFGfDw9PSEUCrFkyRIIhUKMGjVKr79hdXV1u8aTBQUFOov+ra2tMXLkSIwcORL9+vXTEksd2gR0QOtRL1FRUfjqq68we/ZsMtI1MBYnppR/K3vn0f+Aw+Cx4NrYgWNtD46NHb5bPllL+JiqGzZBEPqhUqlQVlamV4qttLS0XbNIHx8fhIaGYty4cWbTYUu0T1FRkUbUKScnh80yDBkyBLNmzWKjTgMHDtT6PpBKpSgoKGjTeDI/P19rzh+Px0NAQACCgoIQExOD2bNnw9PTE+np6fjhhx9gZ2eHtWvXIiEhoUdOvHWNelm+fDliY2Pp+66HsLg0n7pmqjUBbvZIa6ObjyAI00LdYatPiq20tLTdDlt96pAs2SyyL6FUKnH58mWNQvFbt24BaK43jY6OZlN2Y8aMgZubm06bgJbCqaSkROt5vLy82k2/+fn5sSUZSqUSW7duxSuvvILS0lKIRCK8/fbb8PHxMehrVygUOHDgAJKSkpCRkQEXFxc8/fTTeP7557s06sWkMUGfKYuLTK28P1TDAR0A7K15WHk/FdcRhDExtFlkYGAgRo8eTWaRfZi6ujqcPn2aFU+ZmZlsgb+Pjw+ioqLwyCOPwNfXFzY2NigsLMTt27fx22+/IT8/HxKJREuIOzk5QSAQQCAQYNSoUVrF3Z2xCTh16hQSEhJw7tw53HvvvThy5AhGjRpl0PegqqoKX3zxBTZu3Ii8vDzcc8892LBhAxYvXtzmqBezJncvVIeex8kbdYgN4sGuOr/3DQEuAAAgAElEQVTZER3oMUGlDxYXmQJMZzYfQVg6arPIjgRSe2aR+nbYklkkkZ+fzwqn33//HRcvXmS7ob29vcHn82FlZYXGxkYUFRVp2QRYW1u3aTqp/tnV1bXbn7G8vDy89NJL2LNnD4KCgrB+/XrMmTPHoJ/dGzduICUlRWPUy/LlyzFjxgyLbVTKy8vDtn9FIjWzDLeqGOyeZY95w/6u/XINAhIvGfw5+2xkCgBmRgSQeCKIblBfX6+3m7auAc1cLhdeXl6sGBoyZEibIonMIonWyOVy3LlzBydPnsRvv/2G8+fP4+bNm1ojTFpSWloKHo8HgUCAIUOG6LQL8Pb27tHPWkNDAz744AN88MEHAIDXX38dK1euNFidHcMwOHHiBJKSkvDDDz/AysoK8+fPR0JCAiIiIgzyHKaGVCrFoUOHIBaLcezYMTAMg/tCeHh7kg0eDm0hYap1DD/uRSxSTBEEoY2hzSKFQiGZRRKdRqVSobS0VKtG6ebNm7h27RokEolWQTfQbFPg7++PkJAQhIWFoV+/fhoRpYCAAKN1UTMMgz179uCll15Cfn4+5s2bh/fffx8CgcAg+29qasLXX3+N5ORknD9/Hp6envj3v/+NZ555Br6+vgZ5DlMjJycHYrEYu3btQmVlJQQCAdauXYvFyq8QzCvW3sA1sPcX2QISUwRhxjQ1NbFeSIYwi4yOjiazSKJb1NTUtGsTkJ+fr5Xu5XA4Gl2W6tE30dHRmDJlCmJiYky2SSA7OxsJCQlIS0tDREQEdu3ahbFjxxpk3yUlJfj000+xefNmFBcXIywsDF988QXmz5/faZd1c6C8vBxfffUVxGIxzp8/D1tbWzz66KMQiUSYNGlSc1Qxd2hzjZS8RUTc2r65CN2I0JGRIEwMtVmkPlGk1o7IalxdXVkhFBER0aZAIrNIojM0NTVp2AToEkutRTuPx4O/vz/4fD5cXFwQHByMu3fvso9zdHTUGAIcExNjFoXTxcXFWLNmDcRiMby8vPDFF19g8eLFBonIXrx4EcnJydi1axeamposetSLeqyNWCzGwYMHIZPJEBkZiU2bNuHxxx8Hn8/X3EBdZN5L3Xz6YpEF6ARhaiiVSpSWluqVYtPHLLK9ax8fHzKLJDqNUqnE3bt32xRJeXl5bdoEtC7k5vP5qKqqwp07d5Cbm4usrCy2ti44OFhDPA0bNsysUsIymQwbNmzAunXrIJVKkZCQgH//+99wdXXt1n7Vo16Sk5Px888/w97enh31MnjwYAOt3nT466+/sHXrVmzduhUFBQXw8PDAggULEB8fj+HDhxt7eSx9dpwMQfQWhjaL7OiazCKJrsIwDCorK9s1npRIJFp2FE5OTu3OfVPbBNy4cUNjlt2VK1cAAFZWVoiIiNAQT/7+/sZ4C7oNwzA4fPgwVqxYgT///BMzZszARx99hEGDBnVrv/X19di+fTtSUlLwxx9/ICAgAM899xyWLFlicaNe6uvr8e233yI1NRUnTpwAl8vFtGnTIBKJMGPGDNja2hp7iVr06W4+gugqarPItlr7O2MW6ePjg/79+2PMmDFtmkeaah0IYV40NDS0W6OUl5eHhoYGjW2sra0RGBiIoKAgjB07Viu6JBAIdNoENDU1ITs7G4cOHWKNMdURKzc3N8TFxWH+/PkQCoWIiorq9kgUU+Dq1atITEzE0aNHMXjwYBw5cgTTpk3r1j4LCgrYUS+VlZUWO+qFYRicPn0aYrEYX3/9NWprazFgwAC88847WLhwIQICLKPznsQUYfEwDIPq6mq9UmwlJSWQy+Va+7C2tmYFUEBAAEaPHt1mFInMIglDIpfLUVhY2K5Lt67UsK+vLwQCAcLCwvDAAw9oiSUfHx+9bAJKS0uRkZHBRp3Onj2LpqYmAMCAAQMwbdo0Nuo0ZMgQi7K5qKysxBtvvIFPPvkEzs7OSElJwdKlS7slds6cOYOkpCTs27cPDMPg0UcfxfLlyxEXF2dRx43i4mLs2LEDYrEYV69ehYODA+bMmQORSIR7773Xol4rQGKKMFP0NYtUX+tjFhkeHt5mio3P51vcPz9hfBiG0WkT0FI4FRUVscaUatzc3FhhNGbMGK2IUldtAhiGwR9//KExy+769esAmk8oIiMj8dxzz7Gz7Aw9EsVUUCgU+Pzzz/Hqq6+isrIS//rXv7Bu3Tp4enp2eX8HDhxAcnIy0tPT4eLigoSEBDz//PMIDg427OKNiEKhwJEjRyAWi/HDDz9AoVAgLi4OX3zxBebMmWMWjQVdhcQUYVKQWSRhSdTU1LSZdlP/rI7yqLG1tWWF0ZQpU3S6dBvqS6mxsRFnz57VmGVXUVEBAPDw8EBcXBxEIhGEQiEiIyP7RGPDr7/+ioSEBFy8eBETJkxASkpKlwuiq6qq8OWXX2Ljxo24c+cOQkJCkJKSgvj4eIsSFteuXYNYLMaOHTtw9+5d+Pj4YMWKFYiPj7fI4nldkJgiehy1WaQ+I0d0mUUCgKenJyuEhEKhzlEjZBZJ9CZNTU2QSCTt2gRUV1drbMPlcuHv7w+BQIDRo0dj5syZWsXdnp6ePRYFLS4u1og6nTt3jk1rh4aGYubMmWzKbtCgQX0qGnvr1i28+OKL2L9/P4KDg/Htt9/ikUce6dJ7cOPGDWzYsAGpqamoq6vD+PHjkZKSYlGjXmpqarB3716IxWJkZGTAysoKM2bMgEgkwrRp0yyq7ksfqJuP6BItzSI7EkntmUXq08Xm5eXV5/4xCeOiUqk6tAkoLtZ2Yfb09Gx37pu/v3+vGZ+qVCpcuXJFQzzdvHkTQHP0KyoqihVOsbGxXU5hmTt1dXV499138dFHH8HKygqvvPIKVqxY0ekoHMMwOHnyJJKSkvD999/DysoKjz/+OBISEgw+3NhYMAyD33//HWKxGPv27UNDQwOGDh0KkUiEJ5980iLTvtTNR3QaQ5tFjhw5kswiCZODYRhUVVW1G1GSSCRajQiOjo6sMBo+fLhWRCkwMNCo9hX19fU4c+YMK5wyMjLYyJi3tzeEQiGeeeYZCIVCREREmGQbem+iUqmwa9curFq1CkVFRViwYAHefffdTneXtTXqZenSpfDz8+uh1fcuEokE27ZtQ2pqKm7cuAFnZ2c8+eSTEIlEiI6O7lMRzLYgMWXhqM0i9Wn1b8ss0snJiRVCYWFhmDx5MplFEiZLY2NjuxGl/Px8rXSylZUVAgMDIRAIIBQKdUaX3NzcTOpLQyKRaESdzp8/z1p1hIWFYe7cuWzkKSQkxKTWbmxOnz6NhIQEnD59GtHR0di/fz/GjBnTqX2Ulpayo17u3r2LsLAwfP7553jiiScsYtRLU1MTvv/+e4jFYhw9ehQqlQoTJ07E2rVrMWvWLPK9awWl+cwQlUqF8vJyvVJsZWVlWp1AAGBvb9+m91FrgWQJPjGEZaBQKDq0CSgrK9PaztfXV6ePkvo+Hx8fk65lUSqVuHjxooZ4ysvLA9D8vxwTE8MKpzFjxmiP4CAAAIWFhVi9ejW2b98OPz8/vPfee3jyySc71Yhy6dIlJCcnY+fOnWhqasIDDzyAxMRE3HfffRYhWHNzcyEWi7Fz506Ul5cjMDAQ8fHxWLx4MUJCQoy9vF6H0nxmRltmkbquS0pKOjSLDA4ORkxMTJsiycnJySL+8QnLgWEYlJWVtWs8WVhYqHVy4Orqyoqj6OhorYhSQECA2aW0amtrkZmZyXbYZWZmora2FgDg7+8PoVCIFStWQCgUYsSIEVRT2AFSqRRJSUl4++23IZfLsXr1aqxevVrvjjqVSoWffvoJSUlJ7KiX+Ph4LFu2DEOGDOnh1fc8lZWV2L17N8RiMbKzs2FjY4OZM2fin//8JyZPnmzSJxqmAompHoTMIgnif9TW1rYbUcrPz4dUKtXYxtbWlhVGkydP1lnU7eLiYqRXZBgYhkFeXp6GPUFubi5UKhU4HA6GDx+OBQsWsN5O/fr1o/9zPWEYBgcPHsQLL7yAW7du4dFHH8X69ev1jrC0HvXi7++Pd955B0uWLIGHh0cPr75nUalU+OWXXyAWi7F//340NTVh5MiR2LBhA+bPn2/2r6+3ITHVSRiGQV1dnd5eSK09ZIBms0hvb29WCIWHh7cpkMgskjAHZDIZCgoK2vVUqqqq0tiGy+XCz88PAoEAERERePjhh7XEkpeXl8V9/uVyOS5cuMAKp7S0NEgkEgDN9YljxozBq6++iri4OIwZM8bsxaKxuHjxIpYvX45ffvkFw4YNw/HjxzFp0iS9ti0oKMCmTZuwZcsWVFZWIjIyErt27cJjjz1m9lHA27dvY+vWrUhNTUVeXh74fD6WLFmC+Ph4REREGHt5ZotFiqmDORKsP/oHCqsa4e9mj5X3h2JmRPsdGvX19Xq7abdnFqkWQaGhoW2m2Dw8PMgskjAbVCoViouL23XpLi4u1hrk7OHhgaCgIPTv3x/jxo3Tqlfy8/Mz+y8mfaiqqmLHsaSnp+P06dPsnDyBQIBx48axg4DDw8N7zTrBUikrK8PatWuxZcsWuLm5YfPmzXj66af1el+zsrLYUS8qlQqPPPIIEhMTzX7US2NjIw4cOACxWIzjx4+Dw+Fg6tSpWL9+PR566CFqHDIAFleAfjBHgtX7L6JRroRK3gRVQxWsmmrweLgL+jsq2hRJdXV1OvfX0iyyvWsyiyTMEXUquj2bgIKCAq0UtIODg85C7pY/98VuH4ZhcPPmTY2o0+XLl8EwDHg8HkaOHMmm64RCIQIDA429ZItBLpfjP//5D1577TXU1tbi2WefxWuvvQZ3d/d2t1MoFDh48CCSkpLYUS9PPfUUnnvuOfTv37+XVm94GIbB2bNnIRaLsXv3blRXV6N///4QiURYuHAhBAKBsZdoFvTZAvT1R/9Ao1yJ4n2vQXozm73/vRaP4fP5rBCKioois0jCYmlsbERBQUG7NgGtTySsrKwQEBAAgUCA2NhYnWKJ0s/NyGQynDt3TkM8qc08XV1dERsbizlz5kAoFCI6OhpOTk5GXrFl8t///hfLly/H1atXMWXKFCQnJ2Po0KHtblNdXY0vvvhCY9RLcnIy4uPjzTq1Wlpaip07d0IsFuPSpUuwt7fH7NmzIRKJMG7cOMqK9BAdiikOhxMKYE+Lu0IArAWw/e/7gwHcBjCHYRjdTo69SGFVcwrOMfRe2AUMBdeRD56jG6wc+Tj91ix4e3ubXWcPQehCoVCgqKioXU8lXTYBPj4+CAoKwpAhQzB16lSt9Jup2wQYk/LycqSnp7PCKSsriy2aDwkJwdSpU9moU1hYGH1x9TB//vknXnjhBXz//fcYMGAAvvvuO8yYMaNdof/XX39hw4YNEIvF7KiX5ORk/OMf/zDbz71CocDRo0chFovx/fffQy6XIyYmBlu2bMHcuXPh6upq7CVaPB2KKYZh/gAwEgA4HA4PgATAAQAvAzjOMMx7HA7n5b9vr+rBteqFv5s9JFWNcBo+ReP+ADd7BAUFGWlVBNE5GIZBeXl5uxGlwsJCLYsMFxcXVhhFRkbqdOmmkwn9YBgG169fZ4VTWloarl27BqC5y3bUqFFYunQpm7azFLdrc6CmpgZvvfUWkpOTYWdnhw8++ADLli1r87PNMAx+++03JCUl4bvvvoOVlRXmzZuH5cuXm/Wolz///BOpqanYtm0bCgsL4eXlhWXLliE+Ph5hYWHGXl6forNpvskA/mIY5g6Hw3kYwIS/798G4ARMQEytvD+UrZlSY2/Nw8r7Q424KoLQpK6urkObgNaNDjY2NqwwmjhxolaNUlBQEJ2BdgOpVIrs7GwNiwJ1ZI/P5yMuLg4LFy6EUChEVFSURbhcmxsqlQpbt27F6tWrUVpaivj4eLz99tvw9fXV+fimpibs2bMHycnJyMnJgYeHB9asWYOlS5fC39+/l1dvGOrq6rBv3z6IxWKcOnUKPB4P06dPx6ZNm/Dggw9SaYqR6KyYmgdg998/+zAMUwQADMMUcTgcb4OurIuou/Y6281HEIZCJpNBIpG0K5ZazzbkcDjw9/dHUFAQRowYgX/84x9aYsnLy4vSRgakpKREI+qUnZ0NmUwGABg4cCBmzJjBuoqHhobSe29k0tLSkJCQgOzsbMTFxeHw4cOIjNRdF9x61MvQoUPx2Wef4cknnzRLEcwwDNLT0yEWi7Fnzx7U19cjNDQU77//PhYsWEBRURNA724+DodjA6AQQBjDMMUcDqeKYRi3Fr+vZBhGa4YBh8NZAmAJAAgEgtF37twxzMoJwgioVCqUlJS0axNw9+5dLZsAd3f3NrveBAIB/P396YyyB1GpVLh27ZrGOJYbN24AaI74RUZGssIpLi4OXl5eRl4xoSY/Px+rVq3C7t27ERgYiA8++ADz5s3TWRfVetTLtGnTkJiYiClTpphlw0RRURG2b98OsViM69evw8nJCXPnzoVIJEJsbKxZviZzQ99uvs6IqYcBPMswzNS/b/8BYMLfUSk/ACcYhmk3l0az+QhTRx+bAHX0Qo2Dg0O7c9+CgoJovmEv09DQgKysLFY4ZWRksNFALy8vtkhcKBRi9OjRVEdmgjQ0NGD9+vV4//33wTAMVq1ahZUrV2r9L6lUKhw9ehRJSUk4duwY7O3tsXDhQiQkJJjlqBeZTIbDhw9DLBbjyJEjUCqVGDt2LEQiEWbPnk0dob1MT1gjPI7/pfgA4DsAi9DsOrAIwKFOrZAgehmpVKphE6BLLKnnn6nh8XgIDAxEUFAQYmJi8Nhjj2kJJ3d3dzpDNDJFRUUaUaecnBwoFAoAwJAhQzBr1ixWPA0YMID+XiYMwzDYu3cvVq5cifz8fMydOxfvv/8++vXrp/G4+vp67NixAykpKbh27ZrZj3q5fPkyxGIxduzYgdLSUvj7+2PVqlVYvHgxBg4caOzlER2gV2SKw+E4AMgHEMIwTPXf93kA2AtAACAPwGMMw1S0tx+KTBE9hVKp7NAmoLS0VGs7b2/vNiNKAoEAvr6+ZtsubakolUpcvnxZQzzdvn0bAGBnZ4fo6GhWOMXGxnZo2kiYDufOnUNCQgJOnTqFiIgIpKSkYOzYsRqPkUgk+OSTT9hRL6NHj0ZiYiIee+wx2NjYGGnlXaO6uhpff/01xGIxzpw5A2trazz00EMQiUSYOnUqueGbAAaNTDEM0wDAo9V95Wju7iOIHoVhGFRUVLRrEyCRSLRsApydnVlxNHr0aK2IUmBgII1RMAPq6upw+vRpVjhlZmaipqYGAODr6wuhUIhly5ZBKBRi5MiRZveFSjQ3A6xZswZffvklPD098fnnnyM+Pl7jRCYrKwvJycnYu3cvVCoVZs6cicTERAiFQrOKNKpUKpw8eRJisRjffPMNpFIpwsPDkZSUhCeeeILq9cwUkr2E0amvr283otSWTUBgYCAEAgHGjx+vM7pENgHmSX5+voY9wYULF6BUKsHhcDBs2DDMnz+fjTwFBweb1RcpoYlMJsPGjRuxbt06NDQ0YMWKFXj11VfZ/12FQoFDhw4hKSkJaWlpcHZ2xvPPP4/nn3/e7Ea95OXlYdu2bUhNTcWtW7fg6uqK+Ph4iEQijB49mj7HZo7FzeYjTAu5XN6hTUBFhWZ2mMPhwM/Pr825bwKBgGwCLASFQoHc3FwN8ZSfnw8AcHR0RExMDNthN2bMGLi5uXWwR8IcYBgGP/74IxITE/Hnn3/iwQcfxEcffYTQ0OYepurqanz55ZfYsGED7ty5g/79+yMhIcHsRr1IpVIcOnQIYrEYx44dA8MwuO+++yASiTBz5kyztGnoa/TZ2XxE76FSqVBaWtquTUBRUZGWTQCfz2eFUVxcnFZEyd/fn1I1Fkp1dTUyMzNZ4ZSZmYn6+noAQGBgoIY9wYgRI6hmxAK5evUqVqxYgZ9++gmhoaH48ccf8cADDwDQHvUybtw4JCUl4aGHHjKr2sWcnByIxWLs2rULlZWV6NevH1577TUsWrQIwcHBxl4e0QPQkYpok+rq6g5dulvbBNjb27PC6P7779eKKAUGBlJrbx+BYRjcvn1bI+p08eJFMAwDLpeLESNGID4+nrUpoCn2lk1lZSXWrVuHTz75BI6OjkhKSsKzzz4LKysrnDx5EsnJyTh06BCsrKwwd+5cLF++HKNHjzb2svWmvLwcu3btglgsxoULF2Bra4tZs2ZBJBJh4sSJFEm3cEhM9VGampo6tAlQF/mq4fF4CAgIQFBQEKKjozFr1iytVJyHhwfl/vsocrkcOTk5rHBKS0tDUVERgOZmgNjYWMyaNQtxcXGIiYmBs7OzkVdM9AZKpRJffPEF/v3vf6OiogJPP/003nzzTbi6uuLrr79GUlISO+rllVdewTPPPGM2o16USiV+/vlnfPnllzh06BBkMhkiIyOxefNmzJs3D3y+lo81YaGQmLJAlEol7t69225Rd0lJidZ2Xl5eEAgEGDhwICZNmqSVfvPz8zOrUDvRs1RUVCAjI4MVT2fOnGEbBYKDgzFp0iQ26jRs2DD67PRBTpw4gYSEBOTm5mL8+PFISUlBQEAAPv30U2zatElj1MsTTzwBBwcHYy9ZL/766y9s3boVW7duRUFBATw8PPDMM88gPj4ew4cPN/byCCNAYsrMYBgGlZWV7UaUJBIJa1ioxsnJCQKBAAKBABEREVoRpcDAQCqGJNqEYRjcuHFDI+p05coVAICVlRUiIiLwr3/9ixVP5hJZIHqGW7duYeXKlfj222/Rr18/7Nu3D4MHD0ZKSgp27twJqVSKadOmYfny5Zg6dapZRLPr6+vx7bffIjU1FSdOnACXy8W0adOQnJyMf/zjH1Tn2cchMWViNDQ0dGgT0NDQoLGNtbU1K4rGjh2r04DS1dXVLA5YhGnQ1NSE7OxsVjilp6ez0Uw3NzfExcXhiSeeQFxcHKKjo80mokD0LHV1dXjvvffw4YcfgsfjYd26dQgPD8fmzZtx7Ngx2NnZsaNehg4dauzldgjDMDh9+jTEYjG+/vpr1NbWYsCAAXjnnXewcOFCBAQEGHuJhIlAYqoXkcvlKCwsbFMs5efno7y8XGMbDocDX19fBAUFITw8HNOnT9eyC/D29qbiRqJblJaWsim7tLQ0nD17Fk1NTQCAAQMGYNq0aWyn3ZAhQ+jzRmigUqnw1VdfYdWqVSgsLMTcuXMxcuRIbNu2DdeuXYOfnx/efvttLFmyBJ6ensZebocUFxdjx44dEIvFuHr1KhwcHDBnzhyIRCLce++9dGJKaEE+UwaCYRiUlpa2G1EqKiqCSqXS2I7P52v5KLX8OSAggMLHhEFhGAbXrl1jo05paWm4fv06gOYoZ2RkJJuui4uLg4+Pj5FXTJgyZ86cQUJCAjIzMzFixAgMHz4chw8fRkVFhVmNepHL5Thy5AjEYjEOHz4MhUKBuLg4iEQizJkzhxom+ij6+kyRmNKTmpqaDm0C1Gfyauzs7Nqd+xYUFEQ2AUSP09jYiKysLI2Undoo1cPDgxVOQqEQkZGRNGKH0IuioiKsXr0a27Ztg4eHBwYOHIisrCyNUS/mEMW5evUqUlNTsX37dhQXF8PX1xcLFy5EfHw8Bg8ebOzlEUamT5t2HsyRYP3RP1BY1Qh/N3usvD8UMyPazm2rbQJ0pd3UP1dXV2tsw+Px4O/vj6CgIERGRuKRRx7REk5kE0AYg7t372pEnc6dOwe5XA4AGDx4MGbOnMmKp0GDBtFnlOgUUqkUycnJePvttyGVShEYGIiCggLIZDJ21EtISIixl9kuNTU12Lt3L8RiMTIyMmBlZYUZM2ZAJBLhgQceILNYotNYXGTqYI4Eq/dfRKO8eegtw6hg3VSDJaNc0N9eqlMsFRcXa+3H09Oz3YiSn58f/cMRRkelUuHy5csa4unmzZsAAFtbW0RFRbHCKTY21izqVQjThGEYHDp0CImJibh9+zYcHBzQ0NCA/v37Y9myZRCJRCY96oVhGPz+++8Qi8XYt28fGhoaMHToUIhEIjz55JOUziZ00mcjU+uP/oFGuRLlRzeh8dY5KGvLAJUSq1o8xsnJiRVGI0aM0OnSTTYBhClSX1+PM2fOsMIpIyODjZp6e3tDKBTimWeegVAoxKhRo0y+ToUwDy5duoR//etfSE9PZ5sP1PVQpj7qpaCgANu3b0dqaipu3LgBZ2dnPPnkkxCJRIiOjqbILGEQLE5MFVY1mwbyHN1gFzgUPGdPWLl4wcrFC8defRQCgYBsAgizQSKRsMIpLS0N58+fh1LZHHUNCwvD3Llz2chTSEgIfa4Jg1JWVoYlS5bgwIEDAAAul4t58+ZhxYoVJj3qpampCd9//z3EYjGOHj0KlUqFiRMnYu3atZg1axZZeRAGx+LElL+bPSRVjXC79wmN+wPc7MmZljBplEolLl68qCGe8vLyADTPPIyJicHLL78MoVCIMWPG0KgKoseor6/H//3f/2H37t1QKpWwtbXF0qVLsXLlSpM2ZL1w4QJSU1Oxc+dOlJeXIzAwEGvWrMHixYtNvo6LMG8sTkytvD9Uo2YKAOyteVh5f6gRV0UQ2tTW1iIzM5MVTpmZmairqwMA+Pv7QygUYsWKFRAKhRgxYgSsra2NvGLC0ikrK8PKlSuxc+dOKBQKODg44MUXX8SqVatMNppTWVmJr776CqmpqcjOzoaNjQ0eeeQRiEQiTJ482aRTkITlYHFiSt2115luPoLoaRiGQV5enkbU6eLFi1CpVOByuQgPD8fChQtZb6d+/fpRyo7oNa5cuYI33ngD33zzDVQqFezt7bFmzRqsXbvWJA1aVSoVfvnlF4jFYuzfvx9NTU2IiIjAxo0bMX/+fLi7uxt7iUQfw+K6+QjCFJDL5bhw4QIrnNLT0yGRSAA0N0CMGTOGrXWKiYkx6S4owjJhGAZHjx7F+vXr8csvvwBoNm197rnn8O6778LW1tbIK9Tm9u3b2Lp1K1JTU0lA9aAAACAASURBVJGXlwc+n48nn3wS8fHxiIiIMPbyCAukz3bzEYQxqKqq0hjHcubMGXaGokAgwLhx49ioU3h4ONlqEEajoaEBO3bsQHJyMq5du8ZGnh5//HF89NFH8PPzM/IKNWlsbMSBAwcgFotx/PhxcDgcTJ06FevXr8dDDz1EJrOESUBHdILoJAzD4ObNmxpRp8uXL4NhGPB4PIwcORJPPfUUK54CAwONvWSCgEQiwaZNm7BlyxZUVFSwNVDR0dHYsGEDoqKijLzC/8EwDM6ePQuxWIzdu3ejuroaISEhePPNN7Fo0SIEBQUZe4kEoQGJKYLoAJlMhnPnzrHCKS0tjTV6dXV1RWxsLObOnYu4uDhER0fTiCDCpMjOzkZSUhL27NkDhUKBwMBAVFRUgM/n4/PPP8fjjz9uMvV5paWl2LlzJ8RiMS5dugR7e3vMnj0bIpEI48aNM8n6LYIASEwRhBbl5eVIT09nhVNWVhakUikAICQkBFOnTmXn2YWFhdEBnjA5lEolDh06hKSkJJw6dQpOTk6IiorC+fPnUVZWhldffRWrVq2Co6OjsZcKhUKBo0ePQiwW4/vvv4dcLkdMTAy2bNmCuXPnwtXV1dhLJIgOITFF9GkYhsH169c1ok7Xrl0D0FyMO2rUKCxdupRN2ZlaPQlBtKSmpgZffvklNmzYgNu3b6Nfv35YuHAhfv31V2RkZOCxxx7DBx98gODgYGMvFdevX0dqaiq2bduGoqIieHl5YdmyZYiPj0dYWJixl0cQnYLEFNGnkEqlOHv2LCuc0tPTUVZWBgBwd3dHXFwca1EQFRVFY4UIs+DmzZvYuHEjvvzyS9TW1uLee+/Fs88+i0OHDmH79u0YMWIEduzYgfHjxxt1nXV1ddi3bx/EYjFOnToFHo+HBx98ECKRCNOnTycvNcJsITFFWDQlJSUaUafs7GzIZDIAwKBBgzBjxgzWoiA0NJRSdoTZoB7cm5ycjEOHDoHL5WLu3LlYuHAhvvnmG7z00kvw8PDAli1b8M9//tNo5pUMwyA9PR1isRh79uxBfX09QkND8f7772PhwoXw9fU1yroIwpCQmCIsBpVKhatXr7LCKS0tDTdu3AAA2NjYICoqCgkJCWzKzsvLy8grJojOI5PJsHfvXiQlJeHcuXNwd3fHyy+/jKeeegoHDhzAY489hoaGBixfvhxr166Fm5ubUdZZVFSE7du3QywW4/r163BycsK8efMgEokQGxtrMkXvBGEISEwRZktDQwPOnDnDiqeMjAxUVlYCALy8vBAXF4clS5ZAKBRi9OjRJmlCSBD6UlZWhi1btmDTpk0oKirC4MGD8emnn2LBggU4ceIEpk2bhuvXr+OBBx7Axx9/jMGDB/f6GmUyGQ4fPgyxWIwjR45AqVRi7NixWL16NWbPnk2droTFQmKKMBsKCws1ok45OTlQKBQAgKFDh2LWrFlsym7AgAF05ktYBFeuXEFKSgq2b98OqVSKqVOnQiwWY+rUqbh+/Tpmz56NI0eOYNCgQTh8+DCmT5/e62u8dOkSUlNTsWPHDpSWlsLf3x+rVq3C4sWLMXDgwF5fD0H0NiSmCJNEqVTi8uXLGrPsbt++DQCws7NDdHQ0Vq5cCaFQiNjYWJrFRVgUDMPgv//9L5KSknD06FHY2dlhwYIFSEhIQFhYGKqqqvDiiy9i48aNcHBwwMcff4xnn30WNjY2vbbG6upqfP311xCLxThz5gysra3x8MMPQyQSYcqUKeTyT/Qp6NNOmAR1dXU4ffo0K5wyMzNRU1MDAPD19YVQKMSyZcsgFAoxcuTIXv3SIIjeorGxkR31cvXqVfj5+eGtt97CkiVL4OXlBaVSic8++wxr1qxBeXk5nn76abz55pvw9vbulfWpVCqcOHECYrEY3377LaRSKcLDw5GcnIwnnngCnp6evbIOgjA1SEwRRiE/P18j6nThwgWoVCpwOBwMGzYM8+fPZ1N2wcHBlLIjLJrCwkJ21Et5eTkiIiKwfft2zJ07lz1xOHnyJBISEnDhwgWMGzcOycnJvTbcNy8vD9u2bUNqaipu3boFV1dXiEQiiEQijBo1iv4/iT4PiSmix1EoFMjNzdUQTwUFBQAAR0dHxMTEYM2aNRAKhYiJiTFa9xFB9DbZ2dlITk5mR708/PDDSExMxNixY1mBcvv2bbz00kvYt28fBAIB9u7di9mzZ/e4gJFKpTh06BDEYjGOHTsGhmFw33334e2338bMmTPJg40gWkBiijA41dXVyMzMZIXT6dOnUV9fDwAIDAxkI05CoRDDhw+n2gqiT6FUKvHdd98hKSkJv//+O5ycnPDMM89g2bJlCAkJYR9XX1+P9957Dx9++CE4HA7WrVuHF198scdFTE5ODsRiMXbt2oXKykr069cPr732GhYtWmQSzukEYYrQtxjRLRiGwe3btzWiTpcuXQLDMOByuRgxYgTi4+NZbyeBQGDsJROEUaipqYFYLMaGDRtw69YtBAcH4+OPP4ZIJNKYP8cwDL766iusWrUKEokE8+fPx3vvvYegoKAeW1t5eTl27doFsViMCxcuwNbWFrNmzYJIJMLEiRPJzJYgOoDEFNEp5HI5cnJyWOGUnp6OoqIiAICzszNiY2Mxe/ZsCIVCREdHw9nZ2cgrJgjjcuvWLWzYsIEd9SIUCrF+/Xo8/PDDWlHZrKwsJCQkICMjA6NHj8aePXsgFAp7ZF1KpRLHjh2DWCzGoUOHIJPJEBkZic2bN2PevHng8/k98rwEYYmQmCLapaKiAhkZGaxwOnPmDBobGwEAwcHBmDRpEpuyCwsLM9rICoIwJRiGwalTp5CcnIyDBw+Cy+Vizpw5WL58OaKiorQeX1RUhFdeeQVbt26Fj48PxGLx/7d33+FRVO0bx79DkyZNQAQBRRClKfwAMRGlKCK9SW9ZUBCEJBTpAtLbuwm9TuhFmiBIEVB4pffeEZBeQwkh9fz+IOYVBQkkYVPuz3XlymYzO/OMkuydc2bOQ7NmzWJlROjkyZORDYbPnTvHSy+9RJs2bfDw8KBIkSIxfjyRxEBhSiIZYzhx4sRDvewOHToEQLJkyShatCitWrWKnLLLnj27iysWiVuCg4OZP38+TqeTnTt3kilTJrp06ULbtm3JkSPHP7YPCgrCx8eH/v37ExwcTJcuXejevTvp0qWL0boCAgJYuHAhtm2zfv16kiRJQsWKFfHx8aFq1apaakQkmhSmErGgoCB27twZGZ42bdrElStXAMiQIQNubm40atQINzc3SpYsSerUqV1csUjcdP369chWLxcuXHio1cujfm6MMSxdupSOHTty8uRJqlevzvDhw8mbN2+M1WSMYevWrdi2zdy5c7lz5w558+Zl4MCBNG3a9JHhTkSeTZTClGVZGYDJQCHAAA7gU+AL4GrEZt2NMT/FRpESM65evRoZmjZu3MiOHTsICgoCIG/evHz22We4ubnh7u7O22+/rYtORZ7g8OHD+Pj4MGPGDAIDA6lQoQKTJ0/m008/fezPz4EDB/D29mbNmjUUKFCA1atX88knn8RYTZcvX2bGjBnYts3hw4dJnTo1devWxeFw8MEHH2hNKJFYENWRKV9gpTGmjmVZKYDUPAhTTmPM8FirTp6ZMYYjR448NGV37NgxAJInT07x4sVp164dbm5uuLm58fLLL7u4YpH44c9WLz4+PqxcuZIXXniBJk2a4OXlRcGCBR/7uhs3btC7d2/GjRtHunTpGDVqFK1bt46RpUFCQkJYsWIFtm2zbNkywsLCcHNzY/LkydStW1c3gojEsif+FFuWlQ74EGgOYIwJBoL1103cEhgYyPbt2x+asrtx4wYAL730Eu7u7jgcDtzd3SlevDgpU6Z0ccUi8UtgYCAzZ87Ex8eHQ4cOkS1bNvr160erVq3IkiXLY18XGhrKhAkT+Pbbb/H39+err76ib9++vPTSS9Gu6fDhw/j5+TF9+nQuX75MtmzZ6NixIx4eHrz11lvR3r+IRE1U/iTKw4OpPD/Lst4BdgKeEd/72rKspsAOoKMx5mbslCl/d+nSpcgRp40bN7Jr1y5CQkIAeOutt6hZs2bklN2bb76poX2RZ3Tx4kXGjBnD+PHjH2r1UrduXV544YV/fe2aNWvw8vLi4MGDlCtXDh8fHwoXLhytem7fvs3333+Pbdts3ryZZMmSUbVqVRwOBxUrVtQiuCIuYBlj/n0DyyoObAHcjTFbLcvyBW4Do4FrPLiGqh/wijHG8YjXfwl8CZArV67/O3PmTMyeQSIQHh7OwYMHHwpPp06dAiBlypSUKFEiMji9//77ajYqEgN27dqF0+l8qNWLl5cXH3744RP/ODl58iQdO3ZkyZIl5MmThxEjRlC9evVn/qPGGMOGDRvw8/Nj/vz53Lt3jwIFCtCiRQsaN2783BodiyQ2lmXtNMYUf+J2UQhT2YAtxpjXIr4uDXQ1xlT+yzavAcuMMYX+bV/Fixc3O3bseGLxiV1AQADbtm2LDE6bN2/m1q1bAGTNmvWhdizFihXTbc0iMeTPVi8+Pj5s2LCBtGnT4nA4aN++PW+88cYTX3/nzh0GDBiA0+kkefLk9OzZEy8vr2eeVj937hzTp0/Htm1OnjxJunTpaNCgAQ6HgxIlSmjEWSSWRTVMPXE82BhzybKsPyzLym+MOQqUBw5ZlvWKMeZixGY1gQPRKznxOnfu3EOjTnv27CEsLAyAggULUq9evcjwlCdPHv0CFYlht2/fxs/Pj5EjR3Lq1Cly587NiBEjaNGixUOtXh4nPDyc6dOn061bNy5dukSzZs0YOHDgM63FFhQUxI8//oht26xatYrw8HDKli1Lnz59qFWrlpYoEYmDojq53g6YFXEn3ynAAxhpWda7PJjmOw20ipUKE5iwsDD279//UC+7s2fPApAqVSree+89unbtiru7O6VKlVJLB5FY9PvvvzNq1CimTJnC7du3cXd3Z+jQoY9s9fI4mzdvpn379uzYsYNSpUqxZMkSSpYs+dS17N27Fz8/P2bOnMn169fJmTMnPXr0oHnz5g81QBaRuCdKvy2MMXuAvw9zNYn5chKe27dvs3Xr1sjgtGXLFu7evQtA9uzZcXd3p0OHDri7u/POO++QPHlyF1cskrAZY9i4cSNOpzOy1cvnn3+Ot7f3I1u9PM65c+fo2rUrs2bNInv27MycOZMGDRo81fpsN2/eZPbs2fj5+bFz505SpEhBzZo1cTgclC9fXu2ZROIJ3fYRg4wxnD179qFRp/379xMeHk6SJEkoXLgwTZs2jZyyy5Url6bsRJ6TP1u9+Pj4sGPHDjJmzMg333xD27ZtefXVV6O8n8DAQEaMGMGgQYMICwujZ8+edOnShbRp00bp9eHh4axduxbbtlm8eDFBQUEULVqUUaNG0bBhQzJlyvSspygiLqIwFQ0hISHs3bv3ofB04cIFANKmTUupUqXo1asX7u7uvPfeezHeb0tEnuz69etMnDiR0aNHc+HCBfLnz8+4ceNo0qQJadKkifJ+jDEsXLiQTp06cebMGerUqcPQoUN5/fXXo/T633//nalTpzJ16lTOnj1LxowZ+fLLL/Hw8KBo0aLPenoiEgcoTD0Ff39/Nm/eHBmctm3bxr179wDIlSsXH330UeSoU6FChbTei4gLHT58GF9fX6ZPn05gYCCffPLJE1u9PM6ePXvw8vJi/fr1FClShF9++YUyZco88XWBgYEsWrQI27ZZt24dlmVRoUIFhg0bRrVq1bR4rkgCoXf7xzDGcOrUqcjgtGnTJg4ePIgxhqRJk/Luu+/SsmVL3N3dcXNze6ppAhGJHcYYfv75Z5xOZ2Srl8aNG+Pl5UWhQv+6cssjXb16lZ49ezJ58mQyZszI+PHjadmy5b9ey2SMYceOHdi2zZw5c7h16xZ58uShX79+NGvWjJw5c0bnFEUkDlKYihAcHMyuXbseCk+XL18GIH369Lz//vvUq1cPNzc3SpYsGeXrI0Qk9v291cvLL7/Md999R+vWrf+11cvjBAcHM2bMGPr27UtAQADt27fn22+//de7a69evcrMmTOxbZsDBw6QKlUq6tSpg8Ph4MMPP1TjcJEELEGGqR92n2fYqqNc8A8ke4ZUdP40PzWK5nhom+vXr0eu7bRp0ya2b9/O/fv3AciTJw8VKlSIHHUqWLCgfhGKxEEXL15k7NixjB8/nmvXrvHuu+8ybdo06tWr98RWL4+zYsUKvL29OXr0KJ9++ilOp5O33377kduGhoayatUqbNtm6dKlhIaG8t577zFhwgTq1asXpTWqRCT+S3Bh6ofd5+m2aD+BIQ8WvTzvH0jXhfs4f/okqW6eiAxPR44cASB58uQUK1aMNm3a4ObmhpubG6+88oorT0FEnmD37t04nU7mzp1LaGgo1apVw8vLi48++uiZ75A9evQoHTp04KeffiJfvnwsW7aMSpUqPXJ/x44dw8/Pj2nTpnHx4kWyZMmCp6cnHh4eFCxYMLqnJyLxTIILU8NWHSUwJIygC0e5f3Y/QecPE3T+MF8H3gYgU6ZMuLm50axZM9zc3ChRogSpUqVycdUi8iRhYWH8+OOPOJ3OyFYvX331Fe3atSNv3rzPvF9/f3/69evHyJEjSZ06NcOHD6ddu3b/aNN09+5d5s+fj23b/PbbbyRNmpTKlSvjcDioVKmS1ogTScQSXJi64B8IgP9/Z3L/9G6SZcpBqrwlSZnjbf77n6/Inz+/puxE4pE7d+5g23Zkq5dcuXIxfPhwWrRoQYYMGZ55v2FhYdi2TY8ePbh27RotW7akf//+DzUNNsawadMmbNtm3rx5BAQEkD9/foYOHUqTJk3Ili1bTJyiiMRzCS5MZc+QivP+gWT6pDVJUqYlaeoH1yzkyJDqsdc9iEjcc/r0aUaNGsXkyZO5ffs2bm5uDBkyhBo1akR72ZENGzbg6enJnj17KF26NL6+vg+t9XTx4sXIBsPHjh0jbdq0kQ2GS5UqpcV2ReQhCS5Mdf40P90W7YdM/7vgPFXypHT+NL8LqxKRqPhzJMjpdLJ48eLIVi9eXl7P1O/u786cOcM333zD999/T86cOZk3bx6ff/45lmURHBzM8uXLsW2bFStWEBYWxocffkj37t2pU6fOUy3wKSKJS4ILU3/etfeku/lEJO4ICQlh/vz5OJ3OyFYvnTt35uuvv46RNdwCAgIYOnQoQ4cOxbIs+vbtS6dOnUidOjUHDhzAz8+PGTNmcPXqVbJnz06XLl1o3rw5+fLli4GzE5GELsGFKXgQqBSeROK+GzduMGHChMhWL2+++SZjx46ladOmMTISZIxhzpw5dOnShXPnztGgQQOGDBlCunTpmDFjBrZts23bNpInT0716tVxOBxUqFBBDYZF5KkkyDAlInHbkSNH8PX1Zdq0aQQGBvLxxx8zadIkKlasGGM3iOzYsQNPT082bdpEsWLFmDVrFqGhoXTr1o2FCxdy//59ChcujI+PD40aNSJz5swxclwRSXwUpkTkuTDGsGbNGpxOJytWrIh2q5fHuXTpEt27d2fq1KlkyZKFoUOHcu/ePZo1a8bp06fJkCEDDocDh8NBsWLFdDG5iESbwpSIxKrAwEBmzZqFj48PBw8ejGz10qpVq4eWIYiuoKAgfH196d+/P4GBgVSuXJmAgAC6dOkCQPny5Rk4cCA1atTQ2nIiEqMUpkQkVvy91cs777zD1KlTqV+//jO3enkUYww//vgj3t7enDp1itdee40bN26wbNkycufOTe/evWnWrBmvvfZajB1TROSvFKZEJEbt3r0bHx8f5syZQ2hoKFWrVsXb2ztarV4e5+DBg7Rt25b169dHBrSLFy9Su3ZtHA4HZcuW1SK9IhLrFKZEJNrCwsJYtmwZTqeT9evXkyZNGlq3bk379u2j1erlca5evUrLli358ccfMcYAUKhQIVq0aEH9+vXJmDFjjB9TRORxFKZE5JnduXMHPz8/Ro4cycmTJ2Os1cvjHDlyBG9vb1atWoUxhpQpU9KsWTPatGlDkSJFYvx4IiJRoTAlIk/tUa1eBg0aRM2aNaPd6uXvAgICWLhwISNGjGDfvn3Ag4blPXr04Ouvv/5HQ2IRkedNYUpEouTPVi8+Pj4sWrQIy7L4/PPP8fb2jpFWL38/1tatW7Ftmzlz5nD37l0AMmbMyNChQ2nRooWWNBCROENhSkT+VUhICAsWLMDpdLJ9+/bIVi9t27YlZ86cMXqsy5cvR65MfvjwYZInT05YWBgpU6akV69edOjQgZQpU8boMUVEokthSkQe6caNG0ycOJHRo0dz/vz5GG/18qeQkBBWrFiBbdssW7aMsLAw8uXLR7p06bh9+zZNmzZl0KBBZM+ePcaOKSISkxSmROQhf2/1Ur58eSZMmMBnn30Wo8sMHD58GD8/P6ZPn87ly5fJli0bDRs2ZPfu3Rw4cID33nsPX19f3nvvvRg7pohIbFCYEpFHtnpp1KgRXl5eFC5cOMaOc/v2bb7//nts22bz5s0kS5aMqlWrUr16dVavXs2MGTN45ZVXmD59Oo0aNdIaUSISLyhMiSRigYGBzJ49Gx8fHw4cOMDLL79M3759ad26dYy1ejHGsGHDBvz8/Jg/fz737t2jQIECjBgxgtq1azNz5kzatGlDWFgYPXr0oGvXrqRNmzZGji0i8jwoTIkkQpcuXWLs2LGMGzeOa9euUaRIEfz8/GjQoEGMtXo5d+4c06ZNw8/Pj5MnT5IuXTqaNGmCw+GgePHiLF68mDJlynD69Glq167NsGHDeP3112Pk2CIiz5PClEgismfPHpxOZ2SrlypVquDt7U2ZMmViZKmBoKAgli5dim3brF69mvDwcMqWLUufPn2oVasWqVOnZu/evZQvX55ff/2VwoULs27dOsqWLRsDZyci4hoKUyIJ3J+tXnx8fPj1119JkyYNrVq1on379uTLly9GjrF3715s22bmzJncuHGDnDlz0qNHD5o3b06ePHmABy1gOnbsyMSJE8mYMSPjxo2jZcuWMb7Ip4jI86bfYiIJ1J07d5g6dSq+vr6RrV6GDRtGy5YtY6TVy82bN5k9eza2bbNr1y5SpEhBzZo1cTgclC9fnqRJkwIPlj4YO3Ysffr04c6dO7Rr147evXurf56IJBgKUyIJzJkzZyJbvdy6dYv333+fgQMHUqtWrWiPAoWHh7N27Vps22bx4sUEBQVRtGhRRo0aRcOGDcmUKdND269cuRJvb2+OHDlChQoVcDqdFChQIFo1iIjENQpTIgmAMYbNmzfjdDofavXi5eUVI+s0/f7770ydOpWpU6dy9uxZMmbMyJdffomHhwdFixb9x/bHjh2jQ4cOLF++nLx58/Ljjz9SuXJltYARkQRJYUokHvuz1YuPjw/btm0jQ4YMdOrUia+//jrarV4CAwNZtGgRtm2zbt06LMuiQoUKDBs2jGrVqj2yrcutW7fo168fI0eOJGXKlAwbNoz27durGbGIJGgKUyLx0I0bN5g0aRKjR4/m3Llz5MuXjzFjxtC0adNordFkjGHHjh2RDYZv3bpFnjx56NevH82aNXtsQAsLC8PPz4/u3btz7do1HA4HAwYM4OWXX37mWkRE4guFKZF45OjRo5GtXu7du0f58uUZP358tFu9XL16lZkzZ2LbNgcOHCBVqlR8/vnnOBwOSpcu/a/7/u9//4unpye7d+/mgw8+YOXKlRQrVuyZaxERiW8UpkTiOGMMa9euxel08tNPP5EiRYrIVi9FihR55v2GhoayatUqbNtm6dKlhIaGUqpUKSZOnEjdunVJnz79v77+7NmzfPPNN8ybN4+cOXMyd+5c6tatq+uiRCTRUZgSiaPu37/PrFmzIlu9ZM2alT59+tC6detoTZ8dO3YMPz8/pk2bxsWLF8maNSteXl54eHhE6U67e/fuMXToUIYMGYJlWfTp04fOnTuTOnXqZ65JRCQ+U5gSiWMuXbrEuHHjGDduHFevXo1s9VK/fv1HXvQdFXfv3mX+/PnYts1vv/1G0qRJqVy5Mg6Hg0qVKpE8efIn7sMYw7x58+jcuTPnzp2jfv36DBkyhFy5cj1TTSIiCUWUwpRlWRmAyUAhwAAO4CgwD3gNOA3UNcbcjJUqRRKBPXv24OPjw5w5cwgJCaFKlSp4eXlRtmzZZ5o6M8awadMmbNtm3rx5BAQE8NZbbzF06FCaNGlCtmzZoryvnTt34unpycaNGylatCizZ8+mdOnST12TiEhCFNWRKV9gpTGmjmVZKYDUQHdgrTFmsGVZXYGuQJdYqlMkQQoPD2fZsmU4nU5+/fVXUqdOzRdffIGnp+czt3q5cOEC06dPx8/Pj2PHjpE2bVoaNGiAw+GgVKlSTxXMLl26RI8ePfDz8yNLlixMnjyZ5s2bR65uLiIiUQhTlmWlAz4EmgMYY4KBYMuyqgNlIjabBvyKwpRIlNy9exc/Pz9GjhzJiRMnyJkzJ0OHDqVly5bP1GYlODiYZcuWYds2K1asIDw8nA8//JDu3btTp04d0qRJ81T7CwoKYuTIkfTr14/79+/TsWNHevbs+cSL0kVEEqOojEzlAa4CfpZlvQPsBDyBl40xFwGMMRcty8oae2WKJAxnzpxh9OjRTJo0iVu3blGqVCkGDBjwzK1eDhw4gG3bzJgxg2vXrpE9e3a6du1K8+bNn2lkyxjDsmXL6NChAydOnKBKlSqMGDGCN99886n3JSKSWETlt3cyoBjQzhiz1bIsXx5M6UWJZVlfAl8CulBVEq2/tnoBqFOnDl5eXpQqVeqp9+Xv78/cuXOxbZvt27eTPHlyqlevjsPhoEKFCs88BXfo0CG8vb1ZvXo1b731FitWrKBixYrPtC8RkcQkKmHqHHDOGLM14usFPAhTly3LeiViVOoV4MqjXmyMmQhMBChevLiJgZpF4oWQkBAWLlyIurG/fwAAIABJREFUj48PW7duJUOGDHTs2JG2bds+9R8W4eHh/Prrr9i2zcKFC7l//z6FCxfGx8eHRo0akTlz5meu8+bNm/Tp04cxY8bw4osv4uvry1dffRWlO/xERCQKYcoYc8myrD8sy8pvjDkKlAcORXw0AwZHfF4Sq5WKxBM3b95k4sSJD7V6GT16NM2aNXvqVi9nzpxh2rRp+Pn5cfr0aTJkyIDD4cDhcFCsWLFoLZAZGhrKpEmT6NWrFzdv3qRVq1Z899130QpmIiKJUVQv0mgHzIq4k+8U4AEkAb63LKsFcBb4PHZKFIkfjh07hq+vL1OnTuXevXuUK1eOcePGUalSpadq9XL//n1++OEHbNtmzZo1AJQvX56BAwdSo0YNUqVKFe1a161bh5eXF/v376dMmTL4+vpGazV1EZHELEphyhizByj+iG+Vj9lyROIXYwzr1q3D6XSyfPnyyFYvnp6evPPOO0+1n927d2PbNrNmzcLf35/cuXPTu3dvmjVrxmuvvRYj9Z46dYrOnTuzaNEiXnvtNRYuXEjNmjXVAkZEJBq0ArrIM7h//z6zZ8/Gx8eH/fv3kyVLFnr37s1XX331VK1erl+/zqxZs7Btm71795IyZUpq166Nw+GgTJky0Wpe/Fd3795l0KBBjBgxgmTJkjFgwAA6dOjwzCuqi4jI/yhMiTyFy5cvM3bs2MhWL4ULF8a2bRo0aBDlYBIWFsbPP/+MbdssWbKE4OBgSpQowbhx46hfvz4ZMmSIsXrDw8OZOXMmXbt25eLFizRp0oRBgwaRI0eOGDuGiEhipzAlEgV79+7F6XQyZ84cgoODqVKlCt7e3k/V6uXEiRNMnTqVqVOncv78eTJnzkzbtm3x8PCgcOHCMV7z1q1bad++Pdu2baNkyZIsWrTomZZiEBGRf6cwJfIY4eHhLF++HKfTyS+//BLZ6qV9+/ZRXsQyICCAhQsXYts269evJ0mSJHz22WeMHDmSKlWqkCJFihiv+8KFC3Tt2pUZM2bwyiuvMG3aNBo3bhxjU4YiIvIwhSmRv7l79y5Tp07F19eXEydO8OqrrzJkyBC++OKLKLV6McawdetWbNtm7ty53Llzh3z58jFo0CCaNm1K9uzZY6Xu+/fv85///IeBAwcSEhJCt27d6NatGy+++GKsHE9ERB5QmBKJcPbsWUaNGhXZ6uW9996jf//+1KpVK0oLWF6+fJkZM2Zg2zaHDx8mTZo01K1bF4fDgbu7e6zdMWeMYfHixXTs2JHTp09Tq1Ythg0bRp48eWLleCIi8jCFKUn0Nm/ejI+PDwsXLgSgdu3aeHt7R+n6opCQEH766Sds22b58uWEhYXh7u7OlClT+Pzzz2N9VGjfvn14eXnxyy+/UKhQIdauXUu5cuVi9ZgiIvIwhSlJlEJCQli0aBFOp5OtW7eSPn16OnTowNdffx2lVi+HDx/Gtm2mT5/OlStXyJYtG506dcLDw4P8+fPHev3Xrl3j22+/ZcKECWTIkIGxY8fyxRdfPFOzZBERiR795pVE5ebNm0yaNIlRo0Zx7tw58ubNy6hRo2jevPkTW73cvn2befPmYds2W7ZsIVmyZFStWhWHw0HFihWfS5AJCQlh3Lhx9O7dmzt37vD111/Tu3dvMmXKFOvHFhGRR1OYkkTh761eypYty9ixY6lcufK/3uVmjGHDhg3Yts38+fMJDAykQIECjBgxgsaNG5M1a9bndg6rVq3C29ubw4cP88knn+Dj40OBAgWe2/FFROTRFKYkwfqz1YuPjw/Lli0jRYoUNGzYEC8vrye2ejl37lxkg+GTJ0+SLl06mjZtisPhoESJEs+1/crx48fp0KEDy5YtI2/evCxdupQqVaqoBYyISByhMCUJzv3795kzZw4+Pj7s27cvstVL69atyZYt22NfFxQUxNKlS7Ftm9WrVxMeHk7ZsmXp06cPtWrVInXq1M/xLODWrVv0798fX19fUqZMydChQ2nfvj0vvPDCc61DRET+ncKUJBiXL19m3LhxjBs3jitXrlC4cGGmTJlCw4YN/7XVy969e7Ftm5kzZ3Ljxg1y5sxJjx49aN68uUuWFwgLC2Pq1Kl0796dq1ev4uHhwYABA/41CIqIiOsoTEm8t3fvXnx8fJg9ezbBwcFUrlwZb29vypUr99ipsBs3bjBnzhxs22bXrl2kSJGCmjVr4nA4KF++PEmTJn3OZ/HAb7/9hqenJ7t27cLNzY3ly5dTvHhxl9QiIiJRozAl8dKfrV58fHxYt24dqVOnpmXLlnh6ej621UtYWBjr1q3Dtm0WL15MUFAQxYoVY/To0TRo0MCld8SdPXuWLl26MHfuXF599VVmz55N/fr1dV2UiEg8oDAl8crdu3eZNm0avr6+HD9+PLLVS8uWLR8bhn7//ffIBsNnz54lU6ZMtGrVCg8PD959993nfAYPu3fvHsOGDWPIkCEYY+jduzedO3cmTZo0Lq1LRESiTmFK4oWzZ88yevRoJk2ahL+/PyVLlmTOnDnUrl37ka1eAgMDWbRoEbZts27dOizL4tNPP2X48OFUq1bN5RdxG2P4/vvv6dy5M3/88Qf16tVjyJAh5M6d26V1iYjI01OYkjhty5YtOJ1OFi5ciDEmstXL+++//49tjTHs2LED27aZM2cOt27dIk+ePPTv35+mTZuSM2dOF5zBP+3atQtPT09+++03ihYtyqxZsyhdurSryxIRkWekMCVxTmhoKAsXLsTHx4ctW7aQPn16vL29+frrrx85cnP16lVmzpyJbdscOHCAVKlS8fnnn+NwOChduvS/Lsr5PF2+fJmePXsyZcoUMmfOzKRJk/Dw8HDZxe4iIhIzFKYkzrh58yaTJ09m1KhR/PHHH//a6iU0NJSVK1fi5+fH0qVLCQ0NpVSpUkycOJF69eqRLl06F53FPwUHBzNy5Ej69evHvXv36NChA7169SJ9+vSuLk1ERGKAwpS43PHjxyNbvQQEBFC2bFlGjx5N5cqV/zFqc/ToUfz8/Jg+fToXL14ka9aseHl54eHhEedaqxhjWL58OR06dOD48eNUrlyZESNGPJdGyCIi8vwoTIlLGGP45ZdfcDqdLF++nOTJk9OwYUM8PT3/cYfdnTt3mD9/PrZts3HjRpImTUrlypVxOBxUqlTpkRegu9rhw4fx9vZm1apV5M+fn59++onPPvvM1WWJiEgsUJiS5+pRrV569erFV1999dAK38YYNm7ciG3bfP/99wQEBPDWW28xdOhQmjRpEmdXA7958yZ9+/Zl9OjRpE2bFqfTSdu2beNk4BMRkZihMCXPxeXLlxk/fjxjx47lypUrFCpU6JGtXi5cuMD06dOxbZvjx4+TNm1aGjRogMPhoFSpUnF2EcuwsDAmTZpEz549uXnzJl988QX9+vUjS5Ysri5NRERimcKUxKp9+/bh4+PDrFmzIlu9eHl5Ub58+chgFBwczLJly7BtmxUrVhAeHs6HH35Ijx49qFOnTpxfwPKXX37By8uLffv28dFHH+Hr68s777zj6rJEROQ5UZiSGBceHs5PP/2E0+mMbPXSokULPD09H7r4+sCBA9i2zYwZM7h27RrZs2ena9euNG/enHz58rnwDKLm999/p3PnzixcuJDcuXOzYMECatWqFWdHz0REJHYoTEmMCQgIYOrUqZGtXnLkyMHgwYP54osvIlu9+Pv7M3fuXGzbZvv27SRPnpzq1avjcDioUKFCvFhz6e7duwwePJjhw4eTNGlS+vfvT4cOHUiVKpWrSxMRERdQmJJo++OPPxg9ejQTJ058ZKuX8PDwyAbDCxcu5P79+xQpUgRfX18aNmxI5syZXX0KURIeHs7s2bPp0qULFy5coHHjxgwePJgcOXK4ujQREXEhhSl5Zlu3bsXpdLJgwYLIVi9eXl68//77WJbFmTNnmDZtGn5+fpw+fZoMGTLQokULHA4HRYsWjVfTYdu2bcPT05MtW7ZQokQJFixY8MiWNiIikvgoTMlTCQ0NZdGiRTidTrZs2UK6dOnw8vKiXbt25M6dm/v37zNv3jxs22bNmjUAfPzxxwwaNIgaNWo8dOdefHDhwgW6devG9OnTyZYtG1OnTqVJkyZxpkWNiIi4nsKURIm/vz+TJk2KbPXyxhtvMHLkyMhWL7t372bYsGHMmjULf39/XnvtNfr06UOzZs0e2U8vrrt//z5Op5MBAwYQEhJC165d6d69Oy+++KKrSxMRkThGYUr+1fHjxxk5ciR+fn4EBARQpkyZyFYv/v7++Pn5Yds2e/fuJWXKlNSuXRuHw0GZMmXi5eiNMYYffviBjh078vvvv1OjRg2GDx/OG2+84erSREQkjop/73YS6/5s9VKtWjXy58/PhAkTqFOnDrt27WLNmjWkSJGCBg0akD17djw9PUmRIgXjxo3j4sWLzJw5k3LlysXLILV//34+/vhjatWqRerUqfn5559ZvHixgpSIiPwrjUxJpKCgoMhWL3v37iVz5sz07NmTNm3acPfuXfz8/KhatSrnz58nc+bMtG3bFg8PDwoXLuzq0qPl+vXrfPvtt4wfP54MGTIwZswYvvzyS5Il04+HiIg8md4thCtXrjBu3LjIVi8FCxZk8uTJ1KhRg2XLllGvXj02bNhAkiRJ+Oyzzxg5ciRVqlQhRYoUri49WkJCQhg/fjy9e/fm9u3btG3blj59+kSuiSUiIhIVClOJ2P79+yNbvQQFBVGpUiW8vLxIkyYNfn5+eHt7c+fOHfLly8egQYNo2rQp2bNnd3XZMeLnn3/Gy8uLQ4cO8fHHH+Pj40PBggVdXZaIiMRDClOJTHh4OCtWrMDpdLJ27VpSpUqFw+GgQYMGbNmyhfbt23PkyBHSpElD3bp1cTgcuLu7x6s1of7NiRMn6NixI0uXLuWNN95gyZIlVK1aNcGcn4iIPH8KU4lEQEAA06ZNw9fXl2PHjpEjRw769+9P7ty5mT9/PmXLliUsLAx3d3emTJnC559/nqCWAbh9+zYDBgzA6XTywgsvMGTIEDw9PXnhhRdcXZqIiMRzClMJ3N9bvZQoUYKhQ4dy6dIlRo4cyZUrV8iWLRudOnXCw8PjoUbECUF4eDhTp06le/fuXL58GQ8PDwYOHEi2bNlcXZqIiCQQClMJ1NatW/Hx8WH+/PkYY6hatSpvvfUW69ev55tvviFZsmRUrVoVh8NBxYoVE+Sdaxs3bsTT05OdO3fy/vvv8+OPP1KiRAlXlyUiIglMlN5BLcs6DdwBwoBQY0xxy7L6AF8AVyM2626M+Sk2ipSoCQ0NZfHixTidTjZv3syLL75I7dq1CQ0NZcWKFSxZsoSCBQsyYsQIGjduTNasWV1dcqz4448/6NKlC3PmzCFHjhzMmjWLBg0a6LooERGJFU8zHFHWGHPtb885jTHDY7IgeXr+/v5MnjyZUaNGcfbsWXLlykXlypU5dOgQ33//PenSpaNZs2Y4HA6KFy+eYEPFvXv3GD58OIMHD8YYQ69evejSpQtp0qRxdWkiIpKAJby5nUTkxIkT+Pr6RrZ6KVCgAMWKFWPPnj2cPXuWcuXK0a9fP2rWrEnq1KldXW6sMcYwf/58OnfuzNmzZ6lbty5Dhw6Nlz0BRUQk/olqzw8DrLYsa6dlWV/+5fmvLcvaZ1mWbVlWxke90LKsLy3L2mFZ1o6rV68+ahN5Cn+2eqlevTpvvvkm48ePJ2fOnKRPn55Dhw5x9epVevbsyalTp1i7di2NGjVK0EFq9+7dfPTRR9SrV49MmTKxfv165s2bpyAlIiLPjWWMefJGlpXdGHPBsqyswM9AO+AocI0HQasf8IoxxvFv+ylevLjZsWNH9KtOhIKCgpg7dy4+Pj7s2bOHNGnS8OKLL3Lp0iVeeOEFatasicPhoFy5ciRNmtTV5ca6K1eu0LNnTyZPnsxLL73EwIEDcTgcieLcRUTk+bAsa6cxpviTtovSNJ8x5kLE5yuWZS0GShpjNvzlYJOAZc9arDzelStXGD9+PGPGjOHKlSukS5eOZMmSERAQQP78+enZsycNGjRINC1QgoODGT16NH379uXevXt4e3vTq1cvMmTI4OrSREQkkXpimLIsKw2QxBhzJ+JxBeA7y7JeMcZcjNisJnAgFutMdP5s9TJz5kyCg4NJmTIlAMmSJaNNmzZ4eHjw7rvvurjK5+unn37C29ubY8eOUalSJf7zn/8kuHWxREQk/onKyNTLwOKIO8CSAbONMSsty5phWda7PJjmOw20irUqE4k/W72MGDGCX375hSRJkhAeHo5lWZQpUwaHw0G1atUS3ardR44cwdvbm5UrV5I/f36WL19OpUqVXF2WiIgIEIUwZYw5BbzziOebxEpFidCfrV6GDBnC2bNnI5cuyJUrFy1btqRp06bkzJnTxVU+f/7+/vTt25fRo0eTJk0a/vOf/9C2bVtSpEjh6tJEREQiaWkEFzp37hxDhgxhypQpBAYGApAiRQrq1q1Ly5YtKV26NEmSRPWGy4QjLCyMyZMn07NnT65fv84XX3xBv379EuwioyIiEr8pTLnApk2b6Nq1K//9738jnytYsCDt27enfv36pEuXzoXVudavv/6Kl5cXe/fu5cMPP8TX1zfRXRsmIiLxi8LUcxIaGsqYMWMYNmwY58+fByB16tQ0atQILy8vChQo4OIKXev06dN07tyZBQsWkDt3bubPn0/t2rUT7GrtIiKScChMxbI//viDTp06sWTJEoKCggAoVKgQPXr0oHbt2iRPntzFFbpWQEAAgwcPZtiwYSRNmpR+/frRsWNHUqVK5erSREREokRhKhYYY5g3bx7fffcdhw8fBh6MQjVt2pSBAweSI0cOF1foesYYZs+eTZcuXTh//jyNGjVi8ODBvPrqq64uTURE5KkoTMWg8+fP07dvX2bPnk1AQAAAefPmpUePHjRr1kxTVhG2b9+Op6cnmzdvpnjx4nz//fe4ubm5uiwREZFnkvhuFYthwcHBzJs3j3fffZdXX32VSZMmERQURLVq1Thx4gTHjx+nefPmClLAxYsX8fDwoGTJkpw6dQo/Pz+2bt2qICUiIvGaRqae0YEDBxg9ejQzZszg3r17AGTOnBkvLy86dOiga37+4v79+/j4+DBgwACCg4Pp0qUL3bt3T9R3LYqISMKhMPUU/P39mTt3LmPGjOHAgf91zylevDjfffcdFStW1AjUXxhjWLJkCR07duTUqVNUr16d4cOHkzdvXleXJiIiEmMUpp4gPDycX3/9lSlTprBgwQKCg4MBSJ48OQ0aNKBr1668/fbbLq4y7jlw4ABeXl6sXbuWAgUKsHr1aj755BNXlyUiIhLjFKYe48yZM0ybNg3btjlz5kxkn7zMmTPj7e1Nq1ateOmll1xdZpxz/fp1evfuzbhx40ifPj2jRo2idevWJEumf2oiIpIw6R3uL+7fv8/ixYvx8/NjzZo1GGMiQ8C7775Lx44dqVOnjnrDPUJoaCjjx4/n22+/5fbt27Rp04Y+ffoocIqISIKX6MOUMYZdu3Zh2zazZ8/G39+fNGnSRF77VK1aNby9vXF3d9f1UI+xZs0avLy8OHjwIOXLl8fHx4dChQq5uiwREZHnItGGqWvXrjFr1ixs22bfvn0kT56c9OnTA2BZFp6enrRr147XX3/dxZXGXSdOnIhc3T1Pnjz88MMPVKtWTaFTREQSlUQVpsLCwli9ejW2bbNkyRJCQkLImTMnmTJl4saNG7z44ov07NkTDw8P3bb/L+7cucOAAQNwOp2kSJGCwYMH4+XlxQsvvODq0kRERJ67BBmmfth9nmGrjnLBP5DsGVLR+O0UnNu2gmnTpnH+/HkyZsxIwYIFOXr0KH/88Qcffvgh3t7eVK1alaRJk7q6/DgrPDyc6dOn061bNy5dukTz5s0ZOHAgr7zyiqtLExERcZkEF6Z+2H2ebov2ExAQwL2jG9m5/2c2/XGAJEmSULJkSbJnz8727du5e/cu9evXx8vLi2LFirm67Dhv06ZNeHp6smPHDkqVKsXSpUspUaKEq8sSERFxuQQXpoatOkpgSBhX5vUk6MIRkmV8hdT53TH+59myZQsvvfQSPXr0oE2bNmTPnt3V5cZ5586do0uXLsyePZvs2bMzc+ZMGjZsqOuiREREIiS4MHXePxCAF0tUJ9mpnASe3M69oxtJ/lIuJk6cSOPGjdXqJQoCAwMZPnw4gwcPJiwsjJ49e9KlSxfSpk3r6tJERETilAQXppJaFmHGcHvbDwRfPErK1/+PdCWqk+b1YnzxRWVXlxfnGWNYsGABnTt35syZM9SpU4dhw4bx2muvubo0ERGROCnBhakwYwDIWK4lSVKmIUXmXACEu7KoeGLPnj14enqyYcMG3nnnHaZNm8ZHH33k6rJERETitCSuLiCm5cjwYAov5atvRwapvz4v/3T16lVatWpFsWLFOHToEBMmTGDnzp0KUiIiIlGQ4MJU50/zkyr5w8sbpEqelM6f5ndRRXFXcHAwTqeTfPnyYds2Xl5eHD9+nC+//FJLRIiIiERRgpvmq1E0B8BD60x1/jR/5PPywIoVK/D29ubo0aNUrFgRp9PJW2+95eqyRERE4p0EF6bgQaBSeHq0o0eP0qFDB3766SfefPNNli9fTqVKlVxdloiISLyV4Kb55NH8/f3p0KEDhQoV4rfffmPEiBHs379fQUpERCSaEuTIlPxPWFgYU6ZMoUePHly/fp2WLVvSv39/smbN6urSREREEgSNTCVg69ev5//+7/9o1aoVb7/9Njt37mTixIkKUiIiIjFIYSoBOnPmDHXr1qVMmTLcvHmTefPmsX79eooWLerq0kRERBIcTfMlIAEBAQwZMoRhw4ZhWRZ9+/alU6dOpE6d2tWliYiIJFgKUwmAMYY5c+bwzTffcP78eRo2bMjgwYPJmTOnq0sTERFJ8DTNF8/t2LGDDz74gEaNGpEtWzZ+++03Zs2apSAlIiLynChMxVOXLl3C4XBQokQJTpw4wZQpU9i2bRvu7u6uLk1ERCRR0TRfPBMUFISvry/9+vUjKCiIzp0707NnT9KlS+fq0kRERBIlhal4whjD0qVL6dixIydPnqRq1aqMGDGCfPnyubo0ERGRRE3TfPHAwYMHqVChAjVq1CBFihSsWrWKpUuXKkiJiIjEAQpTcdiNGzdo164d77zzDjt27GDkyJHs3buXChUquLo0ERERiaBpvjgoNDSUCRMm8O233+Lv70/r1q3p27cvmTNndnVpIiIi8jcKU3HM2rVr8fT05ODBg5QtWxZfX18KFy7s6rJERETkMaI0zWdZ1mnLsvZblrXHsqwdEc9lsizrZ8uyjkd8zhi7pSZsJ0+epGbNmnz88cfcu3ePRYsWsXbtWgUpERGROO5prpkqa4x51xhTPOLrrsBaY0w+YG3E1/KU7ty5Q7du3ShQoAA///wzAwcO5NChQ9SsWRPLslxdnoiIiDxBdKb5qgNlIh5PA34FukSznkQjPDycGTNm0LVrVy5dukTTpk0ZNGgQ2bNnd3VpIiIi8hSiOjJlgNWWZe20LOvLiOdeNsZcBIj4nDU2CkyINm/eTKlSpWjevDm5c+dmy5YtTJs2TUFKREQkHopqmHI3xhQDPgPaWpb1YVQPYFnWl5Zl7bAsa8fVq1efqciE4vz58zRp0gQ3NzfOnTvH9OnT2bRpE++9956rSxMREZFnFKUwZYy5EPH5CrAYKAlctizrFYCIz1ce89qJxpjixpjiWbJkiZmq45nAwED69+/Pm2++yfz58+nRowfHjh2jSZMmJEmipb5ERETisye+k1uWlcayrBf/fAxUAA4AS4FmEZs1A5bEVpHxlTGGBQsW8Pbbb9OrVy8+++wzDh8+TP/+/UmbNq2ryxMREZEYEJUL0F8GFkfcWZYMmG2MWWlZ1nbge8uyWgBngc9jr8z4Z+/evXh6erJ+/XoKFy7MunXrKFu2rKvLEhERkRj2xDBljDkFvPOI568D5WOjqPjs6tWr9OrVi0mTJpExY0bGjRtHy5YtSZZM66OKiIgkRHqHjyEhISGMGTOGPn36cPfuXdq1a0fv3r3JmFFrmYqIiCRkClMxYOXKlXh7e3PkyBEqVKiA0+mkQIECri5LREREngPdShYNx44do0qVKnz22WeEhoby448/snLlSgUpERGRRERh6hncunWLTp06UbBgQTZs2MCwYcM4ePAgVapUUQsYERGRREbTfE8hLCwM27bp0aMH165dw+FwMGDAAF5++WVXlyYiIiIuojAVRRs2bMDT05M9e/bwwQcfsHLlSooVK+bqskRERMTFNM33BGfOnKFevXp89NFHXL9+nblz57JhwwYFKREREQE0MvVYAQEBDB06lKFDh2JZFn369KFz586kTp3a1aWJiIhIHKIw9TfGGObOncs333zDuXPnqF+/PkOGDCFXrlyuLk1ERETiIE3z/cXOnTspXbo0DRs2JEuWLGzYsIE5c+YoSImIiMhjKUwBly5dokWLFpQoUYLjx48zefJktm/fTunSpV1dmoiIiMRxiXqaLygoiJEjR9KvXz/u379Px44d6dmzJ+nTp3d1aSIiIhJPJMowZYxh2bJldOjQgRMnTlClShVGjBjBm2++6erSREREJJ5JdNN8hw4domLFilSrVo1kyZKxYsUKfvzxRwUpEREReSaJJkzduHGD9u3bU6RIEbZt24avry/79u2jYsWKri5NRERE4rEEOc33w+7zDFt1lAv+gbzyYgoK393Ogokj8Pf3p1WrVnz33XdkzpzZ1WWKiIhIApDgwtQPu8/TbdF+AkPCCDyzlx1rJrL52hkKFX+fX6aMp0iRIq4uUURERBKQBBemhq06SmBIGNeWjSDg4C8kTf8yWWp058WS5RSkREREJMYluDB1wT8QgBRZXyf5SzlJV6IGVrIUXLx138WViYiISEJuO1dSAAAD60lEQVSU4MJU9gypOO8fSLqStf7xvIiIiEhMS3B383X+ND+pkid96LlUyZPS+dP8LqpIREREErIENzJVo2gOgMi7+bJnSEXnT/NHPi8iIiISkxJcmIIHgUrhSURERJ6HBDfNJyIiIvI8KUyJiIiIRIPClIiIiEg0KEyJiIiIRIPClIiIiEg0KEyJiIiIRIPClIiIiEg0KEyJiIiIRIPClIiIiEg0KEyJiIiIRINljHl+B7Osq8CZ53ZAyAxce47HExERkefjebzH5zbGZHnSRs81TD1vlmXtMMYUd3UdIiIiErPi0nu8pvlEREREokFhSkRERCQaEnqYmujqAkRERCRWxJn3+AR9zZSIiIhIbEvoI1MiIiIisSpehSnLsl6zLOtANPdRzbKsrjFVk4iIiMQey7JOW5aV2dV1/Jtkri7geTPGLAWWuroOERERSRji1chUhGSWZU2zLGufZVkLLMtKbVnWt5Zlbbcs64BlWRMty7IALMtqb1nWoYht50Y819yyrNERj1+2LGuxZVl7Iz7cXHliIiIiiZllWY0ty9pmWdYey7ImWJaV9C/fe82yrCOWZU2OeL+fZVnWx5ZlbbQs67hlWSUjtstkWdYPEe/9WyzLKhLbdcfHMJUfmGiMKQLcBtoAo40xJYwxhYBUQJWIbbsCRSO2bf2IfY0E1htj3gGKAQdjvXoRERH5B8uy3gbqAe7GmHeBMKDR3zbLC/gCRYC3gIbAB0AnoHvENn2B3RHv/d2B6bFde3wMU38YYzZGPJ7Jg/+IZS3L2mpZ1n6gHFAw4vv7gFmWZTUGQh+xr3LAOABjTJgx5lbsli4iIiKPUR74P2C7ZVl7Ir7O87dtfjfG7DfGhPNgAGStebAswX7gtYhtPgBmABhj1gEvWZaVPjYLj49h6u9rORhgLFDHGFMYmASkjPheZWAMD/7n7LQsK9FdIyYiIhJPWMA0Y8y7ER/5jTF9/rZN0F8eh//l63D+dx249Yh9x+o6UPExTOWyLOv9iMcNgN8iHl+zLCstUAfAsqwkQE5jzC/AN0AGIO3f9rUW+Cpi+6SWZaWL7eJFRETkkdYCdSzLygqR1z7lfob9bCBietCyrDLANWPM7Rir8hHi40jNYaCZZVkTgOM8mKbLyIMhvtPA9ojtkgIzI4b2LMBpjPGPuDb9T57ARMuyWvBgbvYrYPPzOAkRERH5H2PMIcuyegKrIwZEQoC2z7CrPoCfZVn7gHtAs5ir8tG0ArqIiIhINMTHaT4RERGROENhSkRERCQaFKZEREREokFhSkRERCQaFKZEREREokFhSkRERCQaFKZEREREokFhSkRERCQa/h+lj1n4huns0gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Pair plot\n",
    "plt.scatter(np.zeros(len(basic_accuracies)), basic_accuracies)\n",
    "plt.scatter(np.ones(len(elmo_accuracies)), elmo_accuracies)\n",
    "\n",
    "for i in range(len(basic_accuracies)):\n",
    "    plt.plot( [0,1], [basic_accuracies[i], elmo_accuracies[i]], c='k')\n",
    "\n",
    "plt.xticks([0,1], ['basic', 'elmo'])\n",
    "\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}