12. Character-Based Neural Language Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to Develop a Character-Based Neural Language Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A language model predicts the next word in a sequence based on the specific words that have come before it in the sequence. it is also possible to develop language models at the character level using neural networks. The benefit of character-based language models is their small vocabulary and flexibility in handling any words, punctuations and other document structures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Sing a Song of Sixpence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the data we'll use; A nursery rhyme. We will use all 4 verses of the rhyme to develop our character-based language model. It is quite short, so fitting a model will be fast, but not so short that we wont see anything interesting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sing a song of sixpence,\n",
    "A pocket full of rye.\n",
    "Four and twenty blackbirds,\n",
    "Baked in a pie.\n",
    "When the pie was opened\n",
    "The birds began to sing;\n",
    "Wasn ' t that a dainty dish,\n",
    "To set before the king.\n",
    "The king was in his counting house,\n",
    "Counting out his money;\n",
    "The queen was in the parlour,\n",
    "Eating bread and honey.\n",
    "The maid was in the garden,\n",
    "Hanging out the clothes,\n",
    "When down came a blackbird\n",
    "And pecked off her nose."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy the text above and save in a file in your working directory called rhyme.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/michael/Documents/Programming#/AMMI Basics/Natural Language Processing Keras\r\n"
     ]
    }
   ],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A language model must be trained on the text, and in the case of a character based LM, the input and output sequences must be characters. The number of characters used as inputs will also determine the number of characters that will need to be provided to the model in order to elicit the first predicted character.\n",
    "\n",
    "After the first character has been generated, it can be appended to the input sequence and used as input for the model to generate the next character.\n",
    "\n",
    "Longer sequences offer more context for the model to learn what character to output next, but take longer to train and impose more burden on seeding the model when generating text.\n",
    "\n",
    "We will use an arbitrary length of 10 characters for this model. There is not a lot of text, and 10 characters is a few words. We can now transform the raw text into a form that our model can learn; specifically, input and output sequences of characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load document\n",
    "def load_doc(filename):\n",
    "    file = open(filename, 'rt')\n",
    "    text = file.read()\n",
    "    file.close()\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sing a song of sixpence, \n",
      "A pocket full of rye. \n",
      "Four and twenty blackbirds, \n",
      "Baked in a pie. \n",
      "When the pie was opened \n",
      "The birds began to sing; \n",
      "Wasn't that a dainty dish, \n",
      "To set before the king. \n",
      "The king was in his counting house, \n",
      "Counting out his money; \n",
      "The queen was in the parlour, \n",
      "Eating bread and honey. \n",
      "The maid was in the garden, \n",
      "Hanging out the clothes, \n",
      "When down came a blackbird \n",
      "And pecked off her nose.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get file and print\n",
    "print(load_doc('Data/rhyme.txt'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sing a song of sixpence, A pocket full of rye. Four and twenty blackbirds, Baked in a pie. When the pie was opened The birds began to sing; Wasn't that a dainty dish, To set before the king. The king was in his counting house, Counting out his money; The queen was in the parlour, Eating bread and honey. The maid was in the garden, Hanging out the clothes, When down came a blackbird And pecked off her nose.\n"
     ]
    }
   ],
   "source": [
    "# Clean the text\n",
    "tokens = load_doc('Data/rhyme.txt')\n",
    "tokens = tokens.split()\n",
    "raw_text = ' '.join(tokens)\n",
    "print(raw_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Sing a song of sixpence, A pocket full of rye. Four and twenty blackbirds, Baked in a pie. When the pie was opened The birds began to sing; Wasn't that a dainty dish, To set before the king. The king was in his counting house, Counting out his money; The queen was in the parlour, Eating bread and honey. The maid was in the garden, Hanging out the clothes, When down came a blackbird And pecked off her nose.\""
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total sequences: 399\n"
     ]
    }
   ],
   "source": [
    "# Create sequences: This creates the input-output sequences used to train the model.\n",
    "# we will use 10 input characters to 1 output character. Making each sequence 11 characters long.\n",
    "# We can create the sequences by enumerating the characters in the text, starting from the 11th character at index 10.\n",
    "length = 10\n",
    "sequences = list()\n",
    "for i in range(length, len(raw_text)):\n",
    "    # select sequence of tokens\n",
    "    seq = raw_text[i-length:i+1]\n",
    "    # store\n",
    "    sequences.append(seq)\n",
    "print('Total sequences: %d'% len(sequences))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Sing a song',\n",
       " 'ing a song ',\n",
       " 'ng a song o',\n",
       " 'g a song of',\n",
       " ' a song of ',\n",
       " 'a song of s',\n",
       " ' song of si',\n",
       " 'song of six',\n",
       " 'ong of sixp',\n",
       " 'ng of sixpe',\n",
       " 'g of sixpen',\n",
       " ' of sixpenc',\n",
       " 'of sixpence',\n",
       " 'f sixpence,',\n",
       " ' sixpence, ',\n",
       " 'sixpence, A',\n",
       " 'ixpence, A ',\n",
       " 'xpence, A p',\n",
       " 'pence, A po',\n",
       " 'ence, A poc',\n",
       " 'nce, A pock',\n",
       " 'ce, A pocke',\n",
       " 'e, A pocket',\n",
       " ', A pocket ',\n",
       " ' A pocket f',\n",
       " 'A pocket fu',\n",
       " ' pocket ful',\n",
       " 'pocket full',\n",
       " 'ocket full ',\n",
       " 'cket full o',\n",
       " 'ket full of',\n",
       " 'et full of ',\n",
       " 't full of r',\n",
       " ' full of ry',\n",
       " 'full of rye',\n",
       " 'ull of rye.',\n",
       " 'll of rye. ',\n",
       " 'l of rye. F',\n",
       " ' of rye. Fo',\n",
       " 'of rye. Fou',\n",
       " 'f rye. Four',\n",
       " ' rye. Four ',\n",
       " 'rye. Four a',\n",
       " 'ye. Four an',\n",
       " 'e. Four and',\n",
       " '. Four and ',\n",
       " ' Four and t',\n",
       " 'Four and tw',\n",
       " 'our and twe',\n",
       " 'ur and twen',\n",
       " 'r and twent',\n",
       " ' and twenty',\n",
       " 'and twenty ',\n",
       " 'nd twenty b',\n",
       " 'd twenty bl',\n",
       " ' twenty bla',\n",
       " 'twenty blac',\n",
       " 'wenty black',\n",
       " 'enty blackb',\n",
       " 'nty blackbi',\n",
       " 'ty blackbir',\n",
       " 'y blackbird',\n",
       " ' blackbirds',\n",
       " 'blackbirds,',\n",
       " 'lackbirds, ',\n",
       " 'ackbirds, B',\n",
       " 'ckbirds, Ba',\n",
       " 'kbirds, Bak',\n",
       " 'birds, Bake',\n",
       " 'irds, Baked',\n",
       " 'rds, Baked ',\n",
       " 'ds, Baked i',\n",
       " 's, Baked in',\n",
       " ', Baked in ',\n",
       " ' Baked in a',\n",
       " 'Baked in a ',\n",
       " 'aked in a p',\n",
       " 'ked in a pi',\n",
       " 'ed in a pie',\n",
       " 'd in a pie.',\n",
       " ' in a pie. ',\n",
       " 'in a pie. W',\n",
       " 'n a pie. Wh',\n",
       " ' a pie. Whe',\n",
       " 'a pie. When',\n",
       " ' pie. When ',\n",
       " 'pie. When t',\n",
       " 'ie. When th',\n",
       " 'e. When the',\n",
       " '. When the ',\n",
       " ' When the p',\n",
       " 'When the pi',\n",
       " 'hen the pie',\n",
       " 'en the pie ',\n",
       " 'n the pie w',\n",
       " ' the pie wa',\n",
       " 'the pie was',\n",
       " 'he pie was ',\n",
       " 'e pie was o',\n",
       " ' pie was op',\n",
       " 'pie was ope',\n",
       " 'ie was open',\n",
       " 'e was opene',\n",
       " ' was opened',\n",
       " 'was opened ',\n",
       " 'as opened T',\n",
       " 's opened Th',\n",
       " ' opened The',\n",
       " 'opened The ',\n",
       " 'pened The b',\n",
       " 'ened The bi',\n",
       " 'ned The bir',\n",
       " 'ed The bird',\n",
       " 'd The birds',\n",
       " ' The birds ',\n",
       " 'The birds b',\n",
       " 'he birds be',\n",
       " 'e birds beg',\n",
       " ' birds bega',\n",
       " 'birds began',\n",
       " 'irds began ',\n",
       " 'rds began t',\n",
       " 'ds began to',\n",
       " 's began to ',\n",
       " ' began to s',\n",
       " 'began to si',\n",
       " 'egan to sin',\n",
       " 'gan to sing',\n",
       " 'an to sing;',\n",
       " 'n to sing; ',\n",
       " ' to sing; W',\n",
       " 'to sing; Wa',\n",
       " 'o sing; Was',\n",
       " ' sing; Wasn',\n",
       " \"sing; Wasn'\",\n",
       " \"ing; Wasn't\",\n",
       " \"ng; Wasn't \",\n",
       " \"g; Wasn't t\",\n",
       " \"; Wasn't th\",\n",
       " \" Wasn't tha\",\n",
       " \"Wasn't that\",\n",
       " \"asn't that \",\n",
       " \"sn't that a\",\n",
       " \"n't that a \",\n",
       " \"'t that a d\",\n",
       " 't that a da',\n",
       " ' that a dai',\n",
       " 'that a dain',\n",
       " 'hat a daint',\n",
       " 'at a dainty',\n",
       " 't a dainty ',\n",
       " ' a dainty d',\n",
       " 'a dainty di',\n",
       " ' dainty dis',\n",
       " 'dainty dish',\n",
       " 'ainty dish,',\n",
       " 'inty dish, ',\n",
       " 'nty dish, T',\n",
       " 'ty dish, To',\n",
       " 'y dish, To ',\n",
       " ' dish, To s',\n",
       " 'dish, To se',\n",
       " 'ish, To set',\n",
       " 'sh, To set ',\n",
       " 'h, To set b',\n",
       " ', To set be',\n",
       " ' To set bef',\n",
       " 'To set befo',\n",
       " 'o set befor',\n",
       " ' set before',\n",
       " 'set before ',\n",
       " 'et before t',\n",
       " 't before th',\n",
       " ' before the',\n",
       " 'before the ',\n",
       " 'efore the k',\n",
       " 'fore the ki',\n",
       " 'ore the kin',\n",
       " 're the king',\n",
       " 'e the king.',\n",
       " ' the king. ',\n",
       " 'the king. T',\n",
       " 'he king. Th',\n",
       " 'e king. The',\n",
       " ' king. The ',\n",
       " 'king. The k',\n",
       " 'ing. The ki',\n",
       " 'ng. The kin',\n",
       " 'g. The king',\n",
       " '. The king ',\n",
       " ' The king w',\n",
       " 'The king wa',\n",
       " 'he king was',\n",
       " 'e king was ',\n",
       " ' king was i',\n",
       " 'king was in',\n",
       " 'ing was in ',\n",
       " 'ng was in h',\n",
       " 'g was in hi',\n",
       " ' was in his',\n",
       " 'was in his ',\n",
       " 'as in his c',\n",
       " 's in his co',\n",
       " ' in his cou',\n",
       " 'in his coun',\n",
       " 'n his count',\n",
       " ' his counti',\n",
       " 'his countin',\n",
       " 'is counting',\n",
       " 's counting ',\n",
       " ' counting h',\n",
       " 'counting ho',\n",
       " 'ounting hou',\n",
       " 'unting hous',\n",
       " 'nting house',\n",
       " 'ting house,',\n",
       " 'ing house, ',\n",
       " 'ng house, C',\n",
       " 'g house, Co',\n",
       " ' house, Cou',\n",
       " 'house, Coun',\n",
       " 'ouse, Count',\n",
       " 'use, Counti',\n",
       " 'se, Countin',\n",
       " 'e, Counting',\n",
       " ', Counting ',\n",
       " ' Counting o',\n",
       " 'Counting ou',\n",
       " 'ounting out',\n",
       " 'unting out ',\n",
       " 'nting out h',\n",
       " 'ting out hi',\n",
       " 'ing out his',\n",
       " 'ng out his ',\n",
       " 'g out his m',\n",
       " ' out his mo',\n",
       " 'out his mon',\n",
       " 'ut his mone',\n",
       " 't his money',\n",
       " ' his money;',\n",
       " 'his money; ',\n",
       " 'is money; T',\n",
       " 's money; Th',\n",
       " ' money; The',\n",
       " 'money; The ',\n",
       " 'oney; The q',\n",
       " 'ney; The qu',\n",
       " 'ey; The que',\n",
       " 'y; The quee',\n",
       " '; The queen',\n",
       " ' The queen ',\n",
       " 'The queen w',\n",
       " 'he queen wa',\n",
       " 'e queen was',\n",
       " ' queen was ',\n",
       " 'queen was i',\n",
       " 'ueen was in',\n",
       " 'een was in ',\n",
       " 'en was in t',\n",
       " 'n was in th',\n",
       " ' was in the',\n",
       " 'was in the ',\n",
       " 'as in the p',\n",
       " 's in the pa',\n",
       " ' in the par',\n",
       " 'in the parl',\n",
       " 'n the parlo',\n",
       " ' the parlou',\n",
       " 'the parlour',\n",
       " 'he parlour,',\n",
       " 'e parlour, ',\n",
       " ' parlour, E',\n",
       " 'parlour, Ea',\n",
       " 'arlour, Eat',\n",
       " 'rlour, Eati',\n",
       " 'lour, Eatin',\n",
       " 'our, Eating',\n",
       " 'ur, Eating ',\n",
       " 'r, Eating b',\n",
       " ', Eating br',\n",
       " ' Eating bre',\n",
       " 'Eating brea',\n",
       " 'ating bread',\n",
       " 'ting bread ',\n",
       " 'ing bread a',\n",
       " 'ng bread an',\n",
       " 'g bread and',\n",
       " ' bread and ',\n",
       " 'bread and h',\n",
       " 'read and ho',\n",
       " 'ead and hon',\n",
       " 'ad and hone',\n",
       " 'd and honey',\n",
       " ' and honey.',\n",
       " 'and honey. ',\n",
       " 'nd honey. T',\n",
       " 'd honey. Th',\n",
       " ' honey. The',\n",
       " 'honey. The ',\n",
       " 'oney. The m',\n",
       " 'ney. The ma',\n",
       " 'ey. The mai',\n",
       " 'y. The maid',\n",
       " '. The maid ',\n",
       " ' The maid w',\n",
       " 'The maid wa',\n",
       " 'he maid was',\n",
       " 'e maid was ',\n",
       " ' maid was i',\n",
       " 'maid was in',\n",
       " 'aid was in ',\n",
       " 'id was in t',\n",
       " 'd was in th',\n",
       " ' was in the',\n",
       " 'was in the ',\n",
       " 'as in the g',\n",
       " 's in the ga',\n",
       " ' in the gar',\n",
       " 'in the gard',\n",
       " 'n the garde',\n",
       " ' the garden',\n",
       " 'the garden,',\n",
       " 'he garden, ',\n",
       " 'e garden, H',\n",
       " ' garden, Ha',\n",
       " 'garden, Han',\n",
       " 'arden, Hang',\n",
       " 'rden, Hangi',\n",
       " 'den, Hangin',\n",
       " 'en, Hanging',\n",
       " 'n, Hanging ',\n",
       " ', Hanging o',\n",
       " ' Hanging ou',\n",
       " 'Hanging out',\n",
       " 'anging out ',\n",
       " 'nging out t',\n",
       " 'ging out th',\n",
       " 'ing out the',\n",
       " 'ng out the ',\n",
       " 'g out the c',\n",
       " ' out the cl',\n",
       " 'out the clo',\n",
       " 'ut the clot',\n",
       " 't the cloth',\n",
       " ' the clothe',\n",
       " 'the clothes',\n",
       " 'he clothes,',\n",
       " 'e clothes, ',\n",
       " ' clothes, W',\n",
       " 'clothes, Wh',\n",
       " 'lothes, Whe',\n",
       " 'othes, When',\n",
       " 'thes, When ',\n",
       " 'hes, When d',\n",
       " 'es, When do',\n",
       " 's, When dow',\n",
       " ', When down',\n",
       " ' When down ',\n",
       " 'When down c',\n",
       " 'hen down ca',\n",
       " 'en down cam',\n",
       " 'n down came',\n",
       " ' down came ',\n",
       " 'down came a',\n",
       " 'own came a ',\n",
       " 'wn came a b',\n",
       " 'n came a bl',\n",
       " ' came a bla',\n",
       " 'came a blac',\n",
       " 'ame a black',\n",
       " 'me a blackb',\n",
       " 'e a blackbi',\n",
       " ' a blackbir',\n",
       " 'a blackbird',\n",
       " ' blackbird ',\n",
       " 'blackbird A',\n",
       " 'lackbird An',\n",
       " 'ackbird And',\n",
       " 'ckbird And ',\n",
       " 'kbird And p',\n",
       " 'bird And pe',\n",
       " 'ird And pec',\n",
       " 'rd And peck',\n",
       " 'd And pecke',\n",
       " ' And pecked',\n",
       " 'And pecked ',\n",
       " 'nd pecked o',\n",
       " 'd pecked of',\n",
       " ' pecked off',\n",
       " 'pecked off ',\n",
       " 'ecked off h',\n",
       " 'cked off he',\n",
       " 'ked off her',\n",
       " 'ed off her ',\n",
       " 'd off her n',\n",
       " ' off her no',\n",
       " 'off her nos',\n",
       " 'ff her nose',\n",
       " 'f her nose.']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sequence creation function\n",
    "def create_sequence(raw_text, sequence_length=10):\n",
    "    if sequence_length==None:\n",
    "        return 'Please include a valid sequence number...'\n",
    "    sequences = list()\n",
    "    for i in range(sequence_length, len(raw_text)):\n",
    "        # select sequence of tokens\n",
    "        seq = raw_text[i-sequence_length:i+1]\n",
    "        # store\n",
    "        sequences.append(seq)\n",
    "    print('Total sequences: %d'% len(sequences))\n",
    "    return sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total sequences: 399\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Sing a song',\n",
       " 'ing a song ',\n",
       " 'ng a song o',\n",
       " 'g a song of',\n",
       " ' a song of ',\n",
       " 'a song of s',\n",
       " ' song of si',\n",
       " 'song of six',\n",
       " 'ong of sixp',\n",
       " 'ng of sixpe',\n",
       " 'g of sixpen',\n",
       " ' of sixpenc',\n",
       " 'of sixpence',\n",
       " 'f sixpence,',\n",
       " ' sixpence, ',\n",
       " 'sixpence, A',\n",
       " 'ixpence, A ',\n",
       " 'xpence, A p',\n",
       " 'pence, A po',\n",
       " 'ence, A poc',\n",
       " 'nce, A pock',\n",
       " 'ce, A pocke',\n",
       " 'e, A pocket',\n",
       " ', A pocket ',\n",
       " ' A pocket f',\n",
       " 'A pocket fu',\n",
       " ' pocket ful',\n",
       " 'pocket full',\n",
       " 'ocket full ',\n",
       " 'cket full o',\n",
       " 'ket full of',\n",
       " 'et full of ',\n",
       " 't full of r',\n",
       " ' full of ry',\n",
       " 'full of rye',\n",
       " 'ull of rye.',\n",
       " 'll of rye. ',\n",
       " 'l of rye. F',\n",
       " ' of rye. Fo',\n",
       " 'of rye. Fou',\n",
       " 'f rye. Four',\n",
       " ' rye. Four ',\n",
       " 'rye. Four a',\n",
       " 'ye. Four an',\n",
       " 'e. Four and',\n",
       " '. Four and ',\n",
       " ' Four and t',\n",
       " 'Four and tw',\n",
       " 'our and twe',\n",
       " 'ur and twen',\n",
       " 'r and twent',\n",
       " ' and twenty',\n",
       " 'and twenty ',\n",
       " 'nd twenty b',\n",
       " 'd twenty bl',\n",
       " ' twenty bla',\n",
       " 'twenty blac',\n",
       " 'wenty black',\n",
       " 'enty blackb',\n",
       " 'nty blackbi',\n",
       " 'ty blackbir',\n",
       " 'y blackbird',\n",
       " ' blackbirds',\n",
       " 'blackbirds,',\n",
       " 'lackbirds, ',\n",
       " 'ackbirds, B',\n",
       " 'ckbirds, Ba',\n",
       " 'kbirds, Bak',\n",
       " 'birds, Bake',\n",
       " 'irds, Baked',\n",
       " 'rds, Baked ',\n",
       " 'ds, Baked i',\n",
       " 's, Baked in',\n",
       " ', Baked in ',\n",
       " ' Baked in a',\n",
       " 'Baked in a ',\n",
       " 'aked in a p',\n",
       " 'ked in a pi',\n",
       " 'ed in a pie',\n",
       " 'd in a pie.',\n",
       " ' in a pie. ',\n",
       " 'in a pie. W',\n",
       " 'n a pie. Wh',\n",
       " ' a pie. Whe',\n",
       " 'a pie. When',\n",
       " ' pie. When ',\n",
       " 'pie. When t',\n",
       " 'ie. When th',\n",
       " 'e. When the',\n",
       " '. When the ',\n",
       " ' When the p',\n",
       " 'When the pi',\n",
       " 'hen the pie',\n",
       " 'en the pie ',\n",
       " 'n the pie w',\n",
       " ' the pie wa',\n",
       " 'the pie was',\n",
       " 'he pie was ',\n",
       " 'e pie was o',\n",
       " ' pie was op',\n",
       " 'pie was ope',\n",
       " 'ie was open',\n",
       " 'e was opene',\n",
       " ' was opened',\n",
       " 'was opened ',\n",
       " 'as opened T',\n",
       " 's opened Th',\n",
       " ' opened The',\n",
       " 'opened The ',\n",
       " 'pened The b',\n",
       " 'ened The bi',\n",
       " 'ned The bir',\n",
       " 'ed The bird',\n",
       " 'd The birds',\n",
       " ' The birds ',\n",
       " 'The birds b',\n",
       " 'he birds be',\n",
       " 'e birds beg',\n",
       " ' birds bega',\n",
       " 'birds began',\n",
       " 'irds began ',\n",
       " 'rds began t',\n",
       " 'ds began to',\n",
       " 's began to ',\n",
       " ' began to s',\n",
       " 'began to si',\n",
       " 'egan to sin',\n",
       " 'gan to sing',\n",
       " 'an to sing;',\n",
       " 'n to sing; ',\n",
       " ' to sing; W',\n",
       " 'to sing; Wa',\n",
       " 'o sing; Was',\n",
       " ' sing; Wasn',\n",
       " \"sing; Wasn'\",\n",
       " \"ing; Wasn't\",\n",
       " \"ng; Wasn't \",\n",
       " \"g; Wasn't t\",\n",
       " \"; Wasn't th\",\n",
       " \" Wasn't tha\",\n",
       " \"Wasn't that\",\n",
       " \"asn't that \",\n",
       " \"sn't that a\",\n",
       " \"n't that a \",\n",
       " \"'t that a d\",\n",
       " 't that a da',\n",
       " ' that a dai',\n",
       " 'that a dain',\n",
       " 'hat a daint',\n",
       " 'at a dainty',\n",
       " 't a dainty ',\n",
       " ' a dainty d',\n",
       " 'a dainty di',\n",
       " ' dainty dis',\n",
       " 'dainty dish',\n",
       " 'ainty dish,',\n",
       " 'inty dish, ',\n",
       " 'nty dish, T',\n",
       " 'ty dish, To',\n",
       " 'y dish, To ',\n",
       " ' dish, To s',\n",
       " 'dish, To se',\n",
       " 'ish, To set',\n",
       " 'sh, To set ',\n",
       " 'h, To set b',\n",
       " ', To set be',\n",
       " ' To set bef',\n",
       " 'To set befo',\n",
       " 'o set befor',\n",
       " ' set before',\n",
       " 'set before ',\n",
       " 'et before t',\n",
       " 't before th',\n",
       " ' before the',\n",
       " 'before the ',\n",
       " 'efore the k',\n",
       " 'fore the ki',\n",
       " 'ore the kin',\n",
       " 're the king',\n",
       " 'e the king.',\n",
       " ' the king. ',\n",
       " 'the king. T',\n",
       " 'he king. Th',\n",
       " 'e king. The',\n",
       " ' king. The ',\n",
       " 'king. The k',\n",
       " 'ing. The ki',\n",
       " 'ng. The kin',\n",
       " 'g. The king',\n",
       " '. The king ',\n",
       " ' The king w',\n",
       " 'The king wa',\n",
       " 'he king was',\n",
       " 'e king was ',\n",
       " ' king was i',\n",
       " 'king was in',\n",
       " 'ing was in ',\n",
       " 'ng was in h',\n",
       " 'g was in hi',\n",
       " ' was in his',\n",
       " 'was in his ',\n",
       " 'as in his c',\n",
       " 's in his co',\n",
       " ' in his cou',\n",
       " 'in his coun',\n",
       " 'n his count',\n",
       " ' his counti',\n",
       " 'his countin',\n",
       " 'is counting',\n",
       " 's counting ',\n",
       " ' counting h',\n",
       " 'counting ho',\n",
       " 'ounting hou',\n",
       " 'unting hous',\n",
       " 'nting house',\n",
       " 'ting house,',\n",
       " 'ing house, ',\n",
       " 'ng house, C',\n",
       " 'g house, Co',\n",
       " ' house, Cou',\n",
       " 'house, Coun',\n",
       " 'ouse, Count',\n",
       " 'use, Counti',\n",
       " 'se, Countin',\n",
       " 'e, Counting',\n",
       " ', Counting ',\n",
       " ' Counting o',\n",
       " 'Counting ou',\n",
       " 'ounting out',\n",
       " 'unting out ',\n",
       " 'nting out h',\n",
       " 'ting out hi',\n",
       " 'ing out his',\n",
       " 'ng out his ',\n",
       " 'g out his m',\n",
       " ' out his mo',\n",
       " 'out his mon',\n",
       " 'ut his mone',\n",
       " 't his money',\n",
       " ' his money;',\n",
       " 'his money; ',\n",
       " 'is money; T',\n",
       " 's money; Th',\n",
       " ' money; The',\n",
       " 'money; The ',\n",
       " 'oney; The q',\n",
       " 'ney; The qu',\n",
       " 'ey; The que',\n",
       " 'y; The quee',\n",
       " '; The queen',\n",
       " ' The queen ',\n",
       " 'The queen w',\n",
       " 'he queen wa',\n",
       " 'e queen was',\n",
       " ' queen was ',\n",
       " 'queen was i',\n",
       " 'ueen was in',\n",
       " 'een was in ',\n",
       " 'en was in t',\n",
       " 'n was in th',\n",
       " ' was in the',\n",
       " 'was in the ',\n",
       " 'as in the p',\n",
       " 's in the pa',\n",
       " ' in the par',\n",
       " 'in the parl',\n",
       " 'n the parlo',\n",
       " ' the parlou',\n",
       " 'the parlour',\n",
       " 'he parlour,',\n",
       " 'e parlour, ',\n",
       " ' parlour, E',\n",
       " 'parlour, Ea',\n",
       " 'arlour, Eat',\n",
       " 'rlour, Eati',\n",
       " 'lour, Eatin',\n",
       " 'our, Eating',\n",
       " 'ur, Eating ',\n",
       " 'r, Eating b',\n",
       " ', Eating br',\n",
       " ' Eating bre',\n",
       " 'Eating brea',\n",
       " 'ating bread',\n",
       " 'ting bread ',\n",
       " 'ing bread a',\n",
       " 'ng bread an',\n",
       " 'g bread and',\n",
       " ' bread and ',\n",
       " 'bread and h',\n",
       " 'read and ho',\n",
       " 'ead and hon',\n",
       " 'ad and hone',\n",
       " 'd and honey',\n",
       " ' and honey.',\n",
       " 'and honey. ',\n",
       " 'nd honey. T',\n",
       " 'd honey. Th',\n",
       " ' honey. The',\n",
       " 'honey. The ',\n",
       " 'oney. The m',\n",
       " 'ney. The ma',\n",
       " 'ey. The mai',\n",
       " 'y. The maid',\n",
       " '. The maid ',\n",
       " ' The maid w',\n",
       " 'The maid wa',\n",
       " 'he maid was',\n",
       " 'e maid was ',\n",
       " ' maid was i',\n",
       " 'maid was in',\n",
       " 'aid was in ',\n",
       " 'id was in t',\n",
       " 'd was in th',\n",
       " ' was in the',\n",
       " 'was in the ',\n",
       " 'as in the g',\n",
       " 's in the ga',\n",
       " ' in the gar',\n",
       " 'in the gard',\n",
       " 'n the garde',\n",
       " ' the garden',\n",
       " 'the garden,',\n",
       " 'he garden, ',\n",
       " 'e garden, H',\n",
       " ' garden, Ha',\n",
       " 'garden, Han',\n",
       " 'arden, Hang',\n",
       " 'rden, Hangi',\n",
       " 'den, Hangin',\n",
       " 'en, Hanging',\n",
       " 'n, Hanging ',\n",
       " ', Hanging o',\n",
       " ' Hanging ou',\n",
       " 'Hanging out',\n",
       " 'anging out ',\n",
       " 'nging out t',\n",
       " 'ging out th',\n",
       " 'ing out the',\n",
       " 'ng out the ',\n",
       " 'g out the c',\n",
       " ' out the cl',\n",
       " 'out the clo',\n",
       " 'ut the clot',\n",
       " 't the cloth',\n",
       " ' the clothe',\n",
       " 'the clothes',\n",
       " 'he clothes,',\n",
       " 'e clothes, ',\n",
       " ' clothes, W',\n",
       " 'clothes, Wh',\n",
       " 'lothes, Whe',\n",
       " 'othes, When',\n",
       " 'thes, When ',\n",
       " 'hes, When d',\n",
       " 'es, When do',\n",
       " 's, When dow',\n",
       " ', When down',\n",
       " ' When down ',\n",
       " 'When down c',\n",
       " 'hen down ca',\n",
       " 'en down cam',\n",
       " 'n down came',\n",
       " ' down came ',\n",
       " 'down came a',\n",
       " 'own came a ',\n",
       " 'wn came a b',\n",
       " 'n came a bl',\n",
       " ' came a bla',\n",
       " 'came a blac',\n",
       " 'ame a black',\n",
       " 'me a blackb',\n",
       " 'e a blackbi',\n",
       " ' a blackbir',\n",
       " 'a blackbird',\n",
       " ' blackbird ',\n",
       " 'blackbird A',\n",
       " 'lackbird An',\n",
       " 'ackbird And',\n",
       " 'ckbird And ',\n",
       " 'kbird And p',\n",
       " 'bird And pe',\n",
       " 'ird And pec',\n",
       " 'rd And peck',\n",
       " 'd And pecke',\n",
       " ' And pecked',\n",
       " 'And pecked ',\n",
       " 'nd pecked o',\n",
       " 'd pecked of',\n",
       " ' pecked off',\n",
       " 'pecked off ',\n",
       " 'ecked off h',\n",
       " 'cked off he',\n",
       " 'ked off her',\n",
       " 'ed off her ',\n",
       " 'd off her n',\n",
       " ' off her no',\n",
       " 'off her nos',\n",
       " 'ff her nose',\n",
       " 'f her nose.']"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequences = create_sequence(raw_text, 10)\n",
    "sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the sequences to file.\n",
    "def save_doc(lines, filename):\n",
    "    data = '\\n'.join(lines)\n",
    "    file = open(filename, 'w')\n",
    "    file.write(data)\n",
    "    file.close()\n",
    "    print('Done saving to %s...'%filename)  # %s means a string, %d means a decimal number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done saving to Data/char_sequences_399.txt...\n"
     ]
    }
   ],
   "source": [
    "# save\n",
    "file_to_save = 'Data/char_sequences_'+str(len(sequences))+'.txt'\n",
    "save_doc(sequences, file_to_save)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Train Language Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load our sequences data.\n",
    "in_file = 'Data/char_sequences_399.txt'\n",
    "raw_text = load_doc(in_file)\n",
    "lines = raw_text.split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Sing a song',\n",
       " 'ing a song ',\n",
       " 'ng a song o',\n",
       " 'g a song of',\n",
       " ' a song of ',\n",
       " 'a song of s',\n",
       " ' song of si',\n",
       " 'song of six',\n",
       " 'ong of sixp',\n",
       " 'ng of sixpe',\n",
       " 'g of sixpen',\n",
       " ' of sixpenc',\n",
       " 'of sixpence',\n",
       " 'f sixpence,',\n",
       " ' sixpence, ',\n",
       " 'sixpence, A',\n",
       " 'ixpence, A ',\n",
       " 'xpence, A p',\n",
       " 'pence, A po',\n",
       " 'ence, A poc',\n",
       " 'nce, A pock',\n",
       " 'ce, A pocke',\n",
       " 'e, A pocket',\n",
       " ', A pocket ',\n",
       " ' A pocket f',\n",
       " 'A pocket fu',\n",
       " ' pocket ful',\n",
       " 'pocket full',\n",
       " 'ocket full ',\n",
       " 'cket full o',\n",
       " 'ket full of',\n",
       " 'et full of ',\n",
       " 't full of r',\n",
       " ' full of ry',\n",
       " 'full of rye',\n",
       " 'ull of rye.',\n",
       " 'll of rye. ',\n",
       " 'l of rye. F',\n",
       " ' of rye. Fo',\n",
       " 'of rye. Fou',\n",
       " 'f rye. Four',\n",
       " ' rye. Four ',\n",
       " 'rye. Four a',\n",
       " 'ye. Four an',\n",
       " 'e. Four and',\n",
       " '. Four and ',\n",
       " ' Four and t',\n",
       " 'Four and tw',\n",
       " 'our and twe',\n",
       " 'ur and twen',\n",
       " 'r and twent',\n",
       " ' and twenty',\n",
       " 'and twenty ',\n",
       " 'nd twenty b',\n",
       " 'd twenty bl',\n",
       " ' twenty bla',\n",
       " 'twenty blac',\n",
       " 'wenty black',\n",
       " 'enty blackb',\n",
       " 'nty blackbi',\n",
       " 'ty blackbir',\n",
       " 'y blackbird',\n",
       " ' blackbirds',\n",
       " 'blackbirds,',\n",
       " 'lackbirds, ',\n",
       " 'ackbirds, B',\n",
       " 'ckbirds, Ba',\n",
       " 'kbirds, Bak',\n",
       " 'birds, Bake',\n",
       " 'irds, Baked',\n",
       " 'rds, Baked ',\n",
       " 'ds, Baked i',\n",
       " 's, Baked in',\n",
       " ', Baked in ',\n",
       " ' Baked in a',\n",
       " 'Baked in a ',\n",
       " 'aked in a p',\n",
       " 'ked in a pi',\n",
       " 'ed in a pie',\n",
       " 'd in a pie.',\n",
       " ' in a pie. ',\n",
       " 'in a pie. W',\n",
       " 'n a pie. Wh',\n",
       " ' a pie. Whe',\n",
       " 'a pie. When',\n",
       " ' pie. When ',\n",
       " 'pie. When t',\n",
       " 'ie. When th',\n",
       " 'e. When the',\n",
       " '. When the ',\n",
       " ' When the p',\n",
       " 'When the pi',\n",
       " 'hen the pie',\n",
       " 'en the pie ',\n",
       " 'n the pie w',\n",
       " ' the pie wa',\n",
       " 'the pie was',\n",
       " 'he pie was ',\n",
       " 'e pie was o',\n",
       " ' pie was op',\n",
       " 'pie was ope',\n",
       " 'ie was open',\n",
       " 'e was opene',\n",
       " ' was opened',\n",
       " 'was opened ',\n",
       " 'as opened T',\n",
       " 's opened Th',\n",
       " ' opened The',\n",
       " 'opened The ',\n",
       " 'pened The b',\n",
       " 'ened The bi',\n",
       " 'ned The bir',\n",
       " 'ed The bird',\n",
       " 'd The birds',\n",
       " ' The birds ',\n",
       " 'The birds b',\n",
       " 'he birds be',\n",
       " 'e birds beg',\n",
       " ' birds bega',\n",
       " 'birds began',\n",
       " 'irds began ',\n",
       " 'rds began t',\n",
       " 'ds began to',\n",
       " 's began to ',\n",
       " ' began to s',\n",
       " 'began to si',\n",
       " 'egan to sin',\n",
       " 'gan to sing',\n",
       " 'an to sing;',\n",
       " 'n to sing; ',\n",
       " ' to sing; W',\n",
       " 'to sing; Wa',\n",
       " 'o sing; Was',\n",
       " ' sing; Wasn',\n",
       " \"sing; Wasn'\",\n",
       " \"ing; Wasn't\",\n",
       " \"ng; Wasn't \",\n",
       " \"g; Wasn't t\",\n",
       " \"; Wasn't th\",\n",
       " \" Wasn't tha\",\n",
       " \"Wasn't that\",\n",
       " \"asn't that \",\n",
       " \"sn't that a\",\n",
       " \"n't that a \",\n",
       " \"'t that a d\",\n",
       " 't that a da',\n",
       " ' that a dai',\n",
       " 'that a dain',\n",
       " 'hat a daint',\n",
       " 'at a dainty',\n",
       " 't a dainty ',\n",
       " ' a dainty d',\n",
       " 'a dainty di',\n",
       " ' dainty dis',\n",
       " 'dainty dish',\n",
       " 'ainty dish,',\n",
       " 'inty dish, ',\n",
       " 'nty dish, T',\n",
       " 'ty dish, To',\n",
       " 'y dish, To ',\n",
       " ' dish, To s',\n",
       " 'dish, To se',\n",
       " 'ish, To set',\n",
       " 'sh, To set ',\n",
       " 'h, To set b',\n",
       " ', To set be',\n",
       " ' To set bef',\n",
       " 'To set befo',\n",
       " 'o set befor',\n",
       " ' set before',\n",
       " 'set before ',\n",
       " 'et before t',\n",
       " 't before th',\n",
       " ' before the',\n",
       " 'before the ',\n",
       " 'efore the k',\n",
       " 'fore the ki',\n",
       " 'ore the kin',\n",
       " 're the king',\n",
       " 'e the king.',\n",
       " ' the king. ',\n",
       " 'the king. T',\n",
       " 'he king. Th',\n",
       " 'e king. The',\n",
       " ' king. The ',\n",
       " 'king. The k',\n",
       " 'ing. The ki',\n",
       " 'ng. The kin',\n",
       " 'g. The king',\n",
       " '. The king ',\n",
       " ' The king w',\n",
       " 'The king wa',\n",
       " 'he king was',\n",
       " 'e king was ',\n",
       " ' king was i',\n",
       " 'king was in',\n",
       " 'ing was in ',\n",
       " 'ng was in h',\n",
       " 'g was in hi',\n",
       " ' was in his',\n",
       " 'was in his ',\n",
       " 'as in his c',\n",
       " 's in his co',\n",
       " ' in his cou',\n",
       " 'in his coun',\n",
       " 'n his count',\n",
       " ' his counti',\n",
       " 'his countin',\n",
       " 'is counting',\n",
       " 's counting ',\n",
       " ' counting h',\n",
       " 'counting ho',\n",
       " 'ounting hou',\n",
       " 'unting hous',\n",
       " 'nting house',\n",
       " 'ting house,',\n",
       " 'ing house, ',\n",
       " 'ng house, C',\n",
       " 'g house, Co',\n",
       " ' house, Cou',\n",
       " 'house, Coun',\n",
       " 'ouse, Count',\n",
       " 'use, Counti',\n",
       " 'se, Countin',\n",
       " 'e, Counting',\n",
       " ', Counting ',\n",
       " ' Counting o',\n",
       " 'Counting ou',\n",
       " 'ounting out',\n",
       " 'unting out ',\n",
       " 'nting out h',\n",
       " 'ting out hi',\n",
       " 'ing out his',\n",
       " 'ng out his ',\n",
       " 'g out his m',\n",
       " ' out his mo',\n",
       " 'out his mon',\n",
       " 'ut his mone',\n",
       " 't his money',\n",
       " ' his money;',\n",
       " 'his money; ',\n",
       " 'is money; T',\n",
       " 's money; Th',\n",
       " ' money; The',\n",
       " 'money; The ',\n",
       " 'oney; The q',\n",
       " 'ney; The qu',\n",
       " 'ey; The que',\n",
       " 'y; The quee',\n",
       " '; The queen',\n",
       " ' The queen ',\n",
       " 'The queen w',\n",
       " 'he queen wa',\n",
       " 'e queen was',\n",
       " ' queen was ',\n",
       " 'queen was i',\n",
       " 'ueen was in',\n",
       " 'een was in ',\n",
       " 'en was in t',\n",
       " 'n was in th',\n",
       " ' was in the',\n",
       " 'was in the ',\n",
       " 'as in the p',\n",
       " 's in the pa',\n",
       " ' in the par',\n",
       " 'in the parl',\n",
       " 'n the parlo',\n",
       " ' the parlou',\n",
       " 'the parlour',\n",
       " 'he parlour,',\n",
       " 'e parlour, ',\n",
       " ' parlour, E',\n",
       " 'parlour, Ea',\n",
       " 'arlour, Eat',\n",
       " 'rlour, Eati',\n",
       " 'lour, Eatin',\n",
       " 'our, Eating',\n",
       " 'ur, Eating ',\n",
       " 'r, Eating b',\n",
       " ', Eating br',\n",
       " ' Eating bre',\n",
       " 'Eating brea',\n",
       " 'ating bread',\n",
       " 'ting bread ',\n",
       " 'ing bread a',\n",
       " 'ng bread an',\n",
       " 'g bread and',\n",
       " ' bread and ',\n",
       " 'bread and h',\n",
       " 'read and ho',\n",
       " 'ead and hon',\n",
       " 'ad and hone',\n",
       " 'd and honey',\n",
       " ' and honey.',\n",
       " 'and honey. ',\n",
       " 'nd honey. T',\n",
       " 'd honey. Th',\n",
       " ' honey. The',\n",
       " 'honey. The ',\n",
       " 'oney. The m',\n",
       " 'ney. The ma',\n",
       " 'ey. The mai',\n",
       " 'y. The maid',\n",
       " '. The maid ',\n",
       " ' The maid w',\n",
       " 'The maid wa',\n",
       " 'he maid was',\n",
       " 'e maid was ',\n",
       " ' maid was i',\n",
       " 'maid was in',\n",
       " 'aid was in ',\n",
       " 'id was in t',\n",
       " 'd was in th',\n",
       " ' was in the',\n",
       " 'was in the ',\n",
       " 'as in the g',\n",
       " 's in the ga',\n",
       " ' in the gar',\n",
       " 'in the gard',\n",
       " 'n the garde',\n",
       " ' the garden',\n",
       " 'the garden,',\n",
       " 'he garden, ',\n",
       " 'e garden, H',\n",
       " ' garden, Ha',\n",
       " 'garden, Han',\n",
       " 'arden, Hang',\n",
       " 'rden, Hangi',\n",
       " 'den, Hangin',\n",
       " 'en, Hanging',\n",
       " 'n, Hanging ',\n",
       " ', Hanging o',\n",
       " ' Hanging ou',\n",
       " 'Hanging out',\n",
       " 'anging out ',\n",
       " 'nging out t',\n",
       " 'ging out th',\n",
       " 'ing out the',\n",
       " 'ng out the ',\n",
       " 'g out the c',\n",
       " ' out the cl',\n",
       " 'out the clo',\n",
       " 'ut the clot',\n",
       " 't the cloth',\n",
       " ' the clothe',\n",
       " 'the clothes',\n",
       " 'he clothes,',\n",
       " 'e clothes, ',\n",
       " ' clothes, W',\n",
       " 'clothes, Wh',\n",
       " 'lothes, Whe',\n",
       " 'othes, When',\n",
       " 'thes, When ',\n",
       " 'hes, When d',\n",
       " 'es, When do',\n",
       " 's, When dow',\n",
       " ', When down',\n",
       " ' When down ',\n",
       " 'When down c',\n",
       " 'hen down ca',\n",
       " 'en down cam',\n",
       " 'n down came',\n",
       " ' down came ',\n",
       " 'down came a',\n",
       " 'own came a ',\n",
       " 'wn came a b',\n",
       " 'n came a bl',\n",
       " ' came a bla',\n",
       " 'came a blac',\n",
       " 'ame a black',\n",
       " 'me a blackb',\n",
       " 'e a blackbi',\n",
       " ' a blackbir',\n",
       " 'a blackbird',\n",
       " ' blackbird ',\n",
       " 'blackbird A',\n",
       " 'lackbird An',\n",
       " 'ackbird And',\n",
       " 'ckbird And ',\n",
       " 'kbird And p',\n",
       " 'bird And pe',\n",
       " 'ird And pec',\n",
       " 'rd And peck',\n",
       " 'd And pecke',\n",
       " ' And pecked',\n",
       " 'And pecked ',\n",
       " 'nd pecked o',\n",
       " 'd pecked of',\n",
       " ' pecked off',\n",
       " 'pecked off ',\n",
       " 'ecked off h',\n",
       " 'cked off he',\n",
       " 'ked off her',\n",
       " 'ed off her ',\n",
       " 'd off her n',\n",
       " ' off her no',\n",
       " 'off her nos',\n",
       " 'ff her nose',\n",
       " 'f her nose.']"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Encode Sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sequences of characters must be encoded as integers. This means that each unique character will be assigned a specific integer value and each sequence of characters will be encoded as a sequence of integers.\n",
    "\n",
    "We can create the mapping given a sorted set of unique characters in the raw input data. The mapping is a dictionary of character values to integer values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\\n', ' ', \"'\", ',', '.', ';', 'A', 'B', 'C', 'E', 'F', 'H', 'S', 'T', 'W', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'w', 'x', 'y']\n"
     ]
    }
   ],
   "source": [
    "# sort the raw_text\n",
    "chars = sorted(list(set(raw_text)))\n",
    "print(chars)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'\\n': 0, ' ': 1, \"'\": 2, ',': 3, '.': 4, ';': 5, 'A': 6, 'B': 7, 'C': 8, 'E': 9, 'F': 10, 'H': 11, 'S': 12, 'T': 13, 'W': 14, 'a': 15, 'b': 16, 'c': 17, 'd': 18, 'e': 19, 'f': 20, 'g': 21, 'h': 22, 'i': 23, 'k': 24, 'l': 25, 'm': 26, 'n': 27, 'o': 28, 'p': 29, 'q': 30, 'r': 31, 's': 32, 't': 33, 'u': 34, 'w': 35, 'x': 36, 'y': 37}\n"
     ]
    }
   ],
   "source": [
    "# create mapping for the text\n",
    "mapping = dict((c, i) for i, c in enumerate(chars))\n",
    "print(mapping)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Process each sequence of characters one at a time and use the dictionary mapping to look up the integer value for each character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[12, 23, 27, 21, 1, 15, 1, 32, 28, 27, 21],\n",
       " [23, 27, 21, 1, 15, 1, 32, 28, 27, 21, 1],\n",
       " [27, 21, 1, 15, 1, 32, 28, 27, 21, 1, 28],\n",
       " [21, 1, 15, 1, 32, 28, 27, 21, 1, 28, 20],\n",
       " [1, 15, 1, 32, 28, 27, 21, 1, 28, 20, 1],\n",
       " [15, 1, 32, 28, 27, 21, 1, 28, 20, 1, 32],\n",
       " [1, 32, 28, 27, 21, 1, 28, 20, 1, 32, 23],\n",
       " [32, 28, 27, 21, 1, 28, 20, 1, 32, 23, 36],\n",
       " [28, 27, 21, 1, 28, 20, 1, 32, 23, 36, 29],\n",
       " [27, 21, 1, 28, 20, 1, 32, 23, 36, 29, 19],\n",
       " [21, 1, 28, 20, 1, 32, 23, 36, 29, 19, 27],\n",
       " [1, 28, 20, 1, 32, 23, 36, 29, 19, 27, 17],\n",
       " [28, 20, 1, 32, 23, 36, 29, 19, 27, 17, 19],\n",
       " [20, 1, 32, 23, 36, 29, 19, 27, 17, 19, 3],\n",
       " [1, 32, 23, 36, 29, 19, 27, 17, 19, 3, 1],\n",
       " [32, 23, 36, 29, 19, 27, 17, 19, 3, 1, 6],\n",
       " [23, 36, 29, 19, 27, 17, 19, 3, 1, 6, 1],\n",
       " [36, 29, 19, 27, 17, 19, 3, 1, 6, 1, 29],\n",
       " [29, 19, 27, 17, 19, 3, 1, 6, 1, 29, 28],\n",
       " [19, 27, 17, 19, 3, 1, 6, 1, 29, 28, 17],\n",
       " [27, 17, 19, 3, 1, 6, 1, 29, 28, 17, 24],\n",
       " [17, 19, 3, 1, 6, 1, 29, 28, 17, 24, 19],\n",
       " [19, 3, 1, 6, 1, 29, 28, 17, 24, 19, 33],\n",
       " [3, 1, 6, 1, 29, 28, 17, 24, 19, 33, 1],\n",
       " [1, 6, 1, 29, 28, 17, 24, 19, 33, 1, 20],\n",
       " [6, 1, 29, 28, 17, 24, 19, 33, 1, 20, 34],\n",
       " [1, 29, 28, 17, 24, 19, 33, 1, 20, 34, 25],\n",
       " [29, 28, 17, 24, 19, 33, 1, 20, 34, 25, 25],\n",
       " [28, 17, 24, 19, 33, 1, 20, 34, 25, 25, 1],\n",
       " [17, 24, 19, 33, 1, 20, 34, 25, 25, 1, 28],\n",
       " [24, 19, 33, 1, 20, 34, 25, 25, 1, 28, 20],\n",
       " [19, 33, 1, 20, 34, 25, 25, 1, 28, 20, 1],\n",
       " [33, 1, 20, 34, 25, 25, 1, 28, 20, 1, 31],\n",
       " [1, 20, 34, 25, 25, 1, 28, 20, 1, 31, 37],\n",
       " [20, 34, 25, 25, 1, 28, 20, 1, 31, 37, 19],\n",
       " [34, 25, 25, 1, 28, 20, 1, 31, 37, 19, 4],\n",
       " [25, 25, 1, 28, 20, 1, 31, 37, 19, 4, 1],\n",
       " [25, 1, 28, 20, 1, 31, 37, 19, 4, 1, 10],\n",
       " [1, 28, 20, 1, 31, 37, 19, 4, 1, 10, 28],\n",
       " [28, 20, 1, 31, 37, 19, 4, 1, 10, 28, 34],\n",
       " [20, 1, 31, 37, 19, 4, 1, 10, 28, 34, 31],\n",
       " [1, 31, 37, 19, 4, 1, 10, 28, 34, 31, 1],\n",
       " [31, 37, 19, 4, 1, 10, 28, 34, 31, 1, 15],\n",
       " [37, 19, 4, 1, 10, 28, 34, 31, 1, 15, 27],\n",
       " [19, 4, 1, 10, 28, 34, 31, 1, 15, 27, 18],\n",
       " [4, 1, 10, 28, 34, 31, 1, 15, 27, 18, 1],\n",
       " [1, 10, 28, 34, 31, 1, 15, 27, 18, 1, 33],\n",
       " [10, 28, 34, 31, 1, 15, 27, 18, 1, 33, 35],\n",
       " [28, 34, 31, 1, 15, 27, 18, 1, 33, 35, 19],\n",
       " [34, 31, 1, 15, 27, 18, 1, 33, 35, 19, 27],\n",
       " [31, 1, 15, 27, 18, 1, 33, 35, 19, 27, 33],\n",
       " [1, 15, 27, 18, 1, 33, 35, 19, 27, 33, 37],\n",
       " [15, 27, 18, 1, 33, 35, 19, 27, 33, 37, 1],\n",
       " [27, 18, 1, 33, 35, 19, 27, 33, 37, 1, 16],\n",
       " [18, 1, 33, 35, 19, 27, 33, 37, 1, 16, 25],\n",
       " [1, 33, 35, 19, 27, 33, 37, 1, 16, 25, 15],\n",
       " [33, 35, 19, 27, 33, 37, 1, 16, 25, 15, 17],\n",
       " [35, 19, 27, 33, 37, 1, 16, 25, 15, 17, 24],\n",
       " [19, 27, 33, 37, 1, 16, 25, 15, 17, 24, 16],\n",
       " [27, 33, 37, 1, 16, 25, 15, 17, 24, 16, 23],\n",
       " [33, 37, 1, 16, 25, 15, 17, 24, 16, 23, 31],\n",
       " [37, 1, 16, 25, 15, 17, 24, 16, 23, 31, 18],\n",
       " [1, 16, 25, 15, 17, 24, 16, 23, 31, 18, 32],\n",
       " [16, 25, 15, 17, 24, 16, 23, 31, 18, 32, 3],\n",
       " [25, 15, 17, 24, 16, 23, 31, 18, 32, 3, 1],\n",
       " [15, 17, 24, 16, 23, 31, 18, 32, 3, 1, 7],\n",
       " [17, 24, 16, 23, 31, 18, 32, 3, 1, 7, 15],\n",
       " [24, 16, 23, 31, 18, 32, 3, 1, 7, 15, 24],\n",
       " [16, 23, 31, 18, 32, 3, 1, 7, 15, 24, 19],\n",
       " [23, 31, 18, 32, 3, 1, 7, 15, 24, 19, 18],\n",
       " [31, 18, 32, 3, 1, 7, 15, 24, 19, 18, 1],\n",
       " [18, 32, 3, 1, 7, 15, 24, 19, 18, 1, 23],\n",
       " [32, 3, 1, 7, 15, 24, 19, 18, 1, 23, 27],\n",
       " [3, 1, 7, 15, 24, 19, 18, 1, 23, 27, 1],\n",
       " [1, 7, 15, 24, 19, 18, 1, 23, 27, 1, 15],\n",
       " [7, 15, 24, 19, 18, 1, 23, 27, 1, 15, 1],\n",
       " [15, 24, 19, 18, 1, 23, 27, 1, 15, 1, 29],\n",
       " [24, 19, 18, 1, 23, 27, 1, 15, 1, 29, 23],\n",
       " [19, 18, 1, 23, 27, 1, 15, 1, 29, 23, 19],\n",
       " [18, 1, 23, 27, 1, 15, 1, 29, 23, 19, 4],\n",
       " [1, 23, 27, 1, 15, 1, 29, 23, 19, 4, 1],\n",
       " [23, 27, 1, 15, 1, 29, 23, 19, 4, 1, 14],\n",
       " [27, 1, 15, 1, 29, 23, 19, 4, 1, 14, 22],\n",
       " [1, 15, 1, 29, 23, 19, 4, 1, 14, 22, 19],\n",
       " [15, 1, 29, 23, 19, 4, 1, 14, 22, 19, 27],\n",
       " [1, 29, 23, 19, 4, 1, 14, 22, 19, 27, 1],\n",
       " [29, 23, 19, 4, 1, 14, 22, 19, 27, 1, 33],\n",
       " [23, 19, 4, 1, 14, 22, 19, 27, 1, 33, 22],\n",
       " [19, 4, 1, 14, 22, 19, 27, 1, 33, 22, 19],\n",
       " [4, 1, 14, 22, 19, 27, 1, 33, 22, 19, 1],\n",
       " [1, 14, 22, 19, 27, 1, 33, 22, 19, 1, 29],\n",
       " [14, 22, 19, 27, 1, 33, 22, 19, 1, 29, 23],\n",
       " [22, 19, 27, 1, 33, 22, 19, 1, 29, 23, 19],\n",
       " [19, 27, 1, 33, 22, 19, 1, 29, 23, 19, 1],\n",
       " [27, 1, 33, 22, 19, 1, 29, 23, 19, 1, 35],\n",
       " [1, 33, 22, 19, 1, 29, 23, 19, 1, 35, 15],\n",
       " [33, 22, 19, 1, 29, 23, 19, 1, 35, 15, 32],\n",
       " [22, 19, 1, 29, 23, 19, 1, 35, 15, 32, 1],\n",
       " [19, 1, 29, 23, 19, 1, 35, 15, 32, 1, 28],\n",
       " [1, 29, 23, 19, 1, 35, 15, 32, 1, 28, 29],\n",
       " [29, 23, 19, 1, 35, 15, 32, 1, 28, 29, 19],\n",
       " [23, 19, 1, 35, 15, 32, 1, 28, 29, 19, 27],\n",
       " [19, 1, 35, 15, 32, 1, 28, 29, 19, 27, 19],\n",
       " [1, 35, 15, 32, 1, 28, 29, 19, 27, 19, 18],\n",
       " [35, 15, 32, 1, 28, 29, 19, 27, 19, 18, 1],\n",
       " [15, 32, 1, 28, 29, 19, 27, 19, 18, 1, 13],\n",
       " [32, 1, 28, 29, 19, 27, 19, 18, 1, 13, 22],\n",
       " [1, 28, 29, 19, 27, 19, 18, 1, 13, 22, 19],\n",
       " [28, 29, 19, 27, 19, 18, 1, 13, 22, 19, 1],\n",
       " [29, 19, 27, 19, 18, 1, 13, 22, 19, 1, 16],\n",
       " [19, 27, 19, 18, 1, 13, 22, 19, 1, 16, 23],\n",
       " [27, 19, 18, 1, 13, 22, 19, 1, 16, 23, 31],\n",
       " [19, 18, 1, 13, 22, 19, 1, 16, 23, 31, 18],\n",
       " [18, 1, 13, 22, 19, 1, 16, 23, 31, 18, 32],\n",
       " [1, 13, 22, 19, 1, 16, 23, 31, 18, 32, 1],\n",
       " [13, 22, 19, 1, 16, 23, 31, 18, 32, 1, 16],\n",
       " [22, 19, 1, 16, 23, 31, 18, 32, 1, 16, 19],\n",
       " [19, 1, 16, 23, 31, 18, 32, 1, 16, 19, 21],\n",
       " [1, 16, 23, 31, 18, 32, 1, 16, 19, 21, 15],\n",
       " [16, 23, 31, 18, 32, 1, 16, 19, 21, 15, 27],\n",
       " [23, 31, 18, 32, 1, 16, 19, 21, 15, 27, 1],\n",
       " [31, 18, 32, 1, 16, 19, 21, 15, 27, 1, 33],\n",
       " [18, 32, 1, 16, 19, 21, 15, 27, 1, 33, 28],\n",
       " [32, 1, 16, 19, 21, 15, 27, 1, 33, 28, 1],\n",
       " [1, 16, 19, 21, 15, 27, 1, 33, 28, 1, 32],\n",
       " [16, 19, 21, 15, 27, 1, 33, 28, 1, 32, 23],\n",
       " [19, 21, 15, 27, 1, 33, 28, 1, 32, 23, 27],\n",
       " [21, 15, 27, 1, 33, 28, 1, 32, 23, 27, 21],\n",
       " [15, 27, 1, 33, 28, 1, 32, 23, 27, 21, 5],\n",
       " [27, 1, 33, 28, 1, 32, 23, 27, 21, 5, 1],\n",
       " [1, 33, 28, 1, 32, 23, 27, 21, 5, 1, 14],\n",
       " [33, 28, 1, 32, 23, 27, 21, 5, 1, 14, 15],\n",
       " [28, 1, 32, 23, 27, 21, 5, 1, 14, 15, 32],\n",
       " [1, 32, 23, 27, 21, 5, 1, 14, 15, 32, 27],\n",
       " [32, 23, 27, 21, 5, 1, 14, 15, 32, 27, 2],\n",
       " [23, 27, 21, 5, 1, 14, 15, 32, 27, 2, 33],\n",
       " [27, 21, 5, 1, 14, 15, 32, 27, 2, 33, 1],\n",
       " [21, 5, 1, 14, 15, 32, 27, 2, 33, 1, 33],\n",
       " [5, 1, 14, 15, 32, 27, 2, 33, 1, 33, 22],\n",
       " [1, 14, 15, 32, 27, 2, 33, 1, 33, 22, 15],\n",
       " [14, 15, 32, 27, 2, 33, 1, 33, 22, 15, 33],\n",
       " [15, 32, 27, 2, 33, 1, 33, 22, 15, 33, 1],\n",
       " [32, 27, 2, 33, 1, 33, 22, 15, 33, 1, 15],\n",
       " [27, 2, 33, 1, 33, 22, 15, 33, 1, 15, 1],\n",
       " [2, 33, 1, 33, 22, 15, 33, 1, 15, 1, 18],\n",
       " [33, 1, 33, 22, 15, 33, 1, 15, 1, 18, 15],\n",
       " [1, 33, 22, 15, 33, 1, 15, 1, 18, 15, 23],\n",
       " [33, 22, 15, 33, 1, 15, 1, 18, 15, 23, 27],\n",
       " [22, 15, 33, 1, 15, 1, 18, 15, 23, 27, 33],\n",
       " [15, 33, 1, 15, 1, 18, 15, 23, 27, 33, 37],\n",
       " [33, 1, 15, 1, 18, 15, 23, 27, 33, 37, 1],\n",
       " [1, 15, 1, 18, 15, 23, 27, 33, 37, 1, 18],\n",
       " [15, 1, 18, 15, 23, 27, 33, 37, 1, 18, 23],\n",
       " [1, 18, 15, 23, 27, 33, 37, 1, 18, 23, 32],\n",
       " [18, 15, 23, 27, 33, 37, 1, 18, 23, 32, 22],\n",
       " [15, 23, 27, 33, 37, 1, 18, 23, 32, 22, 3],\n",
       " [23, 27, 33, 37, 1, 18, 23, 32, 22, 3, 1],\n",
       " [27, 33, 37, 1, 18, 23, 32, 22, 3, 1, 13],\n",
       " [33, 37, 1, 18, 23, 32, 22, 3, 1, 13, 28],\n",
       " [37, 1, 18, 23, 32, 22, 3, 1, 13, 28, 1],\n",
       " [1, 18, 23, 32, 22, 3, 1, 13, 28, 1, 32],\n",
       " [18, 23, 32, 22, 3, 1, 13, 28, 1, 32, 19],\n",
       " [23, 32, 22, 3, 1, 13, 28, 1, 32, 19, 33],\n",
       " [32, 22, 3, 1, 13, 28, 1, 32, 19, 33, 1],\n",
       " [22, 3, 1, 13, 28, 1, 32, 19, 33, 1, 16],\n",
       " [3, 1, 13, 28, 1, 32, 19, 33, 1, 16, 19],\n",
       " [1, 13, 28, 1, 32, 19, 33, 1, 16, 19, 20],\n",
       " [13, 28, 1, 32, 19, 33, 1, 16, 19, 20, 28],\n",
       " [28, 1, 32, 19, 33, 1, 16, 19, 20, 28, 31],\n",
       " [1, 32, 19, 33, 1, 16, 19, 20, 28, 31, 19],\n",
       " [32, 19, 33, 1, 16, 19, 20, 28, 31, 19, 1],\n",
       " [19, 33, 1, 16, 19, 20, 28, 31, 19, 1, 33],\n",
       " [33, 1, 16, 19, 20, 28, 31, 19, 1, 33, 22],\n",
       " [1, 16, 19, 20, 28, 31, 19, 1, 33, 22, 19],\n",
       " [16, 19, 20, 28, 31, 19, 1, 33, 22, 19, 1],\n",
       " [19, 20, 28, 31, 19, 1, 33, 22, 19, 1, 24],\n",
       " [20, 28, 31, 19, 1, 33, 22, 19, 1, 24, 23],\n",
       " [28, 31, 19, 1, 33, 22, 19, 1, 24, 23, 27],\n",
       " [31, 19, 1, 33, 22, 19, 1, 24, 23, 27, 21],\n",
       " [19, 1, 33, 22, 19, 1, 24, 23, 27, 21, 4],\n",
       " [1, 33, 22, 19, 1, 24, 23, 27, 21, 4, 1],\n",
       " [33, 22, 19, 1, 24, 23, 27, 21, 4, 1, 13],\n",
       " [22, 19, 1, 24, 23, 27, 21, 4, 1, 13, 22],\n",
       " [19, 1, 24, 23, 27, 21, 4, 1, 13, 22, 19],\n",
       " [1, 24, 23, 27, 21, 4, 1, 13, 22, 19, 1],\n",
       " [24, 23, 27, 21, 4, 1, 13, 22, 19, 1, 24],\n",
       " [23, 27, 21, 4, 1, 13, 22, 19, 1, 24, 23],\n",
       " [27, 21, 4, 1, 13, 22, 19, 1, 24, 23, 27],\n",
       " [21, 4, 1, 13, 22, 19, 1, 24, 23, 27, 21],\n",
       " [4, 1, 13, 22, 19, 1, 24, 23, 27, 21, 1],\n",
       " [1, 13, 22, 19, 1, 24, 23, 27, 21, 1, 35],\n",
       " [13, 22, 19, 1, 24, 23, 27, 21, 1, 35, 15],\n",
       " [22, 19, 1, 24, 23, 27, 21, 1, 35, 15, 32],\n",
       " [19, 1, 24, 23, 27, 21, 1, 35, 15, 32, 1],\n",
       " [1, 24, 23, 27, 21, 1, 35, 15, 32, 1, 23],\n",
       " [24, 23, 27, 21, 1, 35, 15, 32, 1, 23, 27],\n",
       " [23, 27, 21, 1, 35, 15, 32, 1, 23, 27, 1],\n",
       " [27, 21, 1, 35, 15, 32, 1, 23, 27, 1, 22],\n",
       " [21, 1, 35, 15, 32, 1, 23, 27, 1, 22, 23],\n",
       " [1, 35, 15, 32, 1, 23, 27, 1, 22, 23, 32],\n",
       " [35, 15, 32, 1, 23, 27, 1, 22, 23, 32, 1],\n",
       " [15, 32, 1, 23, 27, 1, 22, 23, 32, 1, 17],\n",
       " [32, 1, 23, 27, 1, 22, 23, 32, 1, 17, 28],\n",
       " [1, 23, 27, 1, 22, 23, 32, 1, 17, 28, 34],\n",
       " [23, 27, 1, 22, 23, 32, 1, 17, 28, 34, 27],\n",
       " [27, 1, 22, 23, 32, 1, 17, 28, 34, 27, 33],\n",
       " [1, 22, 23, 32, 1, 17, 28, 34, 27, 33, 23],\n",
       " [22, 23, 32, 1, 17, 28, 34, 27, 33, 23, 27],\n",
       " [23, 32, 1, 17, 28, 34, 27, 33, 23, 27, 21],\n",
       " [32, 1, 17, 28, 34, 27, 33, 23, 27, 21, 1],\n",
       " [1, 17, 28, 34, 27, 33, 23, 27, 21, 1, 22],\n",
       " [17, 28, 34, 27, 33, 23, 27, 21, 1, 22, 28],\n",
       " [28, 34, 27, 33, 23, 27, 21, 1, 22, 28, 34],\n",
       " [34, 27, 33, 23, 27, 21, 1, 22, 28, 34, 32],\n",
       " [27, 33, 23, 27, 21, 1, 22, 28, 34, 32, 19],\n",
       " [33, 23, 27, 21, 1, 22, 28, 34, 32, 19, 3],\n",
       " [23, 27, 21, 1, 22, 28, 34, 32, 19, 3, 1],\n",
       " [27, 21, 1, 22, 28, 34, 32, 19, 3, 1, 8],\n",
       " [21, 1, 22, 28, 34, 32, 19, 3, 1, 8, 28],\n",
       " [1, 22, 28, 34, 32, 19, 3, 1, 8, 28, 34],\n",
       " [22, 28, 34, 32, 19, 3, 1, 8, 28, 34, 27],\n",
       " [28, 34, 32, 19, 3, 1, 8, 28, 34, 27, 33],\n",
       " [34, 32, 19, 3, 1, 8, 28, 34, 27, 33, 23],\n",
       " [32, 19, 3, 1, 8, 28, 34, 27, 33, 23, 27],\n",
       " [19, 3, 1, 8, 28, 34, 27, 33, 23, 27, 21],\n",
       " [3, 1, 8, 28, 34, 27, 33, 23, 27, 21, 1],\n",
       " [1, 8, 28, 34, 27, 33, 23, 27, 21, 1, 28],\n",
       " [8, 28, 34, 27, 33, 23, 27, 21, 1, 28, 34],\n",
       " [28, 34, 27, 33, 23, 27, 21, 1, 28, 34, 33],\n",
       " [34, 27, 33, 23, 27, 21, 1, 28, 34, 33, 1],\n",
       " [27, 33, 23, 27, 21, 1, 28, 34, 33, 1, 22],\n",
       " [33, 23, 27, 21, 1, 28, 34, 33, 1, 22, 23],\n",
       " [23, 27, 21, 1, 28, 34, 33, 1, 22, 23, 32],\n",
       " [27, 21, 1, 28, 34, 33, 1, 22, 23, 32, 1],\n",
       " [21, 1, 28, 34, 33, 1, 22, 23, 32, 1, 26],\n",
       " [1, 28, 34, 33, 1, 22, 23, 32, 1, 26, 28],\n",
       " [28, 34, 33, 1, 22, 23, 32, 1, 26, 28, 27],\n",
       " [34, 33, 1, 22, 23, 32, 1, 26, 28, 27, 19],\n",
       " [33, 1, 22, 23, 32, 1, 26, 28, 27, 19, 37],\n",
       " [1, 22, 23, 32, 1, 26, 28, 27, 19, 37, 5],\n",
       " [22, 23, 32, 1, 26, 28, 27, 19, 37, 5, 1],\n",
       " [23, 32, 1, 26, 28, 27, 19, 37, 5, 1, 13],\n",
       " [32, 1, 26, 28, 27, 19, 37, 5, 1, 13, 22],\n",
       " [1, 26, 28, 27, 19, 37, 5, 1, 13, 22, 19],\n",
       " [26, 28, 27, 19, 37, 5, 1, 13, 22, 19, 1],\n",
       " [28, 27, 19, 37, 5, 1, 13, 22, 19, 1, 30],\n",
       " [27, 19, 37, 5, 1, 13, 22, 19, 1, 30, 34],\n",
       " [19, 37, 5, 1, 13, 22, 19, 1, 30, 34, 19],\n",
       " [37, 5, 1, 13, 22, 19, 1, 30, 34, 19, 19],\n",
       " [5, 1, 13, 22, 19, 1, 30, 34, 19, 19, 27],\n",
       " [1, 13, 22, 19, 1, 30, 34, 19, 19, 27, 1],\n",
       " [13, 22, 19, 1, 30, 34, 19, 19, 27, 1, 35],\n",
       " [22, 19, 1, 30, 34, 19, 19, 27, 1, 35, 15],\n",
       " [19, 1, 30, 34, 19, 19, 27, 1, 35, 15, 32],\n",
       " [1, 30, 34, 19, 19, 27, 1, 35, 15, 32, 1],\n",
       " [30, 34, 19, 19, 27, 1, 35, 15, 32, 1, 23],\n",
       " [34, 19, 19, 27, 1, 35, 15, 32, 1, 23, 27],\n",
       " [19, 19, 27, 1, 35, 15, 32, 1, 23, 27, 1],\n",
       " [19, 27, 1, 35, 15, 32, 1, 23, 27, 1, 33],\n",
       " [27, 1, 35, 15, 32, 1, 23, 27, 1, 33, 22],\n",
       " [1, 35, 15, 32, 1, 23, 27, 1, 33, 22, 19],\n",
       " [35, 15, 32, 1, 23, 27, 1, 33, 22, 19, 1],\n",
       " [15, 32, 1, 23, 27, 1, 33, 22, 19, 1, 29],\n",
       " [32, 1, 23, 27, 1, 33, 22, 19, 1, 29, 15],\n",
       " [1, 23, 27, 1, 33, 22, 19, 1, 29, 15, 31],\n",
       " [23, 27, 1, 33, 22, 19, 1, 29, 15, 31, 25],\n",
       " [27, 1, 33, 22, 19, 1, 29, 15, 31, 25, 28],\n",
       " [1, 33, 22, 19, 1, 29, 15, 31, 25, 28, 34],\n",
       " [33, 22, 19, 1, 29, 15, 31, 25, 28, 34, 31],\n",
       " [22, 19, 1, 29, 15, 31, 25, 28, 34, 31, 3],\n",
       " [19, 1, 29, 15, 31, 25, 28, 34, 31, 3, 1],\n",
       " [1, 29, 15, 31, 25, 28, 34, 31, 3, 1, 9],\n",
       " [29, 15, 31, 25, 28, 34, 31, 3, 1, 9, 15],\n",
       " [15, 31, 25, 28, 34, 31, 3, 1, 9, 15, 33],\n",
       " [31, 25, 28, 34, 31, 3, 1, 9, 15, 33, 23],\n",
       " [25, 28, 34, 31, 3, 1, 9, 15, 33, 23, 27],\n",
       " [28, 34, 31, 3, 1, 9, 15, 33, 23, 27, 21],\n",
       " [34, 31, 3, 1, 9, 15, 33, 23, 27, 21, 1],\n",
       " [31, 3, 1, 9, 15, 33, 23, 27, 21, 1, 16],\n",
       " [3, 1, 9, 15, 33, 23, 27, 21, 1, 16, 31],\n",
       " [1, 9, 15, 33, 23, 27, 21, 1, 16, 31, 19],\n",
       " [9, 15, 33, 23, 27, 21, 1, 16, 31, 19, 15],\n",
       " [15, 33, 23, 27, 21, 1, 16, 31, 19, 15, 18],\n",
       " [33, 23, 27, 21, 1, 16, 31, 19, 15, 18, 1],\n",
       " [23, 27, 21, 1, 16, 31, 19, 15, 18, 1, 15],\n",
       " [27, 21, 1, 16, 31, 19, 15, 18, 1, 15, 27],\n",
       " [21, 1, 16, 31, 19, 15, 18, 1, 15, 27, 18],\n",
       " [1, 16, 31, 19, 15, 18, 1, 15, 27, 18, 1],\n",
       " [16, 31, 19, 15, 18, 1, 15, 27, 18, 1, 22],\n",
       " [31, 19, 15, 18, 1, 15, 27, 18, 1, 22, 28],\n",
       " [19, 15, 18, 1, 15, 27, 18, 1, 22, 28, 27],\n",
       " [15, 18, 1, 15, 27, 18, 1, 22, 28, 27, 19],\n",
       " [18, 1, 15, 27, 18, 1, 22, 28, 27, 19, 37],\n",
       " [1, 15, 27, 18, 1, 22, 28, 27, 19, 37, 4],\n",
       " [15, 27, 18, 1, 22, 28, 27, 19, 37, 4, 1],\n",
       " [27, 18, 1, 22, 28, 27, 19, 37, 4, 1, 13],\n",
       " [18, 1, 22, 28, 27, 19, 37, 4, 1, 13, 22],\n",
       " [1, 22, 28, 27, 19, 37, 4, 1, 13, 22, 19],\n",
       " [22, 28, 27, 19, 37, 4, 1, 13, 22, 19, 1],\n",
       " [28, 27, 19, 37, 4, 1, 13, 22, 19, 1, 26],\n",
       " [27, 19, 37, 4, 1, 13, 22, 19, 1, 26, 15],\n",
       " [19, 37, 4, 1, 13, 22, 19, 1, 26, 15, 23],\n",
       " [37, 4, 1, 13, 22, 19, 1, 26, 15, 23, 18],\n",
       " [4, 1, 13, 22, 19, 1, 26, 15, 23, 18, 1],\n",
       " [1, 13, 22, 19, 1, 26, 15, 23, 18, 1, 35],\n",
       " [13, 22, 19, 1, 26, 15, 23, 18, 1, 35, 15],\n",
       " [22, 19, 1, 26, 15, 23, 18, 1, 35, 15, 32],\n",
       " [19, 1, 26, 15, 23, 18, 1, 35, 15, 32, 1],\n",
       " [1, 26, 15, 23, 18, 1, 35, 15, 32, 1, 23],\n",
       " [26, 15, 23, 18, 1, 35, 15, 32, 1, 23, 27],\n",
       " [15, 23, 18, 1, 35, 15, 32, 1, 23, 27, 1],\n",
       " [23, 18, 1, 35, 15, 32, 1, 23, 27, 1, 33],\n",
       " [18, 1, 35, 15, 32, 1, 23, 27, 1, 33, 22],\n",
       " [1, 35, 15, 32, 1, 23, 27, 1, 33, 22, 19],\n",
       " [35, 15, 32, 1, 23, 27, 1, 33, 22, 19, 1],\n",
       " [15, 32, 1, 23, 27, 1, 33, 22, 19, 1, 21],\n",
       " [32, 1, 23, 27, 1, 33, 22, 19, 1, 21, 15],\n",
       " [1, 23, 27, 1, 33, 22, 19, 1, 21, 15, 31],\n",
       " [23, 27, 1, 33, 22, 19, 1, 21, 15, 31, 18],\n",
       " [27, 1, 33, 22, 19, 1, 21, 15, 31, 18, 19],\n",
       " [1, 33, 22, 19, 1, 21, 15, 31, 18, 19, 27],\n",
       " [33, 22, 19, 1, 21, 15, 31, 18, 19, 27, 3],\n",
       " [22, 19, 1, 21, 15, 31, 18, 19, 27, 3, 1],\n",
       " [19, 1, 21, 15, 31, 18, 19, 27, 3, 1, 11],\n",
       " [1, 21, 15, 31, 18, 19, 27, 3, 1, 11, 15],\n",
       " [21, 15, 31, 18, 19, 27, 3, 1, 11, 15, 27],\n",
       " [15, 31, 18, 19, 27, 3, 1, 11, 15, 27, 21],\n",
       " [31, 18, 19, 27, 3, 1, 11, 15, 27, 21, 23],\n",
       " [18, 19, 27, 3, 1, 11, 15, 27, 21, 23, 27],\n",
       " [19, 27, 3, 1, 11, 15, 27, 21, 23, 27, 21],\n",
       " [27, 3, 1, 11, 15, 27, 21, 23, 27, 21, 1],\n",
       " [3, 1, 11, 15, 27, 21, 23, 27, 21, 1, 28],\n",
       " [1, 11, 15, 27, 21, 23, 27, 21, 1, 28, 34],\n",
       " [11, 15, 27, 21, 23, 27, 21, 1, 28, 34, 33],\n",
       " [15, 27, 21, 23, 27, 21, 1, 28, 34, 33, 1],\n",
       " [27, 21, 23, 27, 21, 1, 28, 34, 33, 1, 33],\n",
       " [21, 23, 27, 21, 1, 28, 34, 33, 1, 33, 22],\n",
       " [23, 27, 21, 1, 28, 34, 33, 1, 33, 22, 19],\n",
       " [27, 21, 1, 28, 34, 33, 1, 33, 22, 19, 1],\n",
       " [21, 1, 28, 34, 33, 1, 33, 22, 19, 1, 17],\n",
       " [1, 28, 34, 33, 1, 33, 22, 19, 1, 17, 25],\n",
       " [28, 34, 33, 1, 33, 22, 19, 1, 17, 25, 28],\n",
       " [34, 33, 1, 33, 22, 19, 1, 17, 25, 28, 33],\n",
       " [33, 1, 33, 22, 19, 1, 17, 25, 28, 33, 22],\n",
       " [1, 33, 22, 19, 1, 17, 25, 28, 33, 22, 19],\n",
       " [33, 22, 19, 1, 17, 25, 28, 33, 22, 19, 32],\n",
       " [22, 19, 1, 17, 25, 28, 33, 22, 19, 32, 3],\n",
       " [19, 1, 17, 25, 28, 33, 22, 19, 32, 3, 1],\n",
       " [1, 17, 25, 28, 33, 22, 19, 32, 3, 1, 14],\n",
       " [17, 25, 28, 33, 22, 19, 32, 3, 1, 14, 22],\n",
       " [25, 28, 33, 22, 19, 32, 3, 1, 14, 22, 19],\n",
       " [28, 33, 22, 19, 32, 3, 1, 14, 22, 19, 27],\n",
       " [33, 22, 19, 32, 3, 1, 14, 22, 19, 27, 1],\n",
       " [22, 19, 32, 3, 1, 14, 22, 19, 27, 1, 18],\n",
       " [19, 32, 3, 1, 14, 22, 19, 27, 1, 18, 28],\n",
       " [32, 3, 1, 14, 22, 19, 27, 1, 18, 28, 35],\n",
       " [3, 1, 14, 22, 19, 27, 1, 18, 28, 35, 27],\n",
       " [1, 14, 22, 19, 27, 1, 18, 28, 35, 27, 1],\n",
       " [14, 22, 19, 27, 1, 18, 28, 35, 27, 1, 17],\n",
       " [22, 19, 27, 1, 18, 28, 35, 27, 1, 17, 15],\n",
       " [19, 27, 1, 18, 28, 35, 27, 1, 17, 15, 26],\n",
       " [27, 1, 18, 28, 35, 27, 1, 17, 15, 26, 19],\n",
       " [1, 18, 28, 35, 27, 1, 17, 15, 26, 19, 1],\n",
       " [18, 28, 35, 27, 1, 17, 15, 26, 19, 1, 15],\n",
       " [28, 35, 27, 1, 17, 15, 26, 19, 1, 15, 1],\n",
       " [35, 27, 1, 17, 15, 26, 19, 1, 15, 1, 16],\n",
       " [27, 1, 17, 15, 26, 19, 1, 15, 1, 16, 25],\n",
       " [1, 17, 15, 26, 19, 1, 15, 1, 16, 25, 15],\n",
       " [17, 15, 26, 19, 1, 15, 1, 16, 25, 15, 17],\n",
       " [15, 26, 19, 1, 15, 1, 16, 25, 15, 17, 24],\n",
       " [26, 19, 1, 15, 1, 16, 25, 15, 17, 24, 16],\n",
       " [19, 1, 15, 1, 16, 25, 15, 17, 24, 16, 23],\n",
       " [1, 15, 1, 16, 25, 15, 17, 24, 16, 23, 31],\n",
       " [15, 1, 16, 25, 15, 17, 24, 16, 23, 31, 18],\n",
       " [1, 16, 25, 15, 17, 24, 16, 23, 31, 18, 1],\n",
       " [16, 25, 15, 17, 24, 16, 23, 31, 18, 1, 6],\n",
       " [25, 15, 17, 24, 16, 23, 31, 18, 1, 6, 27],\n",
       " [15, 17, 24, 16, 23, 31, 18, 1, 6, 27, 18],\n",
       " [17, 24, 16, 23, 31, 18, 1, 6, 27, 18, 1],\n",
       " [24, 16, 23, 31, 18, 1, 6, 27, 18, 1, 29],\n",
       " [16, 23, 31, 18, 1, 6, 27, 18, 1, 29, 19],\n",
       " [23, 31, 18, 1, 6, 27, 18, 1, 29, 19, 17],\n",
       " [31, 18, 1, 6, 27, 18, 1, 29, 19, 17, 24],\n",
       " [18, 1, 6, 27, 18, 1, 29, 19, 17, 24, 19],\n",
       " [1, 6, 27, 18, 1, 29, 19, 17, 24, 19, 18],\n",
       " [6, 27, 18, 1, 29, 19, 17, 24, 19, 18, 1],\n",
       " [27, 18, 1, 29, 19, 17, 24, 19, 18, 1, 28],\n",
       " [18, 1, 29, 19, 17, 24, 19, 18, 1, 28, 20],\n",
       " [1, 29, 19, 17, 24, 19, 18, 1, 28, 20, 20],\n",
       " [29, 19, 17, 24, 19, 18, 1, 28, 20, 20, 1],\n",
       " [19, 17, 24, 19, 18, 1, 28, 20, 20, 1, 22],\n",
       " [17, 24, 19, 18, 1, 28, 20, 20, 1, 22, 19],\n",
       " [24, 19, 18, 1, 28, 20, 20, 1, 22, 19, 31],\n",
       " [19, 18, 1, 28, 20, 20, 1, 22, 19, 31, 1],\n",
       " [18, 1, 28, 20, 20, 1, 22, 19, 31, 1, 27],\n",
       " [1, 28, 20, 20, 1, 22, 19, 31, 1, 27, 28],\n",
       " [28, 20, 20, 1, 22, 19, 31, 1, 27, 28, 32],\n",
       " [20, 20, 1, 22, 19, 31, 1, 27, 28, 32, 19],\n",
       " [20, 1, 22, 19, 31, 1, 27, 28, 32, 19, 4]]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# encode sequence value for characteers.\n",
    "sequence = list()\n",
    "for line in lines:\n",
    "    # integer encode line\n",
    "    encoded_seq = [mapping[char] for char in line]\n",
    "    # store\n",
    "    sequence.append(encoded_seq)\n",
    "sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size: 38\n"
     ]
    }
   ],
   "source": [
    "# Get the size of the vocabulary\n",
    "vocab_size = len(mapping)\n",
    "print('Vocabulary size: %d'% vocab_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split Inputs and Outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the sequences have been integer encoded, we can seperate the columns into input and output sequences of characters, using array slices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy import array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((399, 10), (399,))"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequences = array(sequence)\n",
    "X = sequences[:,:-1]\n",
    "y = sequences[:,-1]\n",
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to one-hot-encode each character. That is, aech character becomes a vector as long as the vocabulary (38 elements) with a 1 marked for the specific charater.\n",
    "\n",
    "This provides a more precise input representation for the network. It also provides a clear objective for the network to predict, where a probability distribution over characters can be output by the model and compared to the ideal case of all 0 values with a 1 for the actual next character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils import to_categorical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((399, 10, 38), (399, 38))"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seq = [to_categorical(x, num_classes=vocab_size) for x in X]\n",
    "X = array(seq)\n",
    "y = to_categorical(y, num_classes=vocab_size)\n",
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 1., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 1., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 1., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 1., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 1., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]]], dtype=float32)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports for keras model\n",
    "from keras.layers import Dense, Input, LSTM\n",
    "from keras.models import Model, Sequential"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit the model\n",
    "def model_seq(X):\n",
    "    model = Sequential()\n",
    "    model.add(LSTM(75, input_shape=(X.shape[1], X.shape[2])))\n",
    "    model.add(Dense(vocab_size, activation='softmax'))\n",
    "    # model compile\n",
    "    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    # summarize defined model\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm_1 (LSTM)                (None, 75)                34200     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 38)                2888      \n",
      "=================================================================\n",
      "Total params: 37,088\n",
      "Trainable params: 37,088\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = model_seq(X)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaUAAAEnCAYAAAAJqlx9AAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzde1hU5fo//vcwIGdQwANuzLAkA9qoeQJJcCd4NhMIDxiajKJiipJHcovYB9BSMxSU0uyDKQpGml+3eEQRcYsnSrQ0KkPwgAJyEga5f3/wm/VxnEGG48zg/bouLq951pq17rVw5mat9TzPLSIiAmOMMaYBdNQdAGOMMSbDSYkxxpjG4KTEGGNMY3BSYowxpjF0n284d+4c1q9fr45YGGOMvUScnZ2xcOFCuTaFK6W///4biYmJrRYUY21Jbm4uf35UkJiYiNzcXHWHwdQoIyMD586dU2hXuFKS2bdvX4sGxFhbtHfvXvj6+vLnpx4ikQjBwcH44IMP1B0KUxMfHx+l7fxMiTHGmMbgpMQYY0xjcFJijDGmMTgpMcYY0xiclBhjjGkMTkqMaaDk5GR069YN169fV3coGiclJQUHDx7Enj174ODgAJFIBFdXV1RXV8ut9+jRIyxduhSmpqYwNDTEwoULUVBQoKao61dUVITQ0FAsW7ZM6fLMzEx4eXkhJCQEM2fOxM6dOxu9r3379qFv374wMTGBk5MTfvzxR6XrSCQSLFu2DJMmTcLKlSshlUoBAJcvX8bGjRvREvN519klnDGmPsbGxujUqRMMDAzUFkN+fj6sra3Vtn9lYmJiAACzZ88GAAwbNgzW1tY4e/YsFi9eLDfw38LCApGRkaisrERFRYVGTwpw8OBBxMfHY+/evQgKClJYfvXqVbi7u+Po0aNwdnZGRUUFevfujYqKCgQGBjZoX99++y0uXryIHTt24NGjR1i0aBF8fHxw7do19OzZE0Dt0Ia1a9fi/PnzEIvFICKMHj0aoaGhiIqKQp8+fVBUVIQlS5Zg7dq1zXIOZPhKiTEN5OHhgYsXL8LW1lYt+y8sLISfn59a9l2XQ4cO4cSJE0JCAgArKyvo6tb+bb1hwwYkJSUpvK979+54/fXXWy3Oxhg7dizi4uLqXL5o0SIMHDgQzs7OAABDQ0PMnz8fn3zyCUpKSlTej1Qqxa1bt/DVV1/ByckJQ4cOxddffw2pVIrz588L623btg3Ozs4Qi8UAaseVjRw5Uu6KaujQoTA1NcXmzZsbergvxEmJMSanvLwcEydORE5OjrpDEZSUlGDGjBkICwtTWGZnZ4f33nsPAPDRRx/h5s2bcssNDQ1haGjYKnE2hb6+vtL2/Px8HD9+HG5ubnLt77zzDkpLSxEfH6/yPnR0dLBq1Sq5NktLSwBA//79hbaSkhIcO3ZMuF0HAFlZWfjHP/4h996FCxdi9erVzfp/hZMSYxqmsLAQ33zzDTw8PJCcnAwAuHLlCj755BP06NEDZWVlCAgIgJWVFQYMGCB8IWRnZ2PFihWwt7dHXl4exo8fDwsLCwwYMAAZGRkAgN27d8PMzAzdunUDABQXFyM8PBxisVj4K/yHH37A9evXUVBQAIlEgs8//xwAcPbsWXTr1g2HDx9u7VOCuLg46Ovrw97eXmGZjo4O4uPj4ejoiMePH8PLywsVFRUv3F5SUhKCgoIQEhKCkSNHIjQ0FJWVlQBUO9cAQESIjY3F7NmzMXDgQHh6eiokxOaQnZ0NAApXe7Jbbenp6SpvSywWC1eWMt9//z2++uorvPHGG0KbRCLBjRs3MGrUKBQXFyMjIwPnz5/Hhg0b5N5rbGyMfv364bPPPmvQMb0QPSchIYGUNDPGVNAcn5/s7GwKDg4mAJSYmEhERPn5+TRs2DACQHPnzqVr167R5cuXSV9fnyZOnEhEREuXLqX27duTWCym4OBgOnnyJCUlJZGVlRUZGRlRXl4eERF5enqSjY2N3D7feustGjRokPB6zJgx9Oqrr8qtc+jQITI0NKRdu3Y16fiIiABQQkKCyus7OzuTj4+P0mW9e/cmIqKcnByytLQkAOTv7y8sj42NpejoaOH1hg0byMXFhaqqqoiIqKCggHr27Elubm5UU1Oj0rkmIoqIiKBvv/2WiIiqq6vJ3t6eunTpQmVlZSof17OePHlCACgoKEiuPTo6mgDQTz/9pPAefX19cnNza9T+SkpKKCwsjDp37kxHjhxRWB4WFkYAyM7OjsaMGUOFhYVKtxMeHk7m5uZUXV3doP17e3uTt7e3QjtfKTGmYd58803hdpRMly5dhNsrYWFhsLe3R+/evdG/f39cvHgRABAREYFRo0ZBR0cHUVFRcHd3x4QJExATE4Py8nLExsYCAIyMjBT2aWxsXG9co0aNQklJCSZPntzUQ2yQmpoaZGZmCreZ6mJra4vExETo6elh586dSp/R3L9/H6GhoQgMDISenh6A2ttXy5cvR2pqKnbt2qXSuc7Ly8PGjRsxdepUALVXIN7e3rh79y4OHjzYnIePO3fuAABMTEwUlpmYmODevXsN3mZZWRnCwsJw/vx5FBYWYvjw4di+fbvcOitXroSLiwtu3ryJU6dO4dKlS0q31blzZxQXFwtXdE3FSYkxDfT8LRYAwkPnZ5fZ2NjIPeg2MjKCWCwWvnABYPz48dDX18fPP//c5LhkMbSmwsJCSKVSdOjQod513d3dsWnTJgDAvHnzFL5IMzIyUFZWhldeeUWufcyYMQCAkydPAqj/XKenp0MqlWLWrFmQSCSQSCTIy8tDQEBAsz+/kt1qLS8vV1hWXl6ucCyqMDY2xrp163Do0CFcvHgRFhYWcrfgqqurMX36dEybNg1HjhyBvr4+Ro0apbTrePv27QGgUclRGe4Szlgbp6uri65duyqM49EWsgTx9OlTldYPDAxEVlYWYmJi4OPjA4lEAlNTUwDAX3/9BaB2DNOzrKysYGRkhLy8PJX2cf36dRgbG7+wx1xzkT1LKi4ulmuvqqpCRUWF3LOgxnB0dMT8+fOxatUqSKVS6OnpYcGCBbh9+zZ27NgBADh9+jQ8PDzg7++P27dvw8zMTHi/jk7ttU1NTU2T4hC21yxbYYxptPLycvTq1UvdYTSKubk5DAwMUFRUpHQ5KRnAuWnTJri7uyMnJ0fuCkDWxb6u3mKqniMjIyPk5uYqrQn14MEDlbahKkdHR4jFYvz5559y7X/88QcA1WOubx82NjbCFfaePXuEji8AYG9vj4iICBQXF+Py5cty75Ul+C5dujQ5DoCTEmNtXn5+Ph48eABvb28AtVdOpaWlclcepaWlcn/p6ujooLS0VGFbzfXXcEOIRCK4uLgovYohIqW3tXR1dZGYmAhbW1u543B2doaZmZnQq1EmNzcX5eXlGDdunEoxvfXWWyAiLFmyRK79999/x5YtW1Tahqqsra3h6+uL1NRUufbU1FS0a9cOXl5eTd7HjRs35I7dyspKYfxTv379AACdOnWSay8oKICZmRkcHByaHAfASYkxjZSfnw9A/q9u2e2bZ2/D3b9/X+FLubKyElevXhVer1mzBv7+/hgwYACA2i/UoqIiRERE4LfffsOaNWtQWVmJX3/9VfgruGvXrigoKMDFixdx6tQplJeX49ixY+jQoYNaKutOnjwZ6enpCldFubm5uHv3rtx4GhlLS0scOHBAroOApaUloqKicPbsWRw/flxo37RpE/z9/TF06FAA9Z9rDw8P9O/fH99//z28vLwQHx+PLVu2YNasWZg7dy4AYM6cOXB1dcWtW7dUOsaysjIAym9TLlu2DGlpabhy5QqA2lt30dHRCA0NRefOnVXeX1FREaZPn479+/cL5/LWrVtITU1FVFSUsN7MmTOxe/duuf9/KSkpeOeddxRuF6anp8PLy6v5njc+3x2Pu4Qz1njN8fk5fvw4DRkyhABQv379KCUlhY4dO0avvvoqAaA5c+bQ/fv36bvvviMTExMCQKtWraLq6moKCAigdu3aUXBwMPn4+NCMGTMoPDycampqhO0XFxfT2LFjycTEhAYNGkQXLlygadOmkZ+fHx04cICIiK5evUo2NjZkZ2dH+/btIyKiEydOkLW1NSUnJzfp+Iga3iW8qqqKevbsSenp6ULb/v37yc3NjQCQt7c3nTlzRul7k5OTafPmzQptnp6eFBQURJ9++il98cUXwjlS9Vw/fPiQpkyZQp06daKOHTvShx9+SHfu3BH2MXr0aNLR0aElS5bUe3wpKSnk5+dHAKhHjx60detWoQu/zIULF8jX15eWLVtGkyZNoujoaLnfqyr7KykpoTFjxpClpSUNGTKEwsPDKT4+nqRSqcK6W7dupeHDh9OiRYto8eLF9PHHH9PDhw/l1ikvLycLCwu6ceNGvcf4vLq6hHNSYqwZqfvzExAQQAYGBmrbv6oampSIar+Ux40b10IRtYzTp09TZGRkm91faGgorVu3rlHv5XFKjDGt1q9fP0yePFlhVgFNVVJSgoMHD8rN1deW9nf48GFIpVKEhIQ063Y5KTHWhpSWlkIqlbZISQFN4OvrCwcHBxw4cEDdodQrKysLq1evlus+3Vb2d/XqVRQXFyMyMrLZt93kpFRdXY0zZ85gxYoVOHLkSHPE1KLqq1lSH22tc3Ps2DEEBARAJBJBJBJh+PDh2LVrl7rDwr59+zBo0CAhrvnz5wsPc1nDxMTE4OjRo3j69ClmzpyJtLQ0dYfUIjw9PVXuJadOgwcPbtXSI625PycnJ0ycOLFFtt3kwbMXLlzAjh07sGPHDvTo0UPl96mjVkt9NUtUoa11boYNG4Zhw4bhwIEDePDgAbZv364w429reTZ+Hx8fdOvWDc7Ozujduze+/PJLtcTUFsyePbvVbt0w1lKafKXk7OyMefPmNeg96qrVUl/NElVoe50b2aW9ubl5c4XUIMril01Toq6YGGOao1meKbVr107lddVdq6WumiXaoDnOnUgkkvu3NdUVvzpjYoxplhbr6HDlyhVMnz4dUVFReO+99+Dh4QFAea2Wa9euYfny5XjjjTdw584dhIeHo3v37nBwcMDJkyfx5MkTBAcH47XXXsMrr7yitmdXbbHOjSbE3xD37t2DRCJBeHg4JBIJ3n//fTx8+BAA8OOPP8LU1BQikQgbN25EVVUVAODcuXOwtrbG//zP/wB4cR2cO3fuIDIyEo6Ojnj06BGGDx+O7t27C/tgjLWw5/uIN2acxS+//EIA6Ouvvxba7OzsKC0tjYhqB1i5uroKy56v1XL//n2aOnUqAaCZM2fSxYsX6fHjxzRw4EDq0aMHzZ07l7Kzs6mkpIRcXFyoR48eDYrvWXXVLFFFW6hz8/rrrxMAKi0t1Zj4b9y4QQDI3d293vjd3d3J19dXeO3k5ER+fn7C66VLlxIAunDhgtBWWVlJAwcOFF6/qA7O4cOHqVevXiQWi+nf//43bdu2jQYMGCA3KPJF1D1OSVugEeOUWNvSquOUpFIpbt68KdQeMTQ0xKJFi+pcv2PHjhg0aBAAICgoCH379oWpqSlGjBiBnJwcBAQE4M0334SJiQneffdd5OTkNPukh6poi3VuNCH+hhCJRHBychJeOzo6IisrS3g9d+5c6OrqYuvWrULb0aNHhdIE9dXBGTFiBAYPHoynT5/Cz88PEokE58+fR9euXZv1OBhjyrVI6Qo9PT0MHz4cCxYswC+//ILIyEiMHz/+he+RzZskmwYdqK1fItuejKx2SEFBATp27NjcoderIXVunp2DSpPr3GhC/Ko6ceIEAODJkyfYtWsX/vvf/8qNybGxsYGPjw/i4+MREREBKysr7N27F//+978ByNfBedazdXD09PSgq6urUH66Ifj5WP18fX3h6+ur7jCYGskmCX5Wi9VTSkpKgkQiQVxcHH744Qfs3btXmOxQVco+2LI2dcxW3Ny0vc6NOuJ/+vQp1q5di8zMTHz88ccYOHCg8FxLJjg4GLt378a2bdsQEhKCgoICYbhCa9XBSUhIaNHtaztfX18sWLBArjwCe7nUNTNHiyUlXV1d7Nq1C6NHj8aiRYswYsQIXLlyBW+++WZL7VIraXOdG6D14r958yb+8Y9/4P3330enTp2QlJQEAPj6668V1u3fvz8GDx6MzZs3o1evXhg7dqyw7Nk6OLIrcZkHDx4029X3Bx980Czbaat8fX3h7OzM5+kltm/fPqXtLfJMqbKyEtu2bQNQO+V8RkYGiEgoNVxXrZaXjTrq3MhudVEzTEPTXPHXFwsRITAwEJcvX0ZKSgrc3d2FZXVNqbNo0SLk5eVh0aJF8PHxEdpbsw4OY6zhmiUpPX78GMD/1QMBgO3btwtfTl27doW5uTn69u0rvH6+VotsG8/eCpK1FRQUCG2ywlOVlZWNivVFNUtUoe11bmTn9NnSyuqOX7Z/ZZVFi4uLMW3aNHTo0EF45rVz5078/PPP2L59O65du4Z79+4hKysL9+7dE943btw4vPLKK3BycoKlpaXQrkodHFliravSKWOsBT3fHa+hXVrPnz9PI0eOJADUt29fOnToED158oT69+9Pw4cPp8jISJo5cybFxcUJ73m+Vsvx48fpn//8JwGgKVOm0K1bt+jUqVPUp08fAkAjRoygrKwsSktLo759+xIA8vPzo99//71BXRBVqVnyItpc5+bkyZM0Z84cAkAAaOTIkbRnzx61x5+cnEyurq5CXE5OTuTp6UkeHh7Uq1cvateuHQGgrVu3EhFRYGAgmZqa0qBBg+jYsWP0//7f/yMrKyvy9vYWurnLzJo1SzhHz3pRHZxt27ZRx44dCQBNnTqVLl26pPL/DyLuEq4qcJfwl15dXcJFRPL3Pvbu3QtfX982O8uwukgkEsTHx6OiokLdoTSKtsVPRBgwYADOnDnTqvMU8udHNSKRCAkJCfxM6SUmu63+/LOlFuvo0FpUeTC9fft2uYfdLbkdphmOHz+Of/3rX2qdOJcx1nBan5SaaxBtSw/GfbbOjTaOYdGG+NPS0jBr1iw4ODjgl19+wenTp9UdEmsBKSkpqKysRFlZGcLDw5GdnY3Bgwfj1KlTcmPtHj16hLVr12Lz5s2orq7G7NmzsXz5clhZWakx+roVFRXh888/x9OnTxEREaGwPDMzExEREbC1tcXjx48xePBg+Pv7N2pf+/btE579vvbaa1i9erXCxAD79u1DSkoKrKys8Oeff6Jnz5749NNPoaenh8uXLyM1NRXz589v/u+D5+/n8T3x5rdlyxaytLQkABQQEEBnzpxRd0gNoi3xZ2dnU48ePei1116j06dPqyUGdX9+GvKMVJ3bRiOfKW3ZsoW2bNkivH7w4AHp6uoSAAoODlb6ngULFtCsWbMaHWtrOHDgAH3wwQd1ToF25coVMjY2pvT0dCKqnbrNzs6OYmJiGryvHTt2UFBQEF25coVOnDhBffr0IT09Pfrtt9+EdRISEujtt9+m6upqIiKqqamhkSNH0uLFi4V1Tpw4QZ988kmD9y9T1zMlTkqMNSN1fn4ePXpE//rXv7Ri241JSj/99JPSLzEDAwOho4xsTspnbdiwgdatW9foWFtLcXFxnUnp3XffVTj/mzdvJhMTE3r8+LHK+6iqqqIVK1bItV28eJEA0P/+7//K7e/5ODZt2kRvvPGGXNvq1aspOjpa5f0/q1XnvmOMta6WLAmj7nIzQO1QkBkzZiAsLExhmZ2dnXDr6aOPPhJmfJcxNDQUppDSZHWV1cnPz8fx48fh5uYm1/7OO++gtLQU8fHxKu9DR0cHq1atkmuTDZmQzYEJ1J7vY8eOQSqVCm1ZWVkKhUEXLlyI1atXN+v/DU5KjGmApKQkBAUFISQkBCNHjkRoaKgwFq+xZUE0uWRKQ8XFxUFfXx/29vYKy3R0dBAfHw9HR0c8fvwYXl5e9fYSfdH5VqWcC/DiEijNKTs7GwAU5mLs2bMngNr5HFUlFosV5u/8/vvv8dVXX+GNN94Q2iQSCW7cuIFRo0ahuLgYGRkZOH/+vMLUQMbGxujXrx8+++yzBh3TCz1/6cS37xhrvMZ8fjZs2EAuLi5UVVVFREQFBQXUs2dPcnNzE8aBNaYsiCaVTHkeGnj7ztnZmXx8fJQu6927NxER5eTkCM8+/f39heWxsbFyt5jqO9+qlHMhenEJlMaoq6xOdHQ0AaCffvpJ4T36+vrk5ubWqP2VlJRQWFgYde7cmY4cOaKwPCwsjACQnZ0djRkzhgoLC5VuJzw8nMzNzYXnT6ri23eMaaD79+8jNDQUgYGBwgzslpaWWL58OVJTU7Fr1y4AjSsLosklUxqipqYGmZmZcjNzKGNra4vExETo6elh586dSifdVeV8q1LOpb4SKM3pzp07AAATExOFZSYmJnIzmaiqrKwMYWFhOH/+PAoLCzF8+HBs375dbp2VK1fCxcUFN2/exKlTp3Dp0iWl2+rcuTOKi4uFK7qm4qTEmBplZGSgrKxMKMkiI6v/JJsvsrE0uWSKqgoLCyGVStGhQ4d613V3d8emTZsAAPPmzVP4IlX1fNdVzkU2zdmzJVAkEgkkEgny8vLkSqA0F9mt1een/ZK1PX8sqjA2Nsa6detw6NAhXLx4ERYWFnK34KqrqzF9+nRMmzYNR44cgb6+PkaNGoUff/xRYVvt27cHgEYlR2W0fpwSY9rsr7/+AlA7puZZVlZWMDIyQl5eXrPvU9tKpsgShKrzVQYGBiIrKwsxMTHw8fGBRCKBqakpgOY7361VAgX4v2dJz85XCQBVVVWoqKiQexbUGI6Ojpg/fz5WrVoFqVQKPT09LFiwALdv38aOHTsAAKdPn4aHhwf8/f1x+/ZtmJmZCe+X1cBrrnJCfKXEmBrZ2toCQJ29l1qqLIg2lUwxNzeHgYFBnRPkkpIpnTZt2gR3d3fk5OTIXQE01/l+tgTK85p7IL6joyPEYjH+/PNPufY//vgDQPP8H3F0dISNjY1wRb1nzx65Wlf29vaIiIhAcXGxMLmyjCzBd+nSpclxAJyUGFMrZ2dnmJmZITk5Wa49NzcX5eXlGDduHICmlTV5njpKpjSFSCSCi4uL0qsYIlJ6W0tXVxeJiYmwtbWVi1vV812f1iyBYm1tDV9fX6Smpsq1p6amol27dvDy8mryPm7cuCF37FZWVsKtSpl+/foBADp16iTXXlBQADMzMzg4ODQ5DoCTEmNqZWlpiaioKJw9exbHjx8X2jdt2gR/f3+hWnNjy4IAmlEypakmT56M9PR0haui3Nxc3L17V248jYylpSUOHDgg10FA1fNdXzkXVUqgzJkzB66urrh165ZKx/iisjrLli1DWloarly5AqD21l10dDRCQ0PRuXNnlfdXVFSE6dOnY//+/cK5vHXrFlJTUxEVFSWsN3PmTOzevVvuqi8lJQXvvPOOwu3C9PR0eHl5Nd/zxee743GXcMYar7Gfn+TkZPL09KSgoCD69NNP6YsvvmiWsibqLplSFzSwS3hVVRX17NlTmGaHiGj//v3k5uZGAMjb27vO6a+Sk5Np8+bNCm11nW9Vy7m8qAQKEdHo0aNJR0eHlixZUu/xqVJW58KFC+Tr60vLli2jSZMmUXR0tNzvUZX9lZSU0JgxY8jS0pKGDBlC4eHhFB8fT1KpVGHdrVu30vDhw2nRokW0ePFi+vjjj+nhw4dy65SXl5OFhQXduHGj3mN8Hk8zxFgr0LTPT0BAABkYGKg7DAUNTUpEtV/K48aNa6GIWsbp06cpMjKyze4vNDS00VM48TglxphW69evHyZPnqwwq4CmKikpwcGDBzF79uw2ub/Dhw9DKpUiJCSkWbfLSYmxNuzZkiNtga+vLxwcHHDgwAF1h1KvrKwsrF69Wq77dFvZ39WrV1FcXIzIyMhm3zaPU2KsjYqJicHRo0fx9OlTzJw5E/7+/nB1dVV3WE3m6emp7hBUMnjw4Da7PycnJzg5ObXItjkpMdZGzZ49u9Vu5TDWXPj2HWOMMY3BSYkxxpjG4KTEGGNMY3BSYowxpjHq7Oiwd+/e1oyDsTbh3LlzAPjzowrZuWIvp9zcXNjY2CgueH40rWxEOv/wD//wD//wT0v+KJvRQURtZVQdY2okEomQkJCADz74QN2hMKbV+JkSY4wxjcFJiTHGmMbgpMQYY0xjcFJijDGmMTgpMcYY0xiclBhjjGkMTkqMMcY0BiclxhhjGoOTEmOMMY3BSYkxxpjG4KTEGGNMY3BSYowxpjE4KTHGGNMYnJQYY4xpDE5KjDHGNAYnJcYYYxqDkxJjjDGNwUmJMcaYxuCkxBhjTGNwUmKMMaYxOCkxxhjTGJyUGGOMaQxOSowxxjQGJyXGGGMag5MSY4wxjcFJiTHGmMbgpMQYY0xjcFJijDGmMTgpMcYY0xiclBhjjGkMTkqMMcY0BiclxhhjGoOTEmOMMY0hIiJSdxCMaZNZs2bh119/lWu7dOkSbG1t0aFDB6FNLBZj586dsLGxae0QGdNauuoOgDFt07lzZ2zbtk2hPSsrS+51jx49OCEx1kB8+46xBpo8eXK967Rr1w7Tpk1r+WAYa2P49h1jjeDo6Ijs7Gy86OPz66+/ws7OrhWjYkz78ZUSY43w4YcfQiwWK10mEong5OTECYmxRuCkxFgjTJo0CU+fPlW6TCwWw9/fv5UjYqxt4Nt3jDWSi4sLzp8/j5qaGrl2kUiEv//+G//4xz/UFBlj2ouvlBhrpKlTp0IkEsm16ejowNXVlRMSY43ESYmxRvLx8VFoE4lE+PDDD9UQDWNtAyclxhrJysoK7777rlyHB5FIhPfff1+NUTGm3TgpMdYEfn5+QrdwsViM4cOHw9LSUs1RMaa9OCkx1gQTJkxAu3btAABEBD8/PzVHxJh246TEWBMYGxtjzJgxAGpncRg7dqyaI2JMu3FSYqyJpkyZAgB4//33YWxsrOZoGNNuWjVOycfHB4mJieoOgzHGtIoWfc1r3yzhgwYNQnBwsLrDYGrm6+uLBQsWwNnZWd2hAADi4+MxceJE6Opqzkdqw4YNAMCfl5fYuXPnsHHjRnWH0SBad6UEAPv27VNzJEzdRCIREhIS8MEHH6g7FADAkydPYGBgoO4w5PDnhcHdykwAACAASURBVO3duxe+vr5adaXEz5QYawaalpAY01aclBhjjGkMTkqMMcY0BiclxhhjGoOTEmOMMY3BSYm9tJKTk9GtWzdcv35d3aFopJSUFBw8eBB79uyBg4MDRCIRXF1dUV1dLbfeo0ePsHTpUpiamsLQ0BALFy5EQUGBmqKuX1FREUJDQ7Fs2TKlyzMzM+Hl5YWQkBDMnDkTO3fubPS+9u3bh759+8LExAROTk748ccfla4jkUiwbNkyTJo0CStXroRUKgUAXL58GRs3btSq3nNNpTmDKhhrZcbGxujUqZNae87l5+fD2tpabfuvS0xMDABg9uzZAIBhw4bB2toaZ8+exeLFi7F+/XphXQsLC0RGRqKyshIVFRVyyzTNwYMHER8fj7179yIoKEhh+dWrV+Hu7o6jR4/C2dkZFRUV6N27NyoqKhAYGNigfX377be4ePEiduzYgUePHmHRokXw8fHBtWvX0LNnTwC1XbbXrl2L8+fPQywWg4gwevRohIaGIioqCn369EFRURGWLFmCtWvXNss50HR8pcReWh4eHrh48SJsbW3Vsv/CwkKNnMD10KFDOHHihJCQgNoyHbKBwRs2bEBSUpLC+7p3747XX3+91eJsjLFjxyIuLq7O5YsWLcLAgQOFQdmGhoaYP38+PvnkE5SUlKi8H6lUilu3buGrr76Ck5MThg4diq+//hpSqRTnz58X1tu2bRucnZ2F8icikQgjR46Uu6IaOnQoTE1NsXnz5oYerlbipMSYGpSXl2PixInIyclRdyhySkpKMGPGDISFhSkss7Ozw3vvvQcA+Oijj3Dz5k255YaGhjA0NGyVOJtCX19faXt+fj6OHz8ONzc3ufZ33nkHpaWliI+PV3kfOjo6WLVqlVybrKRJ//79hbaSkhIcO3ZMuF0HAFlZWQqVixcuXIjVq1dr3P+XlsBJib2UCgsL8c0338DDwwPJyckAgCtXruCTTz5Bjx49UFZWhoCAAFhZWWHAgAHCl0F2djZWrFgBe3t75OXlYfz48bCwsMCAAQOQkZEBANi9ezfMzMzQrVs3AEBxcTHCw8MhFouFv8B/+OEHXL9+HQUFBZBIJPj8888BAGfPnkW3bt1w+PDh1j4lAIC4uDjo6+vD3t5eYZmOjg7i4+Ph6OiIx48fw8vLCxUVFS/cXlJSEoKCghASEoKRI0ciNDQUlZWVAFQ730DtvG2xsbGYPXs2Bg4cCE9PT4WE2Byys7MBQOFqT3arLT09XeVticVihSmnvv/+e3z11Vd44403hDaJRIIbN25g1KhRKC4uRkZGBs6fPy9MESVjbGyMfv364bPPPmvQMWkl0iLe3t7k7e2t7jCYBgBACQkJjX5/dnY2BQcHEwBKTEwkIqL8/HwaNmwYAaC5c+fStWvX6PLly6Svr08TJ04kIqKlS5dS+/btSSwWU3BwMJ08eZKSkpLIysqKjIyMKC8vj4iIPD09ycbGRm6fb731Fg0aNEh4PWbMGHr11Vfl1jl06BAZGhrSrl27Gn1sMo35vDg7O5OPj4/SZb179yYiopycHLK0tCQA5O/vLyyPjY2l6Oho4fWGDRvIxcWFqqqqiIiooKCAevbsSW5ublRTU6PS+SYiioiIoG+//ZaIiKqrq8ne3p66dOlCZWVlDTo2mSdPnhAACgoKkmuPjo4mAPTTTz8pvEdfX5/c3Nwatb+SkhIKCwujzp0705EjRxSWh4WFEQCys7OjMWPGUGFhodLthIeHk7m5OVVXV6u874SEBNKyr3niKyX2UnrzzTeFW1EyXbp0EW6thIWFwd7eHr1790b//v1x8eJFAEBERARGjRoFHR0dREVFwd3dHRMmTEBMTAzKy8sRGxsLADAyMlLYpyplLUaNGoWSkhJMnjy5qYfYYDU1NcjMzKy3cq6trS0SExOhp6eHnTt3Kn1Gc//+fYSGhiIwMBB6enoAam9fLV++HKmpqdi1a5dK5zsvLw8bN27E1KlTAdRegXh7e+Pu3bs4ePBgcx4+7ty5AwAwMTFRWGZiYoJ79+41eJtlZWUICwvD+fPnUVhYiOHDh2P79u1y66xcuRIuLi64efMmTp06hUuXLindVufOnVFcXCxc0bVVnJTYS0vZjN6yB87PLrOxsZF7yG1kZASxWCx82QLA+PHjoa+vj59//rnJccliaG2FhYWQSqXo0KFDveu6u7tj06ZNAIB58+YpfJFmZGSgrKwMr7zyily7rCDiyZMnAdR/vtPT0yGVSjFr1ixIJBJIJBLk5eUhICCg2Z9fyW63lpeXKywrLy9XOBZVGBsbY926dTh06BAuXrwICwsLuVtw1dXVmD59OqZNm4YjR45AX18fo0aNUtp1vH379gDQqOSoTbhLOGPNQFdXF127dlUYw6NNZAni6dOnKq0fGBiIrKwsxMTEwMfHBxKJBKampgCAv/76C0DtGKZnWVlZwcjICHl5eSrt4/r16zA2Nn5hj7nmInuWVFxcLNdeVVWFiooKuWdBjeHo6Ij58+dj1apVkEql0NPTw4IFC3D79m3s2LEDAHD69Gl4eHjA398ft2/fhpmZmfB+HZ3aa4iampomxaHp+EqJsWZSXl6OXr16qTuMRjM3N4eBgQGKioqULiclAzg3bdoEd3d35OTkyF0ByLrZ19VbTNXzZGRkhNzcXOTm5iose/DggUrbUJWjoyPEYjH+/PNPufY//vgDgOox17cPGxsb4Sp7z549cjXB7O3tERERgeLiYly+fFnuvbIE36VLlybHock4KTHWDPLz8/HgwQN4e3sDqL1yKi0tlbvqKC0tlfsrV0dHB6WlpQrbUtdfwiKRCC4uLkqvYohI6W0tXV1dJCYmwtbWVu5YnJ2dYWZmJvRslMnNzUV5eTnGjRunUkxvvfUWiAhLliyRa//999+xZcsWlbahKmtra/j6+iI1NVWuPTU1Fe3atYOXl1eT93Hjxg25Y7eyslIY/9SvXz8AQKdOneTaCwoKYGZmBgcHhybHock4KbGXVn5+PgD5v7hlt26evQ13//59hS/kyspKXL16VXi9Zs0a+Pv7Y8CAAQBqv0yLiooQERGB3377DWvWrEFlZSV+/fVX4S/grl27oqCgABcvXsSpU6dQXl6OY8eOoUOHDkhMTGyZg67H5MmTkZ6ernBVlJubi7t378qNp5GxtLTEgQMH5DoIWFpaIioqCmfPnsXx48eF9k2bNsHf3x9Dhw4FUP/59vDwQP/+/fH999/Dy8sL8fHx2LJlC2bNmoW5c+cCAObMmQNXV1fcunVLpWMsKysDoPw25bJly5CWloYrV64AqL11Fx0djdDQUHTu3Fnl/RUVFWH69OnYv3+/cC5v3bqF1NRUREVFCevNnDkTu3fvlvs/mJKSgnfeeUfhdmF6ejq8vLzU9syx1ai1718DcZdwJoMmdgk/fvw4DRkyhABQv379KCUlhY4dO0avvvoqAaA5c+bQ/fv36bvvviMTExMCQKtWraLq6moKCAigdu3aUXBwMPn4+NCMGTMoPDycampqhO0XFxfT2LFjycTEhAYNGkQXLlygadOmkZ+fHx04cICIiK5evUo2NjZkZ2dH+/btIyKiEydOkLW1NSUnJzftBFHjPi9VVVXUs2dPSk9PF9r2799Pbm5uBIC8vb3pzJkzSt+bnJxMmzdvVmjz9PSkoKAg+vTTT+mLL74QzpOq5/vhw4c0ZcoU6tSpE3Xs2JE+/PBDunPnjrCP0aNHk46ODi1ZsqTe40tJSSE/Pz8CQD169KCtW7cK3fhlLly4QL6+vrRs2TKaNGkSRUdHy/1uVdlfSUkJjRkzhiwtLWnIkCEUHh5O8fHxJJVKFdbdunUrDR8+nBYtWkSLFy+mjz/+mB4+fCi3Tnl5OVlYWNCNGzfqPcZnaWOXcC6HzrSSOsuhSyQSxMfH1ztwVN0a+3nJzMxEeHi40h5gmurMmTNIT09XuM3XVvb36aefwtzcHCEhIQ16H5dDZ21CVVUV7t+/r+4wmJr069cPkydPVphVQFOVlJTg4MGDcnP1taX9HT58GFKptMEJSVu16aRUXV2NM2fOYMWKFThy5Ii6w6lXfVPq12ffvn0YNGgQRCIRRCIR5s+fL9wbr0taWhreeecd9O3bF/b29vjnP/8JDw8PYcLNwMBAiEQidOrUCU5OTnjjjTcgEolgaWmJt99+G6+//jrEYjEMDQ1x4MABTJgwQdj/L7/88sJ9Ozk5QSQSwcLCAiEhIUofpGui0tJSSKVSrfrrs6F8fX3h4OCAAwcOqDuUemVlZWH16tVy3afbyv6uXr2K4uJiREZGtvi+NIZ67x42TEPvkaenp9P06dMJAH399dcqv+/5e8yt4cCBA/TBBx8onf6kIc6dO0cAhClhXuTnn38mAwMD4XkGEdH3339PxsbGtHLlSiIi8vf3p5UrV9LTp0+JqPY5AADy8/MT3vPLL7+QmZkZ1dTUUEVFBQEgACSRSOrcd1paGonFYgJAISEhDT5ONPGZUmNt2bJFmGInICCgzucrmoCfwTJtfKbUpq+UnJ2dMW/evAa9R13lBOqbUl9VslHf5ubm9a777bffgoiEbswAMGnSJMTExAg900QiEZYvXy4M3FPGwcEBEydORGVlJQwMDGBrawtjY2PEx8fj4cOHSt+zZcsWjB8/XuVYNcXs2bNRUFAAIkJcXBxcXV3VHRJjbUqbTkoA0K5dO5XXVXc5gbqm1G8IkUgk9++L3Lt3D5WVlQrjMqZMmSIkocWLF6sU1+LFi4WpYszNzfHhhx+ioqKiznnRfv31V7i7u6scK2Ps5dDmk5IyV65cwfTp0xEVFYX33nsPHh4eAJSXE7h27RqWL1+ON954A3fu3EF4eDi6d+8OBwcHnDx5Ek+ePEFwcDBee+01vPLKKy327KolShrI6saMGzcOu3btEtp1dHSEyqNvvvmmStt67bXX5OYv+/jjjyESibB582aFqXe+/vprzJw5k5MRY0zBS5mUfH19ERAQgCVLlmDPnj148uQJgNorBCcnJ1hZWSEuLg4hISHo1KkTcnNz8dtvv2H16tUYPXo0fvnlF5iamiIgIAAhISGYOXMmrl69im7dumHOnDktEnNxcTEePnyIwsLCZtvm9OnTMX78eDx+/Bh+fn6YOHGiMIivqQmjV69e8PT0RG5uLvbv3y+0P336FAkJCZgyZUqTts8Ya5teuqQklUpx8+ZNYWp8Q0NDLFq0qM71O3bsiEGDBgEAgoKC0LdvX5iammLEiBHIyclBQEAA3nzzTZiYmODdd99FTk5Os8/JBbRMSQOxWIzExESsW7cOxsbGSEhIQK9evRSmhmms+fPnAwC+/PJLoe3QoUMYNmyYSmUcGGMvn5dulnA9PT0MHz4cCxYswC+//ILIyEjhgXtdZNN6PPuw38bGRtiejGxq+4KCAnTs2LG5Q2+R6UXEYjFCQkLg4+ODwMBA/Oc//4GXlxcSEhLkOkA0xogRI2BnZ4f09HRkZmaiX79+iImJQXR0dLPEfu7cuWbZTlslm8R07969ao6EqYs2fkZeuqQE1JZolkgkiIuLww8//IC9e/cKc3GpStntLVmbNk4t3717dxw+fBgff/wxvvrqK8ybNw9eXl5Nuo0nEonw8ccfIygoCF9++SX+/e9/Q1dXF6+99lqzxLxx40Zs3LixWbbVlvn6+qo7BMZU9tLdvgNqZzbetWsXdu3aBV1dXYwYMQLXr19Xd1it5ubNm8jKysL69esVln355ZewsbHB3bt3Va558yL+/v4wNzfH3r17sXLlSgQFBTV5mzIJCQkgIv6p48fb2xve3t5qj4N/1PeTkJDQbJ+31vLSJaXKykps27YNQO2MyBkZGSAioRJmXeUENIEqV2BEL55lgIgQGBgIOzs7fPHFFwrPv0QiEbp27QozMzNYW1vXGcOL9iObhRmoLSM9Y8YMVFVVITMzE56eng3aFmPs5dLmk9Ljx48ByH9Rbt++XZi2vmvXrjA3N0ffvn2F18+XE5Bt49muzbK2goICoU1WF6WysrJRsb5oSn1VSxrISgEoK9RWXFyMadOmoUOHDjAwMICBgQHee+893LlzR1jnzJkzuHTpElatWqV0wKxsu7Ljf96dO3eQl5cndw6CgoKgo6ODoKAguduBsp6EdW2LMfYSIi3S0GlTzp8/TyNHjiQA1LdvXzp06BA9efKE+vfvT8OHD6fIyEiaOXMmxcXFCe95vpzA8ePH6Z///CcBoClTptCtW7fo1KlT1KdPHwJAI0aMoKysLEpLS6O+ffsKU/D8/vvvDTq2+qbUV6WkQXJyMrm6ugrT/Dg5OZGnpyd5eHhQr169qF27dgSAtm7dSkRE48aNI09PT3J0dKRx48bRiBEjaMCAARQfH6+w7ZqaGtqyZQs5OjoSANLX16dVq1ZRdna2sE5SUpJQDuL999+n06dPC8v8/PyouLiYiIhKS0tp/fr1ZG1tTQDI0tKSli1bRmVlZSqfL6hpmiFtwtMMMW2cZohLVzCtpM7SFdqCPy9MG0tXvJS971qLKt3Ct2/fjrFjx7ZCNIwxpvk4KbWglhhEyxjTHn/88Qc6derEg8UboM13dGCMta6UlBQcPHgQe/bsgYODA0QiEVxdXRXmQHz06BGWLl0KU1NTGBoaYuHChXIdhzSFp6enUCPs+Z+DBw8K65WUlKB9+/ZyyydMmABjY2NcvnwZGzdu1KrbaOrCV0qMNVB+fr7S7vKavu3WIJvIV1aVddiwYbC2tsbZs2exePFiubFxFhYWiIyMRGVlJSoqKpSOm1O3P/74A3///TfWrFmDLl26CO23b9/GunXr8O677wpt33zzDby8vNCjRw+hTTYEok+fPigqKsKSJUuwdu3a1jsALcRJibEGkNXbOn78uFZtuzUcOnQIJ06ckOtYYWVlBV1dXVRXV2PDhg0YPHgwvLy85N7XvXt3hasoTXH06FH85z//Qffu3eXa169fD09PTxgZGQGoHcbx448/4ujRo3Kz5T9r6NChSEtLw+bNmzF37twWj11b8e07xlTUkvW21F3Lq6lKSkowY8YMhIWFKSyzs7PDe++9BwD46KOPcPPmTbnlhoaGMDQ0bJU4G2rmzJkKCQkA9u/fLzdnZlJSEq5evYoZM2YgPj6+zrF3CxcuxOrVq7X299waOCmxl0ZSUhKCgoIQEhKCkSNHIjQ0VBjku3v3bpiZmaFbt24Aagcah4eHQywWw9nZGYDyelvZ2dlYsWIF7O3tkZeXh/Hjx8PCwgIDBgxARkZGk7YNtEwdrZYQFxcHfX192NvbKyzT0dFBfHw8HB0d8fjxY3h5eaGiouKF23vR7+rKlSv45JNP0KNHD5SVlSEgIABWVlYYMGCA3Jc9ESE2NhazZ8/GwIED4enpqZAQG+Pu3bv473//izFjxghtJ0+eRFlZGb777jtMnToV9vb2SElJUXivsbEx+vXrh88++6zJcbRZah0l1UA8GJDJoIGDZzds2EAuLi5UVVVFREQFBQXUs2dPcnNzo5qaGiIi8vT0JBsbG7n3vfXWWzRo0CDh9ZgxY+jVV18VXi9dupTat29PYrGYgoOD6eTJk5SUlERWVlZkZGQkDIBuzLaJiA4dOkSGhoa0a9culY9VpjU/L87OzuTj46N0We/evYmIKCcnhywtLQkA+fv7C8tjY2MpOjpaeF3f7yo/P5+GDRtGAGju3Ll07do1unz5Munr69PEiROF7URERNC3335LRETV1dVkb29PXbp0adAgbWViY2NpyJAhCu1SqZQyMzNp2rRppKOjQwYGBnKDy2XCw8PJ3NycqqurmxSHKrRx8CxfKbE27/79+wgNDUVgYKBQasTS0hLLly9HamqqUHVX9nzgWfV15Y2IiMCoUaOgo6ODqKgouLu7Y8KECYiJiUF5eTliY2MbvW2gZepoNbeamhpkZmbC0tLyhevZ2toiMTERenp62LlzJ+Li4hTWUeV31aVLF/Tv3x8AEBYWBnt7e/Tu3Rv9+/cX6qTl5eVh48aNmDp1KoDaEi3e3t64e/euXI+5xnj+1p2Mrq4u3n77bezYsQP79u1DZWUlVqxYobBe586dUVxcjOzs7CbF0VZxUmJtXkZGBsrKyoR6VzKy2y+yyXgby8jICGKxWK621vjx46Gvr4+ff/65SdsGWqaOVnMqLCyEVCpFhw4d6l3X3d0dmzZtAgDMmzcPly5dkluu6u9Kdk6e7VRgY2MjzD+Znp4OqVSKWbNmQSKRQCKRIC8vDwEBAU16flVUVISTJ08Kz8jqMmHCBHh7e+PKlSsKy9q3bw8AuHfvXqPjaMu49x1r8/766y8AteNinmVlZQUjI6NmKdHxPF1dXXTt2lVje5U1J1mCUDaRsDKBgYHIyspCTEwMfHx8IJFIYGpqCqD5flfXr1+HsbGx0quxpvjpp5/Qq1cvuW7fdXFzc0NaWppCu2yiY22su9Ya+EqJtXm2trYAUGePp169erXIfsvLy1ts25rE3NwcBgYGSmemB5SXJtm0aRPc3d2Rk5Mj99C/uX5XRkZGyM3NFarvPqspM63UdeuuLsrilSXcZ8c9sf/DSYm1ec7OzjAzM0NycrJce25uLsrLyzFu3DgAtVc3paWlcn/xl5aWyv1Fq2q9rfz8fDx48EAoKd+UbWv6X9QikQguLi5Kr2KICOXl5Qrturq6SExMhK2trdwxq/q7qs9bb70FIsKSJUvk2n///Xds2bJFpW08r7y8HEeOHFE5KaWmpmL69OkK7QUFBTAzM4ODg0Oj4mjrOCmxNs/S0hJRUVE4e/as3MDUTZs2wd/fH0OHDgVQ+0VWVFSEiIgI/Pbbb1izZg0qKyvx66+/4vLlywCU19sCamtoXb16Vdj2mjVr4O/vjwEDBjRp26rW0VK3yZMnIz09XeGqKDc3F3fv3oVUKlV4j6WlJQ4cOAATExO5NlV+V7K6Yc/eHr1//77w+/Dw8ED//v3x/fffw8vLC/Hx8diyZQtmzZolDFydM2cOXF1dcevWLZWO8ciRI7C0tBRqr8mcOXMGgwYNwjfffCN0W09OToahoaHQ0eJZ6enp8PLy0vhnhWqj3s5/DcNdwpkMGlFPKTk5mTw9PSkoKIg+/fRT+uKLL4Tu4ERExcXFNHbsWDIxMaFBgwbRhQsXaNq0aeTn50cHDhwgIsV6W0REAQEB1K5dOwoODiYfHx+aMWMGhYeHN8u2VamjVZfW/LxUVVVRz549KT09XWjbv38/ubm5EQDy9vamM2fOKH1vcnIybd68WaGtrt/VsWPH6NVXXyUANGfOHLp//z599913ZGJiQgBo1apVVF1dTQ8fPqQpU6ZQp06dqGPHjvThhx/SnTt3hH2MHj2adHR0aMmSJSodo5+fHwUFBSm0//nnnzRs2DCysLCgvn370ooVK+iHH35Quo3y8nKysLCgGzduqLTPptLGLuFcT4lpJU2qpySRSBAfH1/vgNDW1tqfl8zMTISHh+PHH39slf01hzNnziA9PV3hNl9L+fTTT2Fubo6QkJBW2Z821lPi23eMsWbRr18/TJ48GRs2bFB3KCopKSnBwYMHhcljW9rhw4chlUpbLSFpK05KjDVRaWkppFKpVv012lJ8fX3h4OCAAwcOqDuUemVlZWH16tUwMzNr8X1dvXoVxcXFiIyMbPF9aTsep8RYE8TExODo0aN4+vQpZs6cCX9/f7i6uqo7LLWSlWvQdIMHD261fTk5OcHJyanV9qfNOCkx1gSzZ89utds/jL0M+PYdY4wxjcFJiTHGmMbgpMQYY0xjcFJijDGmMbSuo0NGRoYwKJC93DZs2MADqV9AVvmWPy8vL2UT0mo6rZrRYf369Th37py6w2BMweHDh9GnTx+e+ZlpJG36402rkhJjmkqTpj1iTJvxMyXGGGMag5MSY4wxjcFJiTHGmMbgpMQYY0xjcFJijDGmMTgpMcYY0xiclBhjjGkMTkqMMcY0BiclxhhjGoOTEmOMMY3BSYkxxpjG4KTEGGNMY3BSYowxpjE4KTHGGNMYnJQYY4xpDE5KjDHGNAYnJcYYYxqDkxJjjDGNwUmJMcaYxuCkxBhjTGNwUmKMMaYxOCkxxhjTGJyUGGOMaQxOSowxxjQGJyXGGGMag5MSY4wxjcFJiTHGmMbgpMQYY0xjcFJijDGmMTgpMcYY0xiclBhjjGkMTkqMMcY0hq66A2BM2xQVFYGIFNrLyspQWFgo12ZiYgI9Pb3WCo0xrSciZZ8uxlid/vWvf+HkyZP1ricWi3Hnzh107ty5FaJirG3g23eMNdCkSZMgEoleuI6Ojg6GDBnCCYmxBuKkxFgDeXt7Q1f3xXe+RSIRPvzww1aKiLG2g5MSYw3UoUMHeHp6QiwW17mOjo4O3n///VaMirG2gZMSY43g5+eHmpoapct0dXUxevRomJubt3JUjGk/TkqMNcK4ceOgr6+vdNnTp0/h5+fXyhEx1jZwUmKsEYyMjPD+++8r7e5taGiIUaNGqSEqxrQfJyXGGmny5MmQSqVybXp6evD29oahoaGaomJMu3FSYqyRhg8frvDcSCqVYvLkyWqKiDHtx0mJsUbS09PDxIkT0a5dO6Gtffv2ePfdd9UYFWPajZMSY00wadIkVFVVAahNUn5+fvWOYWKM1Y2nGWKsCWpqatC1a1fcu3cPAJCWlobBgwerOSrGtBdfKTHWBDo6Opg6dSoAwNraGi4uLmqOiDHt1mbvM+Tm5iI9PV3dYbCXgJWVFQBg4MCB2Ldvn5qjYS+Dbt26wdnZWd1htIg2e/tu79698PX1VXcYjDHW7Ly9vdvsH0Bt9kpJpo3mXNZEsj9amuv/R2JiIry9vZtlW5pEJBIhISEBH3zwgbpDYf8/Hx8fdYfQoviZEmPNoC0mJMbUgZMSY4wxjcFJiTHGmMbgpMQYY0xjcFJijDGmMTgpMcZYC/vjjz9QVlam7jC0Aiclm+CrMwAAGUlJREFUxpogOTkZ3bp1w/Xr19UdisZJSUnBwYMHsWfPHjg4OEAkEsHV1RXV1dVy6z169AhLly6FqakpDA0NsXDhQhQUFKgp6rp5enpCJBIp/Tl48KCwXklJCdq3by+3fMKECTA2Nsbly5exceNGHqryAm1+nBJjLcnY2BidOnWCgYGB2mLIz8+HtbW12vavTExMDABg9uzZAIBhw4bB2toaZ8+exeLFi7F+/XphXQsLC0RGRqKyshIVFRVyyzTFH3/8gb///htr1qxBly5dhPbbt29j3bp1cjPDf/PNN/Dy8kKPHj2ENk9PTwBAnz59UFRUhCVLlmDt2rWtdwBahJMSY03g4eEBDw8Pte2/sLAQfn5+OH78uNpieN6hQ4dw4sQJuRkHrKysoKuri+rqamzYsAGDBw+Gl5eX3Pu6d++ucBWlKY4ePYr//Oc/6N69u1z7+vXr4enpCSMjIwDA06dP8eOPP+Lo0aN1zhY/dOhQpKWlYfPmzZg7d26Lx65t+PYdY1qqvLwcEydORE5OjrpDEZSUlGDGjBkICwtTWGZnZ4f33nsPAPDRRx/h5s2bcssNDQ01tmLvzJkzFRISAOzfvx/jx48XXiclJeHq1auYMWMG4uPj8fjxY6XbW7hwIVavXq1RvztNwUmJsUYqLCzEN998Aw8PDyQnJwMArly5gk8++QQ9evRAWVkZAgICYGVlhQEDBghfQNnZ2VixYgXs7e2Rl5eH8ePHw8LCAgMGDEBGRgYAYPfu3TAzM0O3bt0AAMXFxQgPD4dYLBYm4vzhhx9w/fp1FBQUQCKR4PPPPwcAnD17Ft26dcPhw4db+5QgLi4O+vr6sLe3V1imo6OD+Ph4ODo64vHjx/Dy8kJFRcULt5eUlISgoCCEhIRg5MiRCA0NRWVlJQDVzjVQO9VYbGwsZs+ejYEDB8LT01MhITbG3bt38d///hdjxowR2k6ePImysjJ89913mDp1Kuzt7ZGSkqLwXmNjY/Tr1w+fffZZk+Noc6iNSkhIoDZ8eKyJmuP/R3Z2NgUHBxMASkxMJCKi/Px8GjZsGAGguXPn0rVr1+jy5cukr69PEydOJCKipUuXUvv27UksFlNwcDCdPHmSkpKSyMrKioyMjCgvL4+IiDw9PcnGxkZun2+99RYNGjRIeD1mzBh69dVX5dY5dOgQGRoa0q5du5p0fEREACghIUHl9Z2dncnHx0fpst69exMRUU5ODllaWhIA8vf3F5bHxsZSdHS08HrDhg3k4uJCVVVVRERUUFBAPXv2JDc3N6qpqVHpXBMRRURE0LfffktERNXV1WRvb09dunShsrIylY9LmdjYWBoyZIhCu1QqpczMTJo2bRrp6OiQgYEBZWdnK6wXHh5O5ubmVF1d3aD9ent7k7e3d6Pj1nR8pcRYI7355pvC7SiZLl26oH///gCAsLAw2Nvbo3fv3ujfvz8uXrwIAIiIiMCoUaOgo6ODqKgouLu7Y8KECYiJiUF5eTliY2MBQHhO8SxjY+N64xo1ahRKSkowefLkph5ig9TU1CAzMxOWlpYvXM/W1haJiYnQ09PDzp07ERcXp7DO/fv3ERoaisDAQOjp6QEALC0tsXz5cqSmpmLXrl0qneu8vDxs3LhRqHklFovh7e2Nu3fvyvWYa4znb93J6Orq4u2338aOHTuwb98+VFZWYsWKFQrrde7cGcXFxcjOzm5SHG0NJyXGmkDZw2yxWKywzMbGBiUlJcJrIyMjiMVi4QsXAMaPHw99fX38/PPPTY5LFkNrKiwshFQqRYcOHepd193dHZs2bQIAzJs3D5cuXZJbnpGRgbKyMrzyyity7bJbZSdPngRQ/7lOT0+HVCrFrFmzIJFIIJFIkJeXh4CAgCY9vyoqKsLJkycV/ih53oQJE+Dt7Y0rV64oLGvfvj0ACFWLWS3ufceYhtDV1UXXrl01tgdafWQJ4unTpyqtHxgYiKysLMTExMDHxwcSiQSmpqYAgL/++gtA7RimZ1lZWcHIyAh5eXkq7eP69eswNjZWejXWFD/99BN69eol1+27Lm5ubkhLS1No19GpvSaoqalp1ti0HV8pMaZBysvL0atXL3WH0Sjm5uYwMDBAUVGR0uWkZMDopk3/X3t3GtTU1cYB/B/A4oK4BBUoLjgVLaioRWVJX2WsqChURcTiEqmAYrWuI1opA4KVaC0WF9y34o6AMI5FcVcKtVTBuhad0UGwJioIBDHC835A7hDWyxIS4vnN+IG7nHvumZiTc+9znhOBkSNH4vHjx0ov/c3NzQGgxug0vm3Utm1bZGVlISsrq8o+qVTKq4zq1PToribV1be8w60474lhnRLDaIycnBxIpVJubSY9PT0UFBQojTwKCgqUflnr6OigoKCgSlnq+PUtEAhgb29f7SiGiCCXy6ts19PTQ3R0NMzNzZXuw87ODoaGhlxUY7msrCzI5XK4urryqtOAAQNARPD391fa/ujRI2zbto1XGZXJ5XIkJiby7pQuX74MLy+vKttlMhkMDQ1hZWXVoHpoK9YpMUwj5OTkAFD+1Z2XlwcASo/hXrx4UeVLubi4GOnp6dzfoaGhEIvFGDZsGICyL9Tc3FysW7cODx8+RGhoKIqLi/HgwQPcvHkTAGBqagqZTIa0tDRcunQJcrkcSUlJ6NSpE6Kjo1Vz07Xw9PREcnJylVFRVlYWnj9/DoVCUeUcoVCI+Ph4GBgYKG2TSCS4fv260sTgiIgIiMViODo6Aqi7rUePHo2hQ4fi8OHDcHNzQ1RUFLZt24a5c+dyE1fnz58PkUiEzMxMXveYmJgIoVCIIUOGKG2/evUqbG1tsWfPHi5sPS4uDm3atOECLSpKTk6Gm5ubWt7/aTLWKTFMA124cAGbN28GUJZa5ty5czh//jwX1RUYGAipVIrffvsNf/75J968eYPg4GBu5FMefTZ16lR4e3vDxMQEe/bs4cpfvHgxXFxcIJFIIBaLMXbsWDg4OMDFxYV7HOXn5wczMzN4enpCJpNxARTt2rVTCqJoLrNmzYJQKOTmWwFl86lmzpzJRQRW936lf//+iIqKgkAg4LbNmzcPsbGxWL9+PRYuXIjAwEAYGxtj3759AMCrrUtLS/H7779j+vTpuHbtGpYuXYrU1FQcPHgQRkZGAMpSBf3xxx/YvXs3r3uMiYmpNsChR48eaN++PVasWAF7e3sEBAQAAA4cOFDl2KKiIiQnJ1cZwTHQ3ok8bJ4SUxt1fz68vb2pdevWars+X6jnPCUiohs3bpCrq6uKaqQaV65cobCwsGa7XkBAAG3YsKFB57J5SoxSKC/DMLWzsbGBp6cnwsPD1V0VXvLz85GQkMAlj1W1M2fOQKFQYPny5c1yvZaGdUq12LFjB0aMGIHPP/9c3VVpkNzcXAQEBGDVqlUNOj8mJgaOjo5c+n17e3uIRCIMHjwYtra28Pf3x6NHj5q41h+HgoICKBQKrV3CwMPDA1ZWVoiPj1d3VeqUkZGBNWvWwNDQUOXXSk9PR15eHsLCwlR+rZaKdUq18Pb2RmlpKe95F5okISEBc+fOxdq1a6uNzuJj8uTJiIqKAlCWwTk5ORnXrl3DzZs3sXnzZmRkZKBv375YvXo1m2tRD5GRkTh37hxKSkrg6+tb7TsWbeDk5MQ7Sk6dHBwcmm3pEWtra0ybNq1ZrtVSsU6pFrq6ujAzM1N3NRrExcWlSSYMlqe1qTz7fejQoTh9+jQ8PDzw008/QSKRNPpaHws/Pz/IZDIQEXbt2gWRSKTuKjGMxmCdkhbT19dvdBkVo6Eq09HRwbZt29C1a1eEhobi6dOnjb4ewzAfN9YpVXLq1Cn4+vrC398fCxcu5OahlKNa0uDzTaV/69YteHl5QSKR4Ouvv1ZaJK628ptSUy1v0KFDB0ydOhVyuRzHjx8HoD1txDCMGqgz9E+VGhLye+jQIRo+fDgVFRUREZFUKiUjIyMyNjbmjqktDT7fVPoWFhZ07do1IiKSy+UkEol4lV9fb9++JQC0YMGCKvv4Lm+Qm5tLAKhfv341HhMVFUUAyMvLq8570JQ2UndIeEuBBoSEM6ql7SHhWvu/sr5fOoWFhWRiYkKHDx9W2j5p0iSuU3r27Bl169aNSkpKuP2BgYEEgI4ePUpERKtWrSIAJJPJuGNEIhH16dOHiIjevXtHAoGAfv31V25/bGws7/Lro7ZOiYh4rePCp1NKTEwkADRq1KgW00asU+KHdUqaR9s7JZYl/IOrV68iJycHAwYMUNpe8b1MxTT4FVVMg19TKv3yFCatWrXCmDFjsHjxYvzzzz8ICwvjcmjxKb8pNVV6k/JULxYWFi2ujdzd3et9zscmPDwcJ06cUHc1mA9SUlJga2ur7mqoDOuUPrh//z4A4JNPPqnxmKZKg3/y5En4+Phg165diI2NxfHjx+Ho6KiyNPuqdu/ePQBl4a6sjRiGaQzWKX1Q3hk9efIEFhYW1R5TMQ1+5VBxqVSKLl268LqWnp4eDh06hPHjx2PZsmUYO3Ysbt261WTlNyci4lYRHTt2LKKjo1tUG7ERQO0EAgGWLFmCqVOnqrsqzAfaPrpn0XcfDBw4EABw7Ngxpe0VJ882RRr84uJi7Ny5E0BZRuWUlBQQES5evKiSNPu14TPhlerIOLBx40bcvn0b/v7+6Nmzp9a1EcMwzYuNlD5wcHCAo6Mj9u/fjy+++AJisRh37tzBtWvXIJVKceTIEbi6unJp8N++fYtJkybhzZs3iImJwdGjRwHwW7Zg79698PPzg66uLkxNTdGhQwcMGTIEw4cPr7P8+igsLARQ/UqgSUlJcHNzw549e7j1e6pTng2i8rILT548wcaNG7FlyxYsWrQIwcHBAJSXCmgJbcQwjIZRX4yFajUkuiovL4+8vLyoW7du1KNHDwoKCiJfX1/y8vKipKQkKikpoZcvX9L06dOpa9eu1KVLF5o1axY9e/aMiIiSkpKoV69eBIDmz59PL168oIMHD5KBgQEBoKCgICosLKShQ4fSmDFjKCwsjHx9fWnXrl1cHWorvz7Onj1LM2bMIADUu3dv2rFjB2VnZ3P7L1y4QCYmJhQXF1djGXFxceTo6EgACACJRCIaNWoUOTs707hx42jp0qWUnp5e5byW0EYs+o4fsOg7jaPt0XcCIu3MCHn8+HF4eHhobcJLpnHY54MfgUCAY8eOsXdKGqT8nZK2vg9lj+9aED4v8ffu3QsXF5dmqA3DMEzTY4EOLYhUKq3zH+uQGE1x9uxZJCQk4OjRo7CysoJAIIBIJFJ6lwgAr169wsqVK9G+fXu0adMGS5cuhUwmU1Ota3fixAkMGTIEBgYGsLa2xqlTp6o9xsfHB6tWrcI333yDwMBAbhn4mzdvYtOmTWyEXgs2UmIYNcjJyYGJiUmLK5uvyMhIAOAWzvvqq69gYmKC69evY8WKFfjll1+4Yzt37oywsDAUFxejqKhIaZ8m2b9/P9LS0rBv3z68evUKy5Ytg7u7O+7cuYM+ffoAKHssvH79eqSmpkJXVxdEhPHjxyMgIAASiQSDBw9Gbm4u/P39sX79ejXfkWZiIyWGaWavX7/GjBkzWlzZfJ0+fRoXLlxQWsnVyMiIy+ARHh6OkydPVjmvZ8+e+Oyzz5qtnvWhUCiQmZmJzZs3w9raGo6Ojti9ezcUCgVSU1O543bu3Ak7Ozsua4lAIMC4ceOURlSOjo5o3749tm7d2uz30RKwkRLDNCO5XI5p06YpZURvCWXzlZ+fjzlz5uDChQtV9llYWMDc3BynTp3Ct99+i4EDB3IjDKBsza7Kj/Y0hY6ODoKCgpS2CYVCAGVri5XLz89HUlISFAoFWrVqBaBsZdtPP/1U6dylS5eid+/eGDduHHr37q3ayrcwbKTEMPVw8uRJLFiwAMuXL8e4ceMQEBCA4uJiAMCRI0dgaGiI7t27AyibjxUSEgJdXV3Y2dkBAGJjY3Hv3j3IZDL4+Pjg559/xt27d7F69WpYWloiOzsbEydOROfOnTFs2DCkpKQ0qmyg6ZYp4WPXrl3Q19eHpaVllX06OjqIiopC//798ebNG7i5uaGoqKjW8mprb77LoFATLHWiq6urlKsRAA4fPozNmzejb9++3DYfHx/cv38fzs7OyMvLQ0pKClJTUxEeHq50brt27WBjY4O1a9fWqx4fBTWGo6sUm4fC1KYhn4/w8HCyt7end+/eERGRTCajPn360IgRI6i0tJSIiJycnMjMzEzpvAEDBpCtrS3394QJE6hXr17c3ytXrqSOHTuSrq4uLVmyhC5evEgnT54kIyMjatu2LTe/rCFlE/FfpqQ6qOc8JTs7O3J3d69236BBg4iI6PHjxyQUCgkAicVibv/27dtpy5Yt3N91tTffZVCacjkYIqL8/HwKDg6mbt26UWJiYpX9wcHBBIAsLCxowoQJ9Pr162rLCQkJoQ4dOvDK1l+Rts9TYiMlhuHhxYsXCAgIwLx587jHMkKhED/88AMuX76MQ4cOASjLj1hZ+ZLyNVm3bh2cnZ2ho6MDiUSCkSNHYvLkyYiMjIRcLsf27dsbXDYAODs7Iz8/H56ennUe2xilpaX466+/uMdaNTE3N+fyJR44cKDa5Lp82tvY2Jh7dBYcHAxLS0sMGjQIQ4cORVpaGgAgOzsbmzZtwsyZMwGUjXimTJmC58+fIyEhod73WFhYiODgYKSmpuL169cYM2YM9u7dq3RMYGAg7O3t8e+//+LSpUv4+++/qy2rW7duyMvLw927d+tdD23GOiWG4SElJQWFhYXo0aOH0vYJEyYAAC5evNio8tu2bQtdXV3uCxgAJk6cCH19fdy+fbtRZQNNt0xJbV6/fg2FQoFOnTrVeezIkSMREREBAFi4cGGVL26+7V3TMij5+fkAlJc68fHxgY+PD7Kzsxu81Em7du2wYcMGnD59GmlpaejcubPSI7j379/Dy8sLs2fPRmJiIvT19eHs7Fxt6HjHjh0BAP/991+966HNWKADw/Dw5MkTAGVzaioyMjJC27ZtkZ2d3eTX1NPTg6mpqca+/K+svIOoLtdidebNm4eMjAxERkbC3d0dPj4+aN++PYCma29VLnXSv39/LFq0CEFBQVxgw+LFi/H06VPs27cPAHDlyhWMHj0aYrEYT58+haGhIXe+jk7ZmIBPYuSPCRspMQwP5ubmAFBjZFu/fv1Ucl25XK6ysptahw4d0Lp1a+Tm5la7n6qZMBoREYGRI0fi8ePHSiOOpmrvikudVCaVSnmVUZv+/fvDzMyMG+EePXqUCzwBAEtLS6xbtw55eXm4efOm0rnlHa6xsXGj66FNWKfEMDzY2dnB0NAQcXFxStuzsrIgl8vh6uoKoGx0U1BQoDRaKCgoUPo1rKOjw2Vfr01OTg6kUimXxb0xZTfHr3GBQAB7e/tqRzFEVCXTPFB2T9HR0TA3N1eqN9/2rouqlzq5f/++Ul2MjIy4R4flbGxsAABdu3ZV2i6TyWBoaAgrK6tG10ObsE6JYXgQCoWQSCS4fv06zp8/z22PiIiAWCyGo6MjgLIvwdzcXKxbtw4PHz5EaGgoiouL8eDBA+6XsqmpKWQyGdLS0nDp0iXuy7q4uBjp6elc2aGhoRCLxRg2bFijyk5KSkKnTp0QHR2t8nby9PREcnJylVFRVlYWnj9/zqXbqUgoFCI+Ph4GBgZK2/i0d13LoFRcSsXNzQ1RUVHYtm0b5s6di++++w4AMH/+fIhEImRmZtZ4X7m5ufDy8kJMTAx3b5mZmbh8+TIkEgl3nK+vL44cOaI0Cjt79iy+/PJLpdBxoOx9l5ubW7O872tR1Br7p0IsJJypTUM/H3FxceTk5EQLFiygH3/8kTZu3MiFgxOVLX/i4uJCBgYGZGtrSzdu3KDZs2fTjBkzKD4+noiI0tPTyczMjCwsLOjEiRNEROTt7U2ffPIJLVmyhNzd3WnOnDkUEhLSJGXzWaakJqhnSPi7d++oT58+lJyczG2LiYmhESNGEACaMmUKXb16tdpz4+LiaOvWrVW21dTefJZBef/+fZ1LnYwfP550dHTI39+/xvvKz8+nCRMmkFAopP/9738UEhJCUVFRpFAoqhy7Y8cOGjNmDC1btoxWrFhB33//Pb18+VLpGLlcTp07d6b79+/X3aiVaHtIuNZ+a7NOiamNpn0+vL29qXXr1uquRhX17ZSIiG7cuEGurq4qqpFqXLlyhcLCwprtegEBAbRhw4YGnavtnRJ7fMcwTJOysbGBp6dnlSwGmio/Px8JCQlKufpU6cyZM1AoFFi+fHmzXK+lYZ0Sw2iAgoICKBQKrVnSwMPDA1ZWVoiPj1d3VeqUkZGBNWvWKIVrq0p6ejry8vIQFham8mu1VGyeEsOoWWRkJM6dO4eSkhL4+vpCLBZDJBKpu1qN5uTkpO4q8OLg4NBs17K2toa1tXWzXa8lYp0Sw6iZn59fsz06YhhNxx7fMQzDMBqDdUoMwzCMxmCdEsMwDKMxWKfEMAzDaAzWKTEMwzAaQ+uj7wQCgbqrwGgw9vmom4eHBzw8PNRdDaaC8iS92khA2jJbr5KsrCwkJyeruxoMwzBNrnv37kpLZGgTre2UGIZhmJaHvVNiGIZhNAbrlBiGYRiNwTolhmEYRmPoATih7kowDMMwDAD8HwtMI3uP3QunAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# plot the model\n",
    "from keras.utils import plot_model\n",
    "plot_model(model, show_shapes=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      " - 0s - loss: 3.6075 - accuracy: 0.1253\n",
      "Epoch 2/100\n",
      " - 0s - loss: 3.4849 - accuracy: 0.1905\n",
      "Epoch 3/100\n",
      " - 0s - loss: 3.1972 - accuracy: 0.1905\n",
      "Epoch 4/100\n",
      " - 0s - loss: 3.0506 - accuracy: 0.1905\n",
      "Epoch 5/100\n",
      " - 0s - loss: 3.0195 - accuracy: 0.1905\n",
      "Epoch 6/100\n",
      " - 0s - loss: 2.9907 - accuracy: 0.1905\n",
      "Epoch 7/100\n",
      " - 0s - loss: 2.9780 - accuracy: 0.1905\n",
      "Epoch 8/100\n",
      " - 0s - loss: 2.9610 - accuracy: 0.1905\n",
      "Epoch 9/100\n",
      " - 0s - loss: 2.9482 - accuracy: 0.1905\n",
      "Epoch 10/100\n",
      " - 0s - loss: 2.9377 - accuracy: 0.1905\n",
      "Epoch 11/100\n",
      " - 0s - loss: 2.9124 - accuracy: 0.1905\n",
      "Epoch 12/100\n",
      " - 0s - loss: 2.8937 - accuracy: 0.1905\n",
      "Epoch 13/100\n",
      " - 0s - loss: 2.8770 - accuracy: 0.1955\n",
      "Epoch 14/100\n",
      " - 0s - loss: 2.8499 - accuracy: 0.1905\n",
      "Epoch 15/100\n",
      " - 0s - loss: 2.8234 - accuracy: 0.1905\n",
      "Epoch 16/100\n",
      " - 0s - loss: 2.7882 - accuracy: 0.2306\n",
      "Epoch 17/100\n",
      " - 0s - loss: 2.7574 - accuracy: 0.1905\n",
      "Epoch 18/100\n",
      " - 0s - loss: 2.7141 - accuracy: 0.2331\n",
      "Epoch 19/100\n",
      " - 0s - loss: 2.6731 - accuracy: 0.2456\n",
      "Epoch 20/100\n",
      " - 0s - loss: 2.6418 - accuracy: 0.2581\n",
      "Epoch 21/100\n",
      " - 0s - loss: 2.5767 - accuracy: 0.2857\n",
      "Epoch 22/100\n",
      " - 0s - loss: 2.5369 - accuracy: 0.2506\n",
      "Epoch 23/100\n",
      " - 0s - loss: 2.4796 - accuracy: 0.3258\n",
      "Epoch 24/100\n",
      " - 0s - loss: 2.4298 - accuracy: 0.2882\n",
      "Epoch 25/100\n",
      " - 0s - loss: 2.4088 - accuracy: 0.3233\n",
      "Epoch 26/100\n",
      " - 0s - loss: 2.3688 - accuracy: 0.3183\n",
      "Epoch 27/100\n",
      " - 0s - loss: 2.2957 - accuracy: 0.3609\n",
      "Epoch 28/100\n",
      " - 0s - loss: 2.2376 - accuracy: 0.3709\n",
      "Epoch 29/100\n",
      " - 0s - loss: 2.1821 - accuracy: 0.3810\n",
      "Epoch 30/100\n",
      " - 0s - loss: 2.1426 - accuracy: 0.3935\n",
      "Epoch 31/100\n",
      " - 0s - loss: 2.1015 - accuracy: 0.3885\n",
      "Epoch 32/100\n",
      " - 0s - loss: 2.0373 - accuracy: 0.3985\n",
      "Epoch 33/100\n",
      " - 0s - loss: 2.0018 - accuracy: 0.4361\n",
      "Epoch 34/100\n",
      " - 0s - loss: 1.9611 - accuracy: 0.4536\n",
      "Epoch 35/100\n",
      " - 0s - loss: 1.9293 - accuracy: 0.4536\n",
      "Epoch 36/100\n",
      " - 0s - loss: 1.8663 - accuracy: 0.4561\n",
      "Epoch 37/100\n",
      " - 0s - loss: 1.8227 - accuracy: 0.5038\n",
      "Epoch 38/100\n",
      " - 0s - loss: 1.7735 - accuracy: 0.5163\n",
      "Epoch 39/100\n",
      " - 0s - loss: 1.7222 - accuracy: 0.5414\n",
      "Epoch 40/100\n",
      " - 0s - loss: 1.6911 - accuracy: 0.5313\n",
      "Epoch 41/100\n",
      " - 0s - loss: 1.6577 - accuracy: 0.5464\n",
      "Epoch 42/100\n",
      " - 0s - loss: 1.6070 - accuracy: 0.5614\n",
      "Epoch 43/100\n",
      " - 0s - loss: 1.5654 - accuracy: 0.5739\n",
      "Epoch 44/100\n",
      " - 0s - loss: 1.5182 - accuracy: 0.6015\n",
      "Epoch 45/100\n",
      " - 0s - loss: 1.4812 - accuracy: 0.6090\n",
      "Epoch 46/100\n",
      " - 0s - loss: 1.4461 - accuracy: 0.6015\n",
      "Epoch 47/100\n",
      " - 0s - loss: 1.3994 - accuracy: 0.6216\n",
      "Epoch 48/100\n",
      " - 0s - loss: 1.3570 - accuracy: 0.6491\n",
      "Epoch 49/100\n",
      " - 0s - loss: 1.3425 - accuracy: 0.6742\n",
      "Epoch 50/100\n",
      " - 0s - loss: 1.2895 - accuracy: 0.6642\n",
      "Epoch 51/100\n",
      " - 0s - loss: 1.2603 - accuracy: 0.6692\n",
      "Epoch 52/100\n",
      " - 0s - loss: 1.2387 - accuracy: 0.6667\n",
      "Epoch 53/100\n",
      " - 0s - loss: 1.2045 - accuracy: 0.6992\n",
      "Epoch 54/100\n",
      " - 0s - loss: 1.1153 - accuracy: 0.7393\n",
      "Epoch 55/100\n",
      " - 0s - loss: 1.0871 - accuracy: 0.7368\n",
      "Epoch 56/100\n",
      " - 0s - loss: 1.0403 - accuracy: 0.7669\n",
      "Epoch 57/100\n",
      " - 0s - loss: 1.0002 - accuracy: 0.7820\n",
      "Epoch 58/100\n",
      " - 0s - loss: 0.9627 - accuracy: 0.8095\n",
      "Epoch 59/100\n",
      " - 0s - loss: 0.9317 - accuracy: 0.8271\n",
      "Epoch 60/100\n",
      " - 0s - loss: 0.9050 - accuracy: 0.8246\n",
      "Epoch 61/100\n",
      " - 0s - loss: 0.8532 - accuracy: 0.8496\n",
      "Epoch 62/100\n",
      " - 0s - loss: 0.8199 - accuracy: 0.8647\n",
      "Epoch 63/100\n",
      " - 0s - loss: 0.7955 - accuracy: 0.8496\n",
      "Epoch 64/100\n",
      " - 0s - loss: 0.7733 - accuracy: 0.8772\n",
      "Epoch 65/100\n",
      " - 0s - loss: 0.7319 - accuracy: 0.9048\n",
      "Epoch 66/100\n",
      " - 0s - loss: 0.7133 - accuracy: 0.9098\n",
      "Epoch 67/100\n",
      " - 0s - loss: 0.6811 - accuracy: 0.9073\n",
      "Epoch 68/100\n",
      " - 0s - loss: 0.6479 - accuracy: 0.9398\n",
      "Epoch 69/100\n",
      " - 0s - loss: 0.6179 - accuracy: 0.9449\n",
      "Epoch 70/100\n",
      " - 0s - loss: 0.5939 - accuracy: 0.9574\n",
      "Epoch 71/100\n",
      " - 0s - loss: 0.5707 - accuracy: 0.9424\n",
      "Epoch 72/100\n",
      " - 0s - loss: 0.5505 - accuracy: 0.9599\n",
      "Epoch 73/100\n",
      " - 0s - loss: 0.5203 - accuracy: 0.9699\n",
      "Epoch 74/100\n",
      " - 0s - loss: 0.4961 - accuracy: 0.9624\n",
      "Epoch 75/100\n",
      " - 0s - loss: 0.4775 - accuracy: 0.9699\n",
      "Epoch 76/100\n",
      " - 0s - loss: 0.4505 - accuracy: 0.9724\n",
      "Epoch 77/100\n",
      " - 0s - loss: 0.4375 - accuracy: 0.9724\n",
      "Epoch 78/100\n",
      " - 0s - loss: 0.4152 - accuracy: 0.9749\n",
      "Epoch 79/100\n",
      " - 0s - loss: 0.4004 - accuracy: 0.9774\n",
      "Epoch 80/100\n",
      " - 0s - loss: 0.3855 - accuracy: 0.9724\n",
      "Epoch 81/100\n",
      " - 0s - loss: 0.3729 - accuracy: 0.9799\n",
      "Epoch 82/100\n",
      " - 0s - loss: 0.3646 - accuracy: 0.9749\n",
      "Epoch 83/100\n",
      " - 0s - loss: 0.3455 - accuracy: 0.9749\n",
      "Epoch 84/100\n",
      " - 0s - loss: 0.3274 - accuracy: 0.9799\n",
      "Epoch 85/100\n",
      " - 0s - loss: 0.3099 - accuracy: 0.9799\n",
      "Epoch 86/100\n",
      " - 0s - loss: 0.2982 - accuracy: 0.9900\n",
      "Epoch 87/100\n",
      " - 0s - loss: 0.2876 - accuracy: 0.9850\n",
      "Epoch 88/100\n",
      " - 0s - loss: 0.2744 - accuracy: 0.9850\n",
      "Epoch 89/100\n",
      " - 0s - loss: 0.2634 - accuracy: 0.9900\n",
      "Epoch 90/100\n",
      " - 0s - loss: 0.2549 - accuracy: 0.9875\n",
      "Epoch 91/100\n",
      " - 0s - loss: 0.2468 - accuracy: 0.9875\n",
      "Epoch 92/100\n",
      " - 0s - loss: 0.2444 - accuracy: 0.9875\n",
      "Epoch 93/100\n",
      " - 0s - loss: 0.2358 - accuracy: 0.9875\n",
      "Epoch 94/100\n",
      " - 0s - loss: 0.2215 - accuracy: 0.9900\n",
      "Epoch 95/100\n",
      " - 0s - loss: 0.2157 - accuracy: 0.9900\n",
      "Epoch 96/100\n",
      " - 0s - loss: 0.2059 - accuracy: 0.9900\n",
      "Epoch 97/100\n",
      " - 0s - loss: 0.2018 - accuracy: 0.9900\n",
      "Epoch 98/100\n",
      " - 0s - loss: 0.1903 - accuracy: 0.9900\n",
      "Epoch 99/100\n",
      " - 0s - loss: 0.1819 - accuracy: 0.9875\n",
      "Epoch 100/100\n",
      " - 0s - loss: 0.1785 - accuracy: 0.9925\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.callbacks.History at 0x7f123c23f780>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit the model\n",
    "model.fit(X, y, epochs=100, verbose=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dump the model\n",
    "from pickle import dump"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the model to file\n",
    "model.save('Models/model_gen_text.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the mapping\n",
    "dump(mapping, open('Weights/model_gen_text_mapping.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Generate Text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now use the learned model to generate new sequences of text that have the same statistical properties."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Load the saved model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import load_model\n",
    "from pickle import load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Both Model and Mapping loaded...\n"
     ]
    }
   ],
   "source": [
    "# Load the model\n",
    "model = load_model('Models/model_gen_text.h5')\n",
    "# Load the mapping\n",
    "mapping = load(open('Weights/model_gen_text_mapping.pkl', 'rb'))\n",
    "print('Both Model and Mapping loaded...')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'\\n': 0, ' ': 1, \"'\": 2, ',': 3, '.': 4, ';': 5, 'A': 6, 'B': 7, 'C': 8, 'E': 9, 'F': 10, 'H': 11, 'S': 12, 'T': 13, 'W': 14, 'a': 15, 'b': 16, 'c': 17, 'd': 18, 'e': 19, 'f': 20, 'g': 21, 'h': 22, 'i': 23, 'k': 24, 'l': 25, 'm': 26, 'n': 27, 'o': 28, 'p': 29, 'q': 30, 'r': 31, 's': 32, 't': 33, 'u': 34, 'w': 35, 'x': 36, 'y': 37}\n"
     ]
    }
   ],
   "source": [
    "print(mapping)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now our model is loaded, let's Generate characters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We must provide sequences of 10 characters as input to the model in order to start the generation process. We will pick these manually. A given sequence will need to be prepared in the same way as preparing the training data for the model. First, the sequence of characters must be integer encoded using the loaded mapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_text = 'Sing a son'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12, 23, 27, 21, 1, 15, 1, 32, 28, 27]\n"
     ]
    }
   ],
   "source": [
    "# Encode the characters as integers\n",
    "encoded = [mapping[char] for char in in_text]\n",
    "print(encoded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next one-hot encoded using the to_categorical() keras function. Reshape the sequence to be 3 dimensional.\n",
    "\n",
    "Remember: We only have one sequence and LSTMs require 3 dimentional (samples, time steps, features)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils import to_categorical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# one-hot encode\n",
    "encoded = to_categorical(encoded, num_classes=len(mapping))\n",
    "encoded = encoded.reshape(1, encoded.shape[0], encoded.shape[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict next character in sequence. We use predict classes so we can select the interger for the char with the highest probability. Instead of getting the full probability distribution across the entire set of characters.\n",
    "yhat = model.predict_classes(encoded, verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can then decode this integer by looking up the mapping to see the character to which it maps.\n",
    "out_char = ''\n",
    "for char, index in mapping.items():\n",
    "    if index == yhat:\n",
    "        out_char = char\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This character can then be added to the input sequence. We then need to make sure that the input sequence is 10 characters by truncating the first character from the input sequence text.\n",
    "\n",
    "We will use the pad_sequences() function to perform this truncation.\n",
    "\n",
    "Putting it all together, we can create a function to perform all the above mentioned operations to generate a new sequence of text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_sequence(model, mapping, seq_length, seed_text, n_chars):\n",
    "    in_text = seed_text\n",
    "    # generate a fixed number of characters\n",
    "    for _ in range(n_chars):\n",
    "        # encode the characters as integers\n",
    "        encoded = [mapping[char] for char in in_text]\n",
    "        # truncate the sequences to a fixed length\n",
    "        encoded = pad_sequences([encoded], maxlen=seq_length, truncating='pre')\n",
    "        # one-hot encode\n",
    "        encoded = to_categorical(encoded, num_classes=len(mapping))\n",
    "        # print(encoded.shape)\n",
    "        #encoded = encoded.reshape(1, encoded.shape[0], encoded.shape[1])\n",
    "        # predict character\n",
    "        yhat = model.predict_classes(encoded, verbose=0)\n",
    "        # reverse map integer to character\n",
    "        out_char = ''\n",
    "        for char, index in mapping.items():\n",
    "            if index == yhat:\n",
    "                out_char = char\n",
    "                break\n",
    "        # append to input\n",
    "        in_text += char\n",
    "    return in_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sing a song of sixpence, A poc\n"
     ]
    }
   ],
   "source": [
    "# Load model\n",
    "# Load mapping\n",
    "# test start of rhyme\n",
    "print(generate_sequence(model, mapping, 10, 'Sing a son', 20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "king was in his counting house\n"
     ]
    }
   ],
   "source": [
    "# test middle of rhyme\n",
    "print(generate_sequence(model, mapping, 10, 'king was i', 20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello worl,, he ainin Thes aun\n"
     ]
    }
   ],
   "source": [
    "# test never before seen \n",
    "print(generate_sequence(model, mapping, 10, 'hello worl', 20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see in the first two examples the model performed as expected. However in the last one, even though total nonsense, we can see the model tried to predict for a new sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}