text-generation-with-rnn-bible copy.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Generation with RNNs\n",
    "\n",
    "By Bobby Cheng\n",
    "\n",
    "Welcome to my Text Generation Using Sequence Models Notebook.\n",
    "\n",
    "## 1. Introduction - Generating Bible Verses\n",
    "\n",
    "This is a text generation project that uses stateful GRUs to train a character-level language model. This is where we feed text input data into the GRU and make it sample the next character of a sequence with previous characters, based on a probability distribution. In the spirit of making this fun and light hearted, I will be using the dataset of the bible to train a caharacter-level language model which can be used to generate fictional bible verses. \n",
    "\n",
    "Along the way, I'll aim to answer the following questions:\n",
    "- How to process your input data for text generation with GRU?\n",
    "- How to use the Sequential API to build a text generator neural network architecture?\n",
    "- How sensible will the generated texts be?\n",
    "- Can we tune the models to be more conserve or diverse?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Loading the Data\n",
    "\n",
    "Our data will be the King James Bible which consists of 66 books. I downloaded the bible dataset from the following github [here](https://raw.githubusercontent.com/mxw/grmr/master/src/finaltests/bible.txt)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Import Libraries\n",
    "\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import os\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of text: 4451368 characters\n"
     ]
    }
   ],
   "source": [
    "path_to_file = tf.keras.utils.get_file('bible.txt', 'https://raw.githubusercontent.com/mxw/grmr/master/src/finaltests/bible.txt')\n",
    "\n",
    "## read the path_to_file\n",
    "text = open(path_to_file, 'rb').read().decode(encoding = 'utf-8') \n",
    "print(f'Length of text: {len(text)} characters')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1:1 In the beginning God created the heaven and the earth.\n",
      "\n",
      "1:2 And the earth was without form, and void; and darkness was upon\n",
      "the face of the deep. And the Spirit of God moved upon the face of the\n",
      "waters.\n",
      "\n",
      "1:3 And God said, Let there be light: and there was light.\n",
      "\n",
      "1:4 And God saw the light, that it was good: and God divided the light\n",
      "from the darkness.\n",
      "\n",
      "1:5 And God called the light Day, and the darkness he called Night.\n",
      "And the evening and the morning were the first day.\n",
      "\n",
      "1:6 An\n"
     ]
    }
   ],
   "source": [
    "## Print the first 500 characters in the text\n",
    "print(text[:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "81 unique characters\n"
     ]
    }
   ],
   "source": [
    "## The number of unique characters in the file\n",
    "vocab = sorted(set(text))\n",
    "print(f'{len(vocab)} unique characters')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Data Preparation\n",
    "\n",
    "### 3.1. Vectorize our text\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.RaggedTensor [[b'T', b'h', b'i', b's', b' ', b'i', b's', b' ', b'a', b' ', b't', b'e',\n",
       "  b's', b't']                                                            ,\n",
       " [b'g', b'o', b't', b'c', b'h', b'a']]>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Since our model is a character level prediction language model,\n",
    "## we'll want to convert our inputs from words into characters.\n",
    "## For that, we can use tf.strings.unicode_split. \n",
    "## The output will be ragged tensor object\n",
    "\n",
    "sample_texts = ['This is a test', 'gotcha']\n",
    "\n",
    "chars = tf.strings.unicode_split(sample_texts, input_encoding = 'UTF-8')\n",
    "chars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.RaggedTensor [[49, 63, 64, 74, 3, 64, 74, 3, 56, 3, 75, 60, 74, 75],\n",
       " [62, 70, 75, 58, 63, 56]]>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Create a StringLookup layer that will convert a string of characters into numerical IDs.\n",
    "ids_from_chars = tf.keras.layers.StringLookup(vocabulary = list(vocab), mask_token = None)\n",
    "\n",
    "## illustrate how ids_from_chars will convert 'chars' into integers\n",
    "ids = ids_from_chars(chars)\n",
    "ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.RaggedTensor [[b'T', b'h', b'i', b's', b' ', b'i', b's', b' ', b'a', b' ', b't', b'e',\n",
       "  b's', b't']                                                            ,\n",
       " [b'g', b'o', b't', b'c', b'h', b'a']]>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Create a StringLookup later that will convert the string of numerical IDs into characters\n",
    "chars_from_ids = tf.keras.layers.StringLookup(vocabulary = ids_from_chars.get_vocabulary(), invert = True, mask_token = None)\n",
    "\n",
    "chars = chars_from_ids(ids)\n",
    "chars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([b'This is a test', b'gotcha'], dtype=object)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## with the above reinstated chars value, we can reconvert it back to a single string\n",
    "tf.strings.reduce_join(chars, axis=-1).numpy()\n",
    "\n",
    "## Note that 'chars' is a tensor object. Hence, it cannot be joined back with ''.join(<list>)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=string, numpy=array([b'This is a test', b'gotcha'], dtype=object)>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## we can create a useful function that would form ragged tensor objects of integers back into fully joined words.\n",
    "def ids_to_text(ids):\n",
    "  return tf.strings.reduce_join(chars_from_ids(ids), axis=-1)\n",
    "\n",
    "ids_to_text(ids)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Create Training Examples and Targets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4451368\n"
     ]
    }
   ],
   "source": [
    "## Let us now convert the entire text document into numerical ids!\n",
    "all_ids = ids_from_chars(tf.strings.unicode_split(text, 'UTF-8'))\n",
    "print(len(all_ids))\n",
    "\n",
    "## we should see the length as equivalent to the length of characters in the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      ":\n",
      "1\n",
      " \n",
      "I\n",
      "n\n",
      " \n",
      "t\n",
      "h\n",
      "e\n",
      " \n",
      "b\n",
      "e\n",
      "g\n",
      "i\n",
      "n\n",
      "n\n",
      "i\n",
      "n\n",
      "g\n",
      " \n",
      "G\n",
      "o\n",
      "d\n",
      " \n",
      "c\n",
      "r\n",
      "e\n",
      "a\n",
      "t\n"
     ]
    }
   ],
   "source": [
    "## We'll now create a dataset tensor object that will aid us in our text generation operations\n",
    "ids_dataset = tf.data.Dataset.from_tensor_slices(all_ids)\n",
    "\n",
    "## The following will illustrate what the dataset tensor object can do.\n",
    "for ids in ids_dataset.take(30):\n",
    "    print(chars_from_ids(ids).numpy().decode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[17 26 17  3 38 69  3 75 63 60  3 57 60 62 64 69 69 64 69 62  3 36 70 59\n",
      "  3 58 73 60 56 75 60 59  3 75 63 60  3 63 60 56 77 60 69  3 56 69 59  3\n",
      " 75 63 60  3 60 56 73 75 63 14  2  1  2  1 17 26 18  3 30 69 59  3 75 63\n",
      " 60  3 60 56 73 75 63  3 78 56 74  3 78 64 75 63 70 76 75  3 61 70 73 68\n",
      " 12  3 56 69 59], shape=(101,), dtype=int64)\n",
      "---------------------------------------\n",
      "Observe the conversion from ids to text:\n",
      "b'1:1 In the beginning God created the heaven and the earth.\\r\\n\\r\\n1:2 And the earth was without form, and'\n",
      "---------------------------------------\n",
      "\n",
      "tf.Tensor(\n",
      "[ 3 77 70 64 59 27  3 56 69 59  3 59 56 73 66 69 60 74 74  3 78 56 74  3\n",
      " 76 71 70 69  2  1 75 63 60  3 61 56 58 60  3 70 61  3 75 63 60  3 59 60\n",
      " 60 71 14  3 30 69 59  3 75 63 60  3 48 71 64 73 64 75  3 70 61  3 36 70\n",
      " 59  3 68 70 77 60 59  3 76 71 70 69  3 75 63 60  3 61 56 58 60  3 70 61\n",
      "  3 75 63 60  2], shape=(101,), dtype=int64)\n",
      "---------------------------------------\n",
      "Observe the conversion from ids to text:\n",
      "b' void; and darkness was upon\\r\\nthe face of the deep. And the Spirit of God moved upon the face of the\\r'\n",
      "---------------------------------------\n",
      "\n",
      "tf.Tensor(\n",
      "[ 1 78 56 75 60 73 74 14  2  1  2  1 17 26 19  3 30 69 59  3 36 70 59  3\n",
      " 74 56 64 59 12  3 41 60 75  3 75 63 60 73 60  3 57 60  3 67 64 62 63 75\n",
      " 26  3 56 69 59  3 75 63 60 73 60  3 78 56 74  3 67 64 62 63 75 14  2  1\n",
      "  2  1 17 26 20  3 30 69 59  3 36 70 59  3 74 56 78  3 75 63 60  3 67 64\n",
      " 62 63 75 12  3], shape=(101,), dtype=int64)\n",
      "---------------------------------------\n",
      "Observe the conversion from ids to text:\n",
      "b'\\nwaters.\\r\\n\\r\\n1:3 And God said, Let there be light: and there was light.\\r\\n\\r\\n1:4 And God saw the light, '\n",
      "---------------------------------------\n",
      "\n",
      "tf.Tensor(\n",
      "[75 63 56 75  3 64 75  3 78 56 74  3 62 70 70 59 26  3 56 69 59  3 36 70\n",
      " 59  3 59 64 77 64 59 60 59  3 75 63 60  3 67 64 62 63 75  2  1 61 73 70\n",
      " 68  3 75 63 60  3 59 56 73 66 69 60 74 74 14  2  1  2  1 17 26 21  3 30\n",
      " 69 59  3 36 70 59  3 58 56 67 67 60 59  3 75 63 60  3 67 64 62 63 75  3\n",
      " 33 56 80 12  3], shape=(101,), dtype=int64)\n",
      "---------------------------------------\n",
      "Observe the conversion from ids to text:\n",
      "b'that it was good: and God divided the light\\r\\nfrom the darkness.\\r\\n\\r\\n1:5 And God called the light Day, '\n",
      "---------------------------------------\n",
      "\n",
      "tf.Tensor(\n",
      "[56 69 59  3 75 63 60  3 59 56 73 66 69 60 74 74  3 63 60  3 58 56 67 67\n",
      " 60 59  3 43 64 62 63 75 14  2  1 30 69 59  3 75 63 60  3 60 77 60 69 64\n",
      " 69 62  3 56 69 59  3 75 63 60  3 68 70 73 69 64 69 62  3 78 60 73 60  3\n",
      " 75 63 60  3 61 64 73 74 75  3 59 56 80 14  2  1  2  1 17 26 22  3 30 69\n",
      " 59  3 36 70 59], shape=(101,), dtype=int64)\n",
      "---------------------------------------\n",
      "Observe the conversion from ids to text:\n",
      "b'and the darkness he called Night.\\r\\nAnd the evening and the morning were the first day.\\r\\n\\r\\n1:6 And God'\n",
      "---------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## notice how the above dataset tensor object iteratively returns characters\n",
    "## Here, we'll create a new dataset tensor object that returns a 'batch' of characters\n",
    "\n",
    "seq_length = 100 # feel free to vary this integer variable\n",
    "\n",
    "## Note: we added 1 to the seq_length because in the next code chunk, we'll use this object to create our X and y data.\n",
    "sequences = ids_dataset.batch(seq_length+1, drop_remainder=True) \n",
    "\n",
    "for seq in sequences.take(5): # feel free to vary this integer variable\n",
    "  print(seq)\n",
    "  print('---------------------------------------')\n",
    "  print('Observe the conversion from ids to text:')\n",
    "  print(ids_to_text(seq).numpy())\n",
    "  print(\"---------------------------------------\\n\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By this step, we'd have created a dataset tensor object that gives us (seq_length + 1) number of characters. In our example, that would be 100 + 1 = 101. Here's how we use this object to create our input sequences (X) and target sequences (y).\n",
    "\n",
    "For illustration purposes, let's assume our seq_length = 6 and our text is 'Federer'. In that case, our input sequence will be 'Federe' and the target sequence will be 'ederer'. So, we'll create a function that will return 2 output. The first output will be the first 100 (seq_length) number of characters (our X), whilst the second output will be the last 100 (seq_length) number of characters (our y)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input : b'1:1 In the beginning God created the heaven and the earth.\\r\\n\\r\\n1:2 And the earth was without form, an'\n",
      "Target: b':1 In the beginning God created the heaven and the earth.\\r\\n\\r\\n1:2 And the earth was without form, and'\n"
     ]
    }
   ],
   "source": [
    "## Create a function that will produce the X and y variables off a sequence of text, \n",
    "## regardless of the text's given length.\n",
    "\n",
    "def split_into_input_target(sequence):\n",
    "    input_text = sequence[:-1]\n",
    "    target_text = sequence[1:]\n",
    "    return input_text, target_text\n",
    "\n",
    "## use the .map method to create a dataset tensor object \n",
    "## that will always return our X and y outputs respectively when we call it\n",
    "dataset_xy = sequences.map(split_into_input_target)\n",
    "\n",
    "## view an example\n",
    "for input_example, target_example in dataset_xy.take(1):\n",
    "    print(\"Input :\", ids_to_text(input_example).numpy())\n",
    "    print(\"Target:\", ids_to_text(target_example).numpy())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3. Create Training Batches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PrefetchDataset element_spec=(TensorSpec(shape=(64, 100), dtype=tf.int64, name=None), TensorSpec(shape=(64, 100), dtype=tf.int64, name=None))>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Batch size\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "## Before we use our dataset, we'll need to shuffle the data and create batches.\n",
    "## Shuffling helps prevent our model from overfitting.\n",
    "BUFFER_SIZE = 10000\n",
    "\n",
    "dataset = (\n",
    "    dataset_xy\n",
    "    .shuffle(BUFFER_SIZE)\n",
    "    .batch(BATCH_SIZE, drop_remainder=True)\n",
    "    .prefetch(tf.data.experimental.AUTOTUNE)) # this allows later elements to be prepared while the current element is processed\n",
    "\n",
    "dataset"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Building the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "## size of our vocab - i.e. the number of unique characters in our dataset\n",
    "vocab_size = len(ids_from_chars.get_vocabulary())\n",
    "\n",
    "## Our embedding dimension\n",
    "embedding_dim = 256\n",
    "\n",
    "## number of RNN units\n",
    "rnn_units = 1024"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write a function that builds our langage model architecture with the Sequential API\n",
    "# To keep this simple, we'll used 3 simple layers - embedding, GRU and dense. You can\n",
    "# replace GRU with LSTM.\n",
    "\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, GRU, Embedding\n",
    "\n",
    "def build_model(vocab_size, embedding_dim, rnn_units, batch_size):\n",
    "    model = Sequential()\n",
    "    model.add(Embedding(vocab_size\n",
    "                        ,embedding_dim\n",
    "                        ,batch_input_shape=(batch_size,None))) # the use of none gives us flexibility with the input's seqeunce length\n",
    "    model.add(GRU(rnn_units\n",
    "                  ,return_sequences=True # this returns y as the full sequences, rather than the last output\n",
    "                  ,stateful=True)) # This allows LSTMs to have longer context at training time\n",
    "    model.add(Dense(vocab_size)) # Notice how we are not using any softmax activation. \n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = build_model(vocab_size = vocab_size\n",
    "                    ,embedding_dim=embedding_dim\n",
    "                    ,rnn_units=rnn_units\n",
    "                    ,batch_size=BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding (Embedding)       (64, None, 256)           20992     \n",
      "                                                                 \n",
      " gru (GRU)                   (64, None, 1024)          3938304   \n",
      "                                                                 \n",
      " dense (Dense)               (64, None, 82)            84050     \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 4,043,346\n",
      "Trainable params: 4,043,346\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Trying the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(64, 100, 82) # (batch_size, sequence_length, vocab_size)\n"
     ]
    }
   ],
   "source": [
    "for input_example_batch, target_example_batch in dataset.take(1):\n",
    "  example_batch_predictions = model(input_example_batch)\n",
    "  print(example_batch_predictions.shape, \"# (batch_size, sequence_length, vocab_size)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([29, 65,  8, 40,  7, 26, 70, 33, 17, 40, 34,  5, 39,  8,  1, 14, 64,\n",
       "       10,  3, 20, 27, 49, 74, 75, 57, 72, 58, 26, 10, 25, 36, 68, 57,  3,\n",
       "       49, 33, 46, 18, 25, 42,  9, 47, 56, 55, 47, 67,  5, 40, 78, 24, 79,\n",
       "       21, 29, 29, 17, 64, 19, 19, 15, 29, 30, 17, 37, 36, 43, 16, 43, 34,\n",
       "        8,  7, 67, 53, 10,  8, 49, 63, 73, 47, 35, 68, 47, 51, 19, 34, 64,\n",
       "       62, 28, 71, 81, 48,  6, 24, 42, 75, 21, 10,  0, 34, 59, 29])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## To get actual predictions, we will sample from the output distribution.\n",
    "## If we don't sample from an output distribution, it can cause the model to be stuck in a loop. \n",
    "## meaning, it will keep producing a repeated sequence of characters. Hence, we sample.\n",
    "sampled_indices = tf.random.categorical(example_batch_predictions[0], num_samples=1)\n",
    "sampled_indices = tf.squeeze(sampled_indices,axis=-1).numpy()\n",
    "sampled_indices"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Train the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction shape:  (64, 100, 82)  # (batch_size, sequence_length, vocab_size)\n",
      "scalar_loss:       4.405335\n"
     ]
    }
   ],
   "source": [
    "## For our loss calculation, we'll use the 'sparse_categorical_crossentropy'\n",
    "## because the target labels are provided as integers. If they are one-hot representations,\n",
    "## then we'll use 'CategoricalCrossentropy'.\n",
    "def loss(labels, logits):\n",
    "    return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)\n",
    "\n",
    "example_batch_loss  = loss(target_example_batch, example_batch_predictions)\n",
    "print(\"Prediction shape: \", example_batch_predictions.shape, \" # (batch_size, sequence_length, vocab_size)\")\n",
    "print(\"scalar_loss:      \", example_batch_loss.numpy().mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam', loss=loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "688/688 [==============================] - 2109s 3s/step - loss: 1.7533\n",
      "Epoch 2/20\n",
      "688/688 [==============================] - 3030s 4s/step - loss: 1.1778\n",
      "Epoch 3/20\n",
      "688/688 [==============================] - 2841s 4s/step - loss: 1.0850\n",
      "Epoch 4/20\n",
      "688/688 [==============================] - 4240s 6s/step - loss: 1.0392\n",
      "Epoch 5/20\n",
      "688/688 [==============================] - 2553s 4s/step - loss: 1.0081\n",
      "Epoch 6/20\n",
      "688/688 [==============================] - 1470s 2s/step - loss: 0.9836\n",
      "Epoch 7/20\n",
      "688/688 [==============================] - 1433s 2s/step - loss: 0.9639\n",
      "Epoch 8/20\n",
      "688/688 [==============================] - 2031s 3s/step - loss: 0.9466\n",
      "Epoch 9/20\n",
      "688/688 [==============================] - 2386s 3s/step - loss: 0.9323\n",
      "Epoch 10/20\n",
      "688/688 [==============================] - 1660s 2s/step - loss: 0.9200\n",
      "Epoch 11/20\n",
      "688/688 [==============================] - 1462s 2s/step - loss: 0.9093\n",
      "Epoch 12/20\n",
      "688/688 [==============================] - 1339s 2s/step - loss: 0.9004\n",
      "Epoch 13/20\n",
      "688/688 [==============================] - 1025s 1s/step - loss: 0.8938\n",
      "Epoch 14/20\n",
      "688/688 [==============================] - 879s 1s/step - loss: 0.8886\n",
      "Epoch 15/20\n",
      "688/688 [==============================] - 1190s 2s/step - loss: 0.8837\n",
      "Epoch 16/20\n",
      "688/688 [==============================] - 1197s 2s/step - loss: 0.8812\n",
      "Epoch 17/20\n",
      "688/688 [==============================] - 18748s 27s/step - loss: 0.8786\n",
      "Epoch 18/20\n",
      "688/688 [==============================] - 914s 1s/step - loss: 0.8778\n",
      "Epoch 19/20\n",
      "688/688 [==============================] - 1224s 2s/step - loss: 0.8778\n",
      "Epoch 20/20\n",
      "688/688 [==============================] - 1129s 2s/step - loss: 0.8787\n"
     ]
    }
   ],
   "source": [
    "EPOCHS=20\n",
    "history = model.fit(dataset_subclassing, epochs=EPOCHS)#, callbacks=[checkpoint_callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save_weights('sequential-bible-weights.h5')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Generate Text\n",
    "\n",
    "### 7.1. Create a new model with the saved weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "## For the sake of this project, we don't need to produce a BATCH_SIZE worth of different generated texts. \n",
    "## Instead, we just need 1. But, the model has been built to produce the stated BATCH_SIZE.\n",
    "## Hence, we'll need to create a new model, then restore the weights that was saved.\n",
    "\n",
    "bible_text_generator = build_model(vocab_size, embedding_dim, rnn_units, batch_size=1)\n",
    "\n",
    "bible_text_generator.load_weights('sequential-bible-weights.h5')\n",
    "\n",
    "bible_text_generator.build(tf.TensorShape([1, None]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_1 (Embedding)     (1, None, 256)            20992     \n",
      "                                                                 \n",
      " gru_1 (GRU)                 (1, None, 1024)           3938304   \n",
      "                                                                 \n",
      " dense_1 (Dense)             (1, None, 82)             84050     \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 4,043,346\n",
      "Trainable params: 4,043,346\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "bible_text_generator.summary()\n",
    "\n",
    "## observe how index 0 of the output shape are now the batch size of 1."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.2. Create a function that will generate a text for us"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_text(model, start_string, temperature = 1.0, prediction_length = 1000):  \n",
    "  \n",
    "    input_eval = [ids_from_chars(s) for s in start_string]\n",
    "    input_eval = tf.expand_dims(input_eval, 0)\n",
    "   \n",
    "    text_generated = []\n",
    "\n",
    "    model.reset_states() # \n",
    "    for i in range(prediction_length):\n",
    "        predictions = model(input_eval)\n",
    "        predictions = tf.squeeze(predictions, 0)\n",
    "        predictions = predictions / temperature\n",
    "        predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()\n",
    "\n",
    "        input_eval = tf.expand_dims([predicted_id], 0)\n",
    "\n",
    "        text_generated.append(chars_from_ids(predicted_id))\n",
    "\n",
    "    return (start_string + tf.strings.join(text_generated).numpy().decode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Jesus said unto him, All the\n",
      "people which ye had read throughout Pilate that believed not: but if\n",
      "it be for my members shall the flesh of an only persecutions.\n",
      "\n",
      "1:13 How long shall I obtailedo him?  17:12 And when the other sea: and\n",
      "they shall be absent because of the LORD's house.\n",
      "\n",
      "4:12 Behold, I will send a fire unto the body to concerning him that placeth, where\n",
      "are the children of the flesh, that ye be not seen situate by faith unto the commandment\n",
      "of God, and from the strange Lord Jesus\n"
     ]
    }
   ],
   "source": [
    "print(generate_text(bible_text_generator, start_string=u\"Jesus\", temperature = 0.8, prediction_length = 500))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Love the Lord with all your hearts:\n",
      "for they are many that were not ashamed of the faith of the LORD, and the strangers shall be as the\n",
      "same sacrifice was just, and the stars of heaven stood by the coast of the earth, and the hair of his hands toward the\n",
      "south.\n",
      "\n",
      "13:17 And he said unto them, Who are the seven lambs of the first year of king Cyrus, and the\n",
      "children of Israel, and the first and the scribes, and the\n",
      "Levites, and the Jebusites, and the Jebusites, and the\n",
      "Gentiles, and the sons of Zebedee, and of the chi\n"
     ]
    }
   ],
   "source": [
    "print(generate_text(bible_text_generator, start_string=u\"Love the Lord with all your \", temperature = 0.2, prediction_length = 500))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Love the Lord with all your hearts:\n",
      "for they are many that believeth on him that sent me.\n",
      "\n",
      "11:25 And the scribes and Pharisees and all his household, and\n",
      "his mother, and his companions that were with him, and said, Lord, who shall say, Thou art my Son, thou shalt not be ashamed of\n",
      "me that shall be for the sin offering.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(generate_text(bible_text_generator, start_string=u\"Love the Lord with all your \", temperature = 0.2, prediction_length = 300))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "For God so loved the Lord Jesus Christ.\n",
      "\n",
      "1:11 For the things that are sanctified in the flesh of any man of the world, and the\n",
      "world hath not seen him.\n",
      "\n",
      "1:15 And when the chief priests and Pharisees which are sanctified in the wilderness, and the stream is a doer of the flesh.\n",
      "\n",
      "1:18 The LORD shall receive the law\n"
     ]
    }
   ],
   "source": [
    "print(generate_text(bible_text_generator, start_string=u\"For God so loved the \", temperature = 0.2, prediction_length = 300))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ideal Model Building"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# evaluation using BLEU\n",
    "from nltk.translate.bleu_score import corpus_bleu\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Jesus\n",
      "Christ is the spirit of the LORD. Now there eat the water of the sea;\n",
      "in silver and gold, and in the world by the way, through him that believeth in the\n",
      "ends of the earth, and not that we should bear the pen your souls.\n",
      "\n",
      "1:9 For what will I bring again the chief priests for the increase of the General that told him, saying, Abide in the house of\n",
      "Israel, that he saith unto the servants the chief caperia, and all the\n",
      "children of Israel that bare not the holy mouth of the world, and it\n",
      "sha\n"
     ]
    }
   ],
   "source": [
    "pred = generate_text(bible_text_generator, start_string=u\"Jesus\", temperature = 0.8, prediction_length = 500)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_text_mod(model, start_string, temperature = 1.0, prediction_length = 1000):  \n",
    "  \n",
    "    num_generate = prediction_length\n",
    "    input_eval = [ids_from_chars(s) for s in start_string]\n",
    "    input_eval = tf.expand_dims(input_eval, 0)\n",
    "   \n",
    "    text_generated = []\n",
    "\n",
    "    model.reset_states()\n",
    "    for i in range(num_generate):\n",
    "        predictions = model(input_eval)\n",
    "        predictions = tf.squeeze(predictions, 0)\n",
    "        predictions = predictions / temperature\n",
    "        predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()\n",
    "\n",
    "        input_eval = tf.expand_dims([predicted_id], 0)\n",
    "\n",
    "        text_generated.append(chars_from_ids(predicted_id))\n",
    "\n",
    "    return text_generated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=string, numpy=b','>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'w'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'y'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'k'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'c'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'v'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b','>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'u'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'g'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'm'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'g'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'w'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b','>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'v'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'y'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'.'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'2'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'2'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b':'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'2'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'2'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'A'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'u'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'm'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b','>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'M'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'k'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'p'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'p'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'.'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'3'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b':'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'5'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'W'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'u'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'j'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'g'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'm'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'y'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'!'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'2'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'4'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b':'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'2'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'2'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'T'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'u'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'y'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'b'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'b'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'u'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'g'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b','>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'y'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'w'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'b'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'b'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'c'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'm'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'm'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'm'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'G'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b','>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'b'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'u'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'g'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'P'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'g'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'p'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'w'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b';'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'1'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b':'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'5'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'T'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'u'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'I'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'm'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'u'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'w'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'g'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'b'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'u'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b','>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'l'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'd'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'w'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'f'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'i'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b's'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b't'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'h'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b' '>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'o'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'e'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\r'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'\\n'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'w'>,\n",
       " <tf.Tensor: shape=(), dtype=string, numpy=b'a'>]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred = generate_text_mod(bible_text_generator, start_string=u\"Jesus\", temperature = 0.8, prediction_length = 500)\n",
    "pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[b'',\n",
       " b'',\n",
       " b'who',\n",
       " b'did',\n",
       " b'they',\n",
       " b'think',\n",
       " b'their',\n",
       " b'',\n",
       " b'covenant',\n",
       " b'',\n",
       " b'and',\n",
       " b'our',\n",
       " b'transgressions',\n",
       " b'among',\n",
       " b'the',\n",
       " b'world',\n",
       " b'',\n",
       " b'even',\n",
       " b'',\n",
       " b'as',\n",
       " b'this',\n",
       " b'day.',\n",
       " b'',\n",
       " b'',\n",
       " b'',\n",
       " b'22:22',\n",
       " b'And',\n",
       " b'he',\n",
       " b'said',\n",
       " b'unto',\n",
       " b'them',\n",
       " b'',\n",
       " b'Matten',\n",
       " b'seek',\n",
       " b'hold',\n",
       " b'the',\n",
       " b'people',\n",
       " b'of',\n",
       " b'the',\n",
       " b'land.',\n",
       " b'',\n",
       " b'',\n",
       " b'',\n",
       " b'3:5',\n",
       " b'Wilt',\n",
       " b'thou',\n",
       " b'join',\n",
       " b'the',\n",
       " b'light',\n",
       " b'of',\n",
       " b'my',\n",
       " b'hand!',\n",
       " b'',\n",
       " b'24:22',\n",
       " b'Thou',\n",
       " b'hast',\n",
       " b'already',\n",
       " b'to',\n",
       " b'be',\n",
       " b'brought',\n",
       " b'forth',\n",
       " b'',\n",
       " b'and',\n",
       " b'if',\n",
       " b'ye',\n",
       " b'will',\n",
       " b'be',\n",
       " b'no',\n",
       " b'',\n",
       " b'bread',\n",
       " b'for',\n",
       " b'the',\n",
       " b'commandment',\n",
       " b'of',\n",
       " b'God',\n",
       " b'',\n",
       " b'and',\n",
       " b'then',\n",
       " b'shall',\n",
       " b'he',\n",
       " b'eat',\n",
       " b'the',\n",
       " b'',\n",
       " b'burnt',\n",
       " b'offering',\n",
       " b'of',\n",
       " b'a',\n",
       " b'Prine',\n",
       " b'of',\n",
       " b'great',\n",
       " b'power;',\n",
       " b'1:5',\n",
       " b'To',\n",
       " b'the',\n",
       " b'house',\n",
       " b'of',\n",
       " b'Israel',\n",
       " b'',\n",
       " b'must',\n",
       " b'need',\n",
       " b'with',\n",
       " b'a',\n",
       " b'great',\n",
       " b'stead',\n",
       " b'of',\n",
       " b'a',\n",
       " b'burnt',\n",
       " b'dead',\n",
       " b'',\n",
       " b'fell',\n",
       " b'down',\n",
       " b'at',\n",
       " b'the',\n",
       " b'first',\n",
       " b'set',\n",
       " b'on',\n",
       " b'the',\n",
       " b'one',\n",
       " b'',\n",
       " b'wa']"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "text = tf.strings.join(pred).numpy()\n",
    "re.split(b',|\\n|\\r| ', text)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['who',\n",
       " 'did',\n",
       " 'they',\n",
       " 'think',\n",
       " 'their',\n",
       " 'covenant',\n",
       " 'and',\n",
       " 'our',\n",
       " 'transgressions',\n",
       " 'among',\n",
       " 'the',\n",
       " 'world',\n",
       " 'even',\n",
       " 'as',\n",
       " 'this',\n",
       " 'day',\n",
       " '22',\n",
       " '22',\n",
       " 'And',\n",
       " 'he',\n",
       " 'said',\n",
       " 'unto',\n",
       " 'them',\n",
       " 'Matten',\n",
       " 'seek',\n",
       " 'hold',\n",
       " 'the',\n",
       " 'people',\n",
       " 'of',\n",
       " 'the',\n",
       " 'land',\n",
       " '3',\n",
       " '5',\n",
       " 'Wilt',\n",
       " 'thou',\n",
       " 'join',\n",
       " 'the',\n",
       " 'light',\n",
       " 'of',\n",
       " 'my',\n",
       " 'hand',\n",
       " '24',\n",
       " '22',\n",
       " 'Thou',\n",
       " 'hast',\n",
       " 'already',\n",
       " 'to',\n",
       " 'be',\n",
       " 'brought',\n",
       " 'forth',\n",
       " 'and',\n",
       " 'if',\n",
       " 'ye',\n",
       " 'will',\n",
       " 'be',\n",
       " 'no',\n",
       " 'bread',\n",
       " 'for',\n",
       " 'the',\n",
       " 'commandment',\n",
       " 'of',\n",
       " 'God',\n",
       " 'and',\n",
       " 'then',\n",
       " 'shall',\n",
       " 'he',\n",
       " 'eat',\n",
       " 'the',\n",
       " 'burnt',\n",
       " 'offering',\n",
       " 'of',\n",
       " 'a',\n",
       " 'Prine',\n",
       " 'of',\n",
       " 'great',\n",
       " 'power',\n",
       " '1',\n",
       " '5',\n",
       " 'To',\n",
       " 'the',\n",
       " 'house',\n",
       " 'of',\n",
       " 'Israel',\n",
       " 'must',\n",
       " 'need',\n",
       " 'with',\n",
       " 'a',\n",
       " 'great',\n",
       " 'stead',\n",
       " 'of',\n",
       " 'a',\n",
       " 'burnt',\n",
       " 'dead',\n",
       " 'fell',\n",
       " 'down',\n",
       " 'at',\n",
       " 'the',\n",
       " 'first',\n",
       " 'set',\n",
       " 'on',\n",
       " 'the',\n",
       " 'one',\n",
       " 'wa']"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.tokenize import RegexpTokenizer\n",
    "\n",
    "tokenizer = RegexpTokenizer(r'\\w+')\n",
    "candidates = tokenizer.tokenize(tf.strings.join(pred).numpy().decode('utf-8'))\n",
    "candidates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['1',\n",
       "  '1',\n",
       "  'In',\n",
       "  'the',\n",
       "  'beginning',\n",
       "  'God',\n",
       "  'created',\n",
       "  'the',\n",
       "  'heaven',\n",
       "  'and',\n",
       "  'the',\n",
       "  'earth',\n",
       "  '1',\n",
       "  '2',\n",
       "  'And',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'was',\n",
       "  'without',\n",
       "  'form',\n",
       "  'and',\n",
       "  'void',\n",
       "  'and',\n",
       "  'darkness',\n",
       "  'was',\n",
       "  'upon',\n",
       "  'the',\n",
       "  'face',\n",
       "  'of',\n",
       "  'the',\n",
       "  'deep',\n",
       "  'And',\n",
       "  'the',\n",
       "  'Spirit',\n",
       "  'of',\n",
       "  'God',\n",
       "  'moved',\n",
       "  'upon',\n",
       "  'the',\n",
       "  'face',\n",
       "  'of',\n",
       "  'the',\n",
       "  'waters',\n",
       "  '1',\n",
       "  '3',\n",
       "  'And',\n",
       "  'God',\n",
       "  'said',\n",
       "  'Let',\n",
       "  'there',\n",
       "  'be',\n",
       "  'light',\n",
       "  'and',\n",
       "  'there',\n",
       "  'was',\n",
       "  'light',\n",
       "  '1',\n",
       "  '4',\n",
       "  'And',\n",
       "  'God',\n",
       "  'saw',\n",
       "  'the',\n",
       "  'light',\n",
       "  'that',\n",
       "  'it',\n",
       "  'was',\n",
       "  'good',\n",
       "  'and',\n",
       "  'God',\n",
       "  'divided',\n",
       "  'the',\n",
       "  'light',\n",
       "  'from',\n",
       "  'the',\n",
       "  'darkness',\n",
       "  '1',\n",
       "  '5',\n",
       "  'And',\n",
       "  'God',\n",
       "  'called',\n",
       "  'the',\n",
       "  'light',\n",
       "  'Day',\n",
       "  'and',\n",
       "  'the',\n",
       "  'darkness',\n",
       "  'he',\n",
       "  'called',\n",
       "  'Night',\n",
       "  'And',\n",
       "  'the',\n",
       "  'evening',\n",
       "  'and',\n",
       "  'the',\n",
       "  'morning',\n",
       "  'were',\n",
       "  'the',\n",
       "  'first',\n",
       "  'day',\n",
       "  '1',\n",
       "  '6',\n",
       "  'And',\n",
       "  'God',\n",
       "  'said',\n",
       "  'Let',\n",
       "  'there',\n",
       "  'be',\n",
       "  'a',\n",
       "  'firmament',\n",
       "  'in',\n",
       "  'the',\n",
       "  'midst',\n",
       "  'of',\n",
       "  'the',\n",
       "  'waters',\n",
       "  'and',\n",
       "  'let',\n",
       "  'it',\n",
       "  'divide',\n",
       "  'the',\n",
       "  'waters',\n",
       "  'from',\n",
       "  'the',\n",
       "  'waters',\n",
       "  '1',\n",
       "  '7',\n",
       "  'And',\n",
       "  'God',\n",
       "  'made',\n",
       "  'the',\n",
       "  'firmament',\n",
       "  'and',\n",
       "  'divided',\n",
       "  'the',\n",
       "  'waters',\n",
       "  'which',\n",
       "  'were',\n",
       "  'under',\n",
       "  'the',\n",
       "  'firmament',\n",
       "  'from',\n",
       "  'the',\n",
       "  'waters',\n",
       "  'which',\n",
       "  'were',\n",
       "  'above',\n",
       "  'the',\n",
       "  'firmament',\n",
       "  'and',\n",
       "  'it',\n",
       "  'was',\n",
       "  'so',\n",
       "  '1',\n",
       "  '8',\n",
       "  'And',\n",
       "  'God',\n",
       "  'called',\n",
       "  'the',\n",
       "  'firmament',\n",
       "  'Heaven',\n",
       "  'And',\n",
       "  'the',\n",
       "  'evening',\n",
       "  'and',\n",
       "  'the',\n",
       "  'morning',\n",
       "  'were',\n",
       "  'the',\n",
       "  'second',\n",
       "  'day',\n",
       "  '1',\n",
       "  '9',\n",
       "  'And',\n",
       "  'God',\n",
       "  'said',\n",
       "  'Let',\n",
       "  'the',\n",
       "  'waters',\n",
       "  'under',\n",
       "  'the',\n",
       "  'heaven',\n",
       "  'be',\n",
       "  'gathered',\n",
       "  'together',\n",
       "  'unto',\n",
       "  'one',\n",
       "  'place',\n",
       "  'and',\n",
       "  'let',\n",
       "  'the',\n",
       "  'dry',\n",
       "  'land',\n",
       "  'appear',\n",
       "  'and',\n",
       "  'it',\n",
       "  'was',\n",
       "  'so',\n",
       "  '1',\n",
       "  '10',\n",
       "  'And',\n",
       "  'God',\n",
       "  'called',\n",
       "  'the',\n",
       "  'dry',\n",
       "  'land',\n",
       "  'Earth',\n",
       "  'and',\n",
       "  'the',\n",
       "  'gathering',\n",
       "  'together',\n",
       "  'of',\n",
       "  'the',\n",
       "  'waters',\n",
       "  'called',\n",
       "  'he',\n",
       "  'Seas',\n",
       "  'and',\n",
       "  'God',\n",
       "  'saw',\n",
       "  'that',\n",
       "  'it',\n",
       "  'was',\n",
       "  'good',\n",
       "  '1',\n",
       "  '11',\n",
       "  'And',\n",
       "  'God',\n",
       "  'said',\n",
       "  'Let',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'bring',\n",
       "  'forth',\n",
       "  'grass',\n",
       "  'the',\n",
       "  'herb',\n",
       "  'yielding',\n",
       "  'seed',\n",
       "  'and',\n",
       "  'the',\n",
       "  'fruit',\n",
       "  'tree',\n",
       "  'yielding',\n",
       "  'fruit',\n",
       "  'after',\n",
       "  'his',\n",
       "  'kind',\n",
       "  'whose',\n",
       "  'seed',\n",
       "  'is',\n",
       "  'in',\n",
       "  'itself',\n",
       "  'upon',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'and',\n",
       "  'it',\n",
       "  'was',\n",
       "  'so',\n",
       "  '1',\n",
       "  '12',\n",
       "  'And',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'brought',\n",
       "  'forth',\n",
       "  'grass',\n",
       "  'and',\n",
       "  'herb',\n",
       "  'yielding',\n",
       "  'seed',\n",
       "  'after',\n",
       "  'his',\n",
       "  'kind',\n",
       "  'and',\n",
       "  'the',\n",
       "  'tree',\n",
       "  'yielding',\n",
       "  'fruit',\n",
       "  'whose',\n",
       "  'seed',\n",
       "  'was',\n",
       "  'in',\n",
       "  'itself',\n",
       "  'after',\n",
       "  'his',\n",
       "  'kind',\n",
       "  'and',\n",
       "  'God',\n",
       "  'saw',\n",
       "  'that',\n",
       "  'it',\n",
       "  'was',\n",
       "  'good',\n",
       "  '1',\n",
       "  '13',\n",
       "  'And',\n",
       "  'the',\n",
       "  'evening',\n",
       "  'and',\n",
       "  'the',\n",
       "  'morning',\n",
       "  'were',\n",
       "  'the',\n",
       "  'third',\n",
       "  'day',\n",
       "  '1',\n",
       "  '14',\n",
       "  'And',\n",
       "  'God',\n",
       "  'said',\n",
       "  'Let',\n",
       "  'there',\n",
       "  'be',\n",
       "  'lights',\n",
       "  'in',\n",
       "  'the',\n",
       "  'firmament',\n",
       "  'of',\n",
       "  'the',\n",
       "  'heaven',\n",
       "  'to',\n",
       "  'divide',\n",
       "  'the',\n",
       "  'day',\n",
       "  'from',\n",
       "  'the',\n",
       "  'night',\n",
       "  'and',\n",
       "  'let',\n",
       "  'them',\n",
       "  'be',\n",
       "  'for',\n",
       "  'signs',\n",
       "  'and',\n",
       "  'for',\n",
       "  'seasons',\n",
       "  'and',\n",
       "  'for',\n",
       "  'days',\n",
       "  'and',\n",
       "  'years',\n",
       "  '1',\n",
       "  '15',\n",
       "  'And',\n",
       "  'let',\n",
       "  'them',\n",
       "  'be',\n",
       "  'for',\n",
       "  'lights',\n",
       "  'in',\n",
       "  'the',\n",
       "  'firmament',\n",
       "  'of',\n",
       "  'the',\n",
       "  'heaven',\n",
       "  'to',\n",
       "  'give',\n",
       "  'light',\n",
       "  'upon',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'and',\n",
       "  'it',\n",
       "  'was',\n",
       "  'so',\n",
       "  '1',\n",
       "  '16',\n",
       "  'And',\n",
       "  'God',\n",
       "  'made',\n",
       "  'two',\n",
       "  'great',\n",
       "  'lights',\n",
       "  'the',\n",
       "  'greater',\n",
       "  'light',\n",
       "  'to',\n",
       "  'rule',\n",
       "  'the',\n",
       "  'day',\n",
       "  'and',\n",
       "  'the',\n",
       "  'lesser',\n",
       "  'light',\n",
       "  'to',\n",
       "  'rule',\n",
       "  'the',\n",
       "  'night',\n",
       "  'he',\n",
       "  'made',\n",
       "  'the',\n",
       "  'stars',\n",
       "  'also',\n",
       "  '1',\n",
       "  '17',\n",
       "  'And',\n",
       "  'God',\n",
       "  'set',\n",
       "  'them',\n",
       "  'in',\n",
       "  'the',\n",
       "  'firmament',\n",
       "  'of',\n",
       "  'the',\n",
       "  'heaven',\n",
       "  'to',\n",
       "  'give',\n",
       "  'light',\n",
       "  'upon',\n",
       "  'the',\n",
       "  'earth',\n",
       "  '1',\n",
       "  '18',\n",
       "  'And',\n",
       "  'to',\n",
       "  'rule',\n",
       "  'over',\n",
       "  'the',\n",
       "  'day',\n",
       "  'and',\n",
       "  'over',\n",
       "  'the',\n",
       "  'night',\n",
       "  'and',\n",
       "  'to',\n",
       "  'divide',\n",
       "  'the',\n",
       "  'light',\n",
       "  'from',\n",
       "  'the',\n",
       "  'darkness',\n",
       "  'and',\n",
       "  'God',\n",
       "  'saw',\n",
       "  'that',\n",
       "  'it',\n",
       "  'was',\n",
       "  'good',\n",
       "  '1',\n",
       "  '19',\n",
       "  'And',\n",
       "  'the',\n",
       "  'evening',\n",
       "  'and',\n",
       "  'the',\n",
       "  'morning',\n",
       "  'were',\n",
       "  'the',\n",
       "  'fourth',\n",
       "  'day',\n",
       "  '1',\n",
       "  '20',\n",
       "  'And',\n",
       "  'God',\n",
       "  'said',\n",
       "  'Let',\n",
       "  'the',\n",
       "  'waters',\n",
       "  'bring',\n",
       "  'forth',\n",
       "  'abundantly',\n",
       "  'the',\n",
       "  'moving',\n",
       "  'creature',\n",
       "  'that',\n",
       "  'hath',\n",
       "  'life',\n",
       "  'and',\n",
       "  'fowl',\n",
       "  'that',\n",
       "  'may',\n",
       "  'fly',\n",
       "  'above',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'in',\n",
       "  'the',\n",
       "  'open',\n",
       "  'firmament',\n",
       "  'of',\n",
       "  'heaven',\n",
       "  '1',\n",
       "  '21',\n",
       "  'And',\n",
       "  'God',\n",
       "  'created',\n",
       "  'great',\n",
       "  'whales',\n",
       "  'and',\n",
       "  'every',\n",
       "  'living',\n",
       "  'creature',\n",
       "  'that',\n",
       "  'moveth',\n",
       "  'which',\n",
       "  'the',\n",
       "  'waters',\n",
       "  'brought',\n",
       "  'forth',\n",
       "  'abundantly',\n",
       "  'after',\n",
       "  'their',\n",
       "  'kind',\n",
       "  'and',\n",
       "  'every',\n",
       "  'winged',\n",
       "  'fowl',\n",
       "  'after',\n",
       "  'his',\n",
       "  'kind',\n",
       "  'and',\n",
       "  'God',\n",
       "  'saw',\n",
       "  'that',\n",
       "  'it',\n",
       "  'was',\n",
       "  'good',\n",
       "  '1',\n",
       "  '22',\n",
       "  'And',\n",
       "  'God',\n",
       "  'blessed',\n",
       "  'them',\n",
       "  'saying',\n",
       "  'Be',\n",
       "  'fruitful',\n",
       "  'and',\n",
       "  'multiply',\n",
       "  'and',\n",
       "  'fill',\n",
       "  'the',\n",
       "  'waters',\n",
       "  'in',\n",
       "  'the',\n",
       "  'seas',\n",
       "  'and',\n",
       "  'let',\n",
       "  'fowl',\n",
       "  'multiply',\n",
       "  'in',\n",
       "  'the',\n",
       "  'earth',\n",
       "  '1',\n",
       "  '23',\n",
       "  'And',\n",
       "  'the',\n",
       "  'evening',\n",
       "  'and',\n",
       "  'the',\n",
       "  'morning',\n",
       "  'were',\n",
       "  'the',\n",
       "  'fifth',\n",
       "  'day',\n",
       "  '1',\n",
       "  '24',\n",
       "  'And',\n",
       "  'God',\n",
       "  'said',\n",
       "  'Let',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'bring',\n",
       "  'forth',\n",
       "  'the',\n",
       "  'living',\n",
       "  'creature',\n",
       "  'after',\n",
       "  'his',\n",
       "  'kind',\n",
       "  'cattle',\n",
       "  'and',\n",
       "  'creeping',\n",
       "  'thing',\n",
       "  'and',\n",
       "  'beast',\n",
       "  'of',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'after',\n",
       "  'his',\n",
       "  'kind',\n",
       "  'and',\n",
       "  'it',\n",
       "  'was',\n",
       "  'so',\n",
       "  '1',\n",
       "  '25',\n",
       "  'And',\n",
       "  'God',\n",
       "  'made',\n",
       "  'the',\n",
       "  'beast',\n",
       "  'of',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'after',\n",
       "  'his',\n",
       "  'kind',\n",
       "  'and',\n",
       "  'cattle',\n",
       "  'after',\n",
       "  'their',\n",
       "  'kind',\n",
       "  'and',\n",
       "  'every',\n",
       "  'thing',\n",
       "  'that',\n",
       "  'creepeth',\n",
       "  'upon',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'after',\n",
       "  'his',\n",
       "  'kind',\n",
       "  'and',\n",
       "  'God',\n",
       "  'saw',\n",
       "  'that',\n",
       "  'it',\n",
       "  'was',\n",
       "  'good',\n",
       "  '1',\n",
       "  '26',\n",
       "  'And',\n",
       "  'God',\n",
       "  'said',\n",
       "  'Let',\n",
       "  'us',\n",
       "  'make',\n",
       "  'man',\n",
       "  'in',\n",
       "  'our',\n",
       "  'image',\n",
       "  'after',\n",
       "  'our',\n",
       "  'likeness',\n",
       "  'and',\n",
       "  'let',\n",
       "  'them',\n",
       "  'have',\n",
       "  'dominion',\n",
       "  'over',\n",
       "  'the',\n",
       "  'fish',\n",
       "  'of',\n",
       "  'the',\n",
       "  'sea',\n",
       "  'and',\n",
       "  'over',\n",
       "  'the',\n",
       "  'fowl',\n",
       "  'of',\n",
       "  'the',\n",
       "  'air',\n",
       "  'and',\n",
       "  'over',\n",
       "  'the',\n",
       "  'cattle',\n",
       "  'and',\n",
       "  'over',\n",
       "  'all',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'and',\n",
       "  'over',\n",
       "  'every',\n",
       "  'creeping',\n",
       "  'thing',\n",
       "  'that',\n",
       "  'creepeth',\n",
       "  'upon',\n",
       "  'the',\n",
       "  'earth',\n",
       "  '1',\n",
       "  '27',\n",
       "  'So',\n",
       "  'God',\n",
       "  'created',\n",
       "  'man',\n",
       "  'in',\n",
       "  'his',\n",
       "  'own',\n",
       "  'image',\n",
       "  'in',\n",
       "  'the',\n",
       "  'image',\n",
       "  'of',\n",
       "  'God',\n",
       "  'created',\n",
       "  'he',\n",
       "  'him',\n",
       "  'male',\n",
       "  'and',\n",
       "  'female',\n",
       "  'created',\n",
       "  'he',\n",
       "  'them',\n",
       "  '1',\n",
       "  '28',\n",
       "  'And',\n",
       "  'God',\n",
       "  'blessed',\n",
       "  'them',\n",
       "  'and',\n",
       "  'God',\n",
       "  'said',\n",
       "  'unto',\n",
       "  'them',\n",
       "  'Be',\n",
       "  'fruitful',\n",
       "  'and',\n",
       "  'multiply',\n",
       "  'and',\n",
       "  'replenish',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'and',\n",
       "  'subdue',\n",
       "  'it',\n",
       "  'and',\n",
       "  'have',\n",
       "  'dominion',\n",
       "  'over',\n",
       "  'the',\n",
       "  'fish',\n",
       "  'of',\n",
       "  'the',\n",
       "  'sea',\n",
       "  'and',\n",
       "  'over',\n",
       "  'the',\n",
       "  'fowl',\n",
       "  'of',\n",
       "  'the',\n",
       "  'air',\n",
       "  'and',\n",
       "  'over',\n",
       "  'every',\n",
       "  'living',\n",
       "  'thing',\n",
       "  'that',\n",
       "  'moveth',\n",
       "  'upon',\n",
       "  'the',\n",
       "  'earth',\n",
       "  '1',\n",
       "  '29',\n",
       "  'And',\n",
       "  'God',\n",
       "  'said',\n",
       "  'Behold',\n",
       "  'I',\n",
       "  'have',\n",
       "  'given',\n",
       "  'you',\n",
       "  'every',\n",
       "  'herb',\n",
       "  'bearing',\n",
       "  'seed',\n",
       "  'which',\n",
       "  'is',\n",
       "  'upon',\n",
       "  'the',\n",
       "  'face',\n",
       "  'of',\n",
       "  'all',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'and',\n",
       "  'every',\n",
       "  'tree',\n",
       "  'in',\n",
       "  'the',\n",
       "  'which',\n",
       "  'is',\n",
       "  'the',\n",
       "  'fruit',\n",
       "  'of',\n",
       "  'a',\n",
       "  'tree',\n",
       "  'yielding',\n",
       "  'seed',\n",
       "  'to',\n",
       "  'you',\n",
       "  'it',\n",
       "  'shall',\n",
       "  'be',\n",
       "  'for',\n",
       "  'meat',\n",
       "  '1',\n",
       "  '30',\n",
       "  'And',\n",
       "  'to',\n",
       "  'every',\n",
       "  'beast',\n",
       "  'of',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'and',\n",
       "  'to',\n",
       "  'every',\n",
       "  'fowl',\n",
       "  'of',\n",
       "  'the',\n",
       "  'air',\n",
       "  'and',\n",
       "  'to',\n",
       "  'every',\n",
       "  'thing',\n",
       "  'that',\n",
       "  'creepeth',\n",
       "  'upon',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'wherein',\n",
       "  'there',\n",
       "  'is',\n",
       "  'life',\n",
       "  'I',\n",
       "  'have',\n",
       "  'given',\n",
       "  'every',\n",
       "  'green',\n",
       "  'herb',\n",
       "  'for',\n",
       "  'meat',\n",
       "  'and',\n",
       "  'it',\n",
       "  'was',\n",
       "  'so',\n",
       "  '1',\n",
       "  '31',\n",
       "  'And',\n",
       "  'God',\n",
       "  'saw',\n",
       "  'every',\n",
       "  'thing',\n",
       "  'that',\n",
       "  'he',\n",
       "  'had',\n",
       "  'made',\n",
       "  'and',\n",
       "  'behold',\n",
       "  'it',\n",
       "  'was',\n",
       "  'very',\n",
       "  'good',\n",
       "  'And',\n",
       "  'the',\n",
       "  'evening',\n",
       "  'and',\n",
       "  'the',\n",
       "  'morning',\n",
       "  'were',\n",
       "  'the',\n",
       "  'sixth',\n",
       "  'day',\n",
       "  '2',\n",
       "  '1',\n",
       "  'Thus',\n",
       "  'the',\n",
       "  'heavens',\n",
       "  'and',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'were',\n",
       "  'finished',\n",
       "  'and',\n",
       "  'all',\n",
       "  'the',\n",
       "  'host',\n",
       "  'of',\n",
       "  'them',\n",
       "  '2',\n",
       "  '2',\n",
       "  'And',\n",
       "  'on',\n",
       "  'the',\n",
       "  'seventh',\n",
       "  'day',\n",
       "  'God',\n",
       "  'ended',\n",
       "  'his',\n",
       "  'work',\n",
       "  'which',\n",
       "  'he',\n",
       "  'had',\n",
       "  'made',\n",
       "  'and',\n",
       "  'he',\n",
       "  'rested',\n",
       "  'on',\n",
       "  'the',\n",
       "  'seventh',\n",
       "  'day',\n",
       "  'from',\n",
       "  'all',\n",
       "  'his',\n",
       "  'work',\n",
       "  'which',\n",
       "  'he',\n",
       "  'had',\n",
       "  'made',\n",
       "  '2',\n",
       "  '3',\n",
       "  'And',\n",
       "  'God',\n",
       "  'blessed',\n",
       "  'the',\n",
       "  'seventh',\n",
       "  'day',\n",
       "  'and',\n",
       "  'sanctified',\n",
       "  'it',\n",
       "  'because',\n",
       "  'that',\n",
       "  'in',\n",
       "  'it',\n",
       "  'he',\n",
       "  'had',\n",
       "  'rested',\n",
       "  'from',\n",
       "  'all',\n",
       "  'his',\n",
       "  'work',\n",
       "  'which',\n",
       "  'God',\n",
       "  'created',\n",
       "  'and',\n",
       "  'made',\n",
       "  '2',\n",
       "  '4',\n",
       "  'These',\n",
       "  'are',\n",
       "  'the',\n",
       "  'generations',\n",
       "  'of',\n",
       "  'the',\n",
       "  'heavens',\n",
       "  'and',\n",
       "  'of',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'when',\n",
       "  'they',\n",
       "  'were',\n",
       "  'created',\n",
       "  'in',\n",
       "  'the',\n",
       "  'day',\n",
       "  'that',\n",
       "  'the',\n",
       "  'LORD',\n",
       "  'God',\n",
       "  'made',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'and',\n",
       "  'the',\n",
       "  'heavens',\n",
       "  '2',\n",
       "  '5',\n",
       "  'And',\n",
       "  'every',\n",
       "  'plant',\n",
       "  'of',\n",
       "  'the',\n",
       "  'field',\n",
       "  'before',\n",
       "  'it',\n",
       "  'was',\n",
       "  'in',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'and',\n",
       "  'every',\n",
       "  'herb',\n",
       "  'of',\n",
       "  'the',\n",
       "  'field',\n",
       "  'before',\n",
       "  'it',\n",
       "  'grew',\n",
       "  'for',\n",
       "  'the',\n",
       "  'LORD',\n",
       "  'God',\n",
       "  'had',\n",
       "  'not',\n",
       "  'caused',\n",
       "  'it',\n",
       "  'to',\n",
       "  'rain',\n",
       "  'upon',\n",
       "  'the',\n",
       "  'earth',\n",
       "  'and',\n",
       "  'there',\n",
       "  ...]]"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "references = [tokenizer.tokenize(text)]\n",
    "references\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7.431554310846816e-291\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/nltk/translate/bleu_score.py:552: UserWarning: \n",
      "The hypothesis contains 0 counts of 2-gram overlaps.\n",
      "Therefore the BLEU score evaluates to 0, independently of\n",
      "how many N-gram overlaps of lower order it contains.\n",
      "Consider using lower n-gram order or use SmoothingFunction()\n",
      "  warnings.warn(_msg)\n",
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/nltk/translate/bleu_score.py:552: UserWarning: \n",
      "The hypothesis contains 0 counts of 3-gram overlaps.\n",
      "Therefore the BLEU score evaluates to 0, independently of\n",
      "how many N-gram overlaps of lower order it contains.\n",
      "Consider using lower n-gram order or use SmoothingFunction()\n",
      "  warnings.warn(_msg)\n",
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/nltk/translate/bleu_score.py:552: UserWarning: \n",
      "The hypothesis contains 0 counts of 4-gram overlaps.\n",
      "Therefore the BLEU score evaluates to 0, independently of\n",
      "how many N-gram overlaps of lower order it contains.\n",
      "Consider using lower n-gram order or use SmoothingFunction()\n",
      "  warnings.warn(_msg)\n"
     ]
    }
   ],
   "source": [
    "from nltk.translate.bleu_score import sentence_bleu\n",
    "\n",
    "score = sentence_bleu([list(set(tokenizer.tokenize(text)))], candidates)\n",
    "print(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Perazim',\n",
       " 'Betharabah',\n",
       " 'chop',\n",
       " 'Libni',\n",
       " 'inheritor',\n",
       " 'unstable',\n",
       " 'desolation',\n",
       " 'clothes',\n",
       " 'almond',\n",
       " 'Urijah',\n",
       " '171',\n",
       " 'disposing',\n",
       " 'spindle',\n",
       " 'Joiarib',\n",
       " 'Spring',\n",
       " 'Arba',\n",
       " 'temper',\n",
       " 'row',\n",
       " 'accompanying',\n",
       " 'desiredst',\n",
       " 'rings',\n",
       " 'Should',\n",
       " 'seats',\n",
       " 'hart',\n",
       " 'Booz',\n",
       " 'within',\n",
       " 'Tabbath',\n",
       " 'whirlwind',\n",
       " '91',\n",
       " 'Sargon',\n",
       " 'gutenberg',\n",
       " 'runneth',\n",
       " 'Tekoa',\n",
       " 'spoilest',\n",
       " 'Beloved',\n",
       " 'Naarath',\n",
       " 'gold',\n",
       " 'gave',\n",
       " 'TRADEMARK',\n",
       " 'accursed',\n",
       " 'overthrow',\n",
       " 'Gilboa',\n",
       " 'Hannathon',\n",
       " 'Remain',\n",
       " 'Rohgah',\n",
       " 'ghost',\n",
       " 'bemoan',\n",
       " 'Irshemesh',\n",
       " 'Kedar',\n",
       " 'tingle',\n",
       " 'odd',\n",
       " 'shameful',\n",
       " 'can',\n",
       " 'Geshur',\n",
       " 'shamefacedness',\n",
       " 'Eliphaz',\n",
       " 'enjoin',\n",
       " 'company',\n",
       " 'washing',\n",
       " 'poorer',\n",
       " 'hasty',\n",
       " 'Zephon',\n",
       " 'Meshullemeth',\n",
       " 'gender',\n",
       " 'Gemariah',\n",
       " 'couches',\n",
       " 'spared',\n",
       " 'possessions',\n",
       " 'Tilgathpilneser',\n",
       " 'deceiveth',\n",
       " 'Havilah',\n",
       " 'Forasmuch',\n",
       " 'gropeth',\n",
       " 'Pharisees',\n",
       " 'transforming',\n",
       " 'Happy',\n",
       " 'Withal',\n",
       " 'Cherith',\n",
       " 'wagon',\n",
       " 'Hazaraddar',\n",
       " 'Obil',\n",
       " 'Carcas',\n",
       " 'warning',\n",
       " 'buriers',\n",
       " 'drowned',\n",
       " 'contemned',\n",
       " 'bucklers',\n",
       " 'bolster',\n",
       " 'Arnon',\n",
       " 'ringleader',\n",
       " 'succoured',\n",
       " 'quenched',\n",
       " 'Hod',\n",
       " 'assemblies',\n",
       " 'jewels',\n",
       " 'Zobebah',\n",
       " 'slaughter',\n",
       " 'Moladah',\n",
       " 'North',\n",
       " 'thankful',\n",
       " 'fix',\n",
       " 'fellows',\n",
       " 'suffered',\n",
       " 'Hadattah',\n",
       " 'Azgad',\n",
       " 'coulters',\n",
       " 'breathed',\n",
       " 'neglecting',\n",
       " 'heels',\n",
       " 'his',\n",
       " 'shall',\n",
       " 'Hul',\n",
       " 'promising',\n",
       " 'has',\n",
       " 'seducing',\n",
       " 'quantity',\n",
       " 'gluttonous',\n",
       " 'straightway',\n",
       " 'Godhead',\n",
       " 'self',\n",
       " 'Bedan',\n",
       " 'ladder',\n",
       " 'stumblingblock',\n",
       " 'wickedness',\n",
       " 'fit',\n",
       " 'obedience',\n",
       " 'Being',\n",
       " 'mightest',\n",
       " 'well',\n",
       " 'bondservant',\n",
       " 'defended',\n",
       " 'Luhith',\n",
       " 'rough',\n",
       " 'Royalty',\n",
       " 'Lahad',\n",
       " 'bow',\n",
       " 'Eltekeh',\n",
       " 'dreamer',\n",
       " 'enjoined',\n",
       " 'retire',\n",
       " 'consulter',\n",
       " 'of',\n",
       " 'doctrines',\n",
       " 'Haziel',\n",
       " 'bags',\n",
       " 'Abez',\n",
       " 'modest',\n",
       " 'shepherds',\n",
       " 'breeches',\n",
       " 'crowned',\n",
       " 'Gibeonite',\n",
       " 'patriarch',\n",
       " 'foreseeing',\n",
       " 'Hirah',\n",
       " 'ornament',\n",
       " 'pangs',\n",
       " 'livest',\n",
       " 'proving',\n",
       " 'Shillemites',\n",
       " 'Every',\n",
       " 'hatest',\n",
       " 'railed',\n",
       " 'heave',\n",
       " 'vehement',\n",
       " 'Blastus',\n",
       " 'bats',\n",
       " 'Ezrahite',\n",
       " 'main',\n",
       " 'stank',\n",
       " 'scourgeth',\n",
       " 'meanest',\n",
       " 'gbnewby',\n",
       " 'indebted',\n",
       " 'air',\n",
       " 'vanities',\n",
       " 'Nahshon',\n",
       " 'educational',\n",
       " 'cruel',\n",
       " 'seas',\n",
       " 'wrestle',\n",
       " 'privily',\n",
       " 'forgiveness',\n",
       " 'slips',\n",
       " 'soft',\n",
       " 'lord',\n",
       " 'Defend',\n",
       " 'Rumah',\n",
       " 'SEND',\n",
       " 'Again',\n",
       " 'Ur',\n",
       " 'approvest',\n",
       " 'dyed',\n",
       " 'Benejaakan',\n",
       " 'saveth',\n",
       " 'oxen',\n",
       " 'seventh',\n",
       " 'voluntary',\n",
       " 'taste',\n",
       " 'fourteenth',\n",
       " 'wist',\n",
       " 'singleness',\n",
       " 'gained',\n",
       " 'another',\n",
       " '138',\n",
       " 'havock',\n",
       " 'Bethul',\n",
       " 'twentieth',\n",
       " 'lodge',\n",
       " 'weavest',\n",
       " 'Betah',\n",
       " 'fists',\n",
       " 'alter',\n",
       " 'eggs',\n",
       " 'setting',\n",
       " 'Dannah',\n",
       " 'mortality',\n",
       " 'malefactor',\n",
       " 'discontinue',\n",
       " 'sup',\n",
       " 'sparingly',\n",
       " 'borders',\n",
       " 'mattocks',\n",
       " 'dissolved',\n",
       " 'Sina',\n",
       " 'beauty',\n",
       " 'cuckow',\n",
       " 'vanity',\n",
       " 'ensuring',\n",
       " 'ouches',\n",
       " 'Ephratah',\n",
       " 'iniquity',\n",
       " 'bringing',\n",
       " 'oliveyard',\n",
       " 'equals',\n",
       " 'asp',\n",
       " 'Chinneroth',\n",
       " 'vagabond',\n",
       " 'rejoiced',\n",
       " 'straits',\n",
       " 'both',\n",
       " 'suddenly',\n",
       " 'avenger',\n",
       " 'Eliphal',\n",
       " 'Mishraites',\n",
       " 'Arabians',\n",
       " 'Truth',\n",
       " 'buttocks',\n",
       " 'gods',\n",
       " 'withs',\n",
       " 'compasseth',\n",
       " 'bestow',\n",
       " 'join',\n",
       " 'Especially',\n",
       " 'crossway',\n",
       " 'tabernacle',\n",
       " 'heretofore',\n",
       " 'wedge',\n",
       " 'yokefellow',\n",
       " 'petitions',\n",
       " 'relief',\n",
       " 'dwelling',\n",
       " 'Alammelech',\n",
       " 'mockings',\n",
       " 'idolaters',\n",
       " 'Bimhal',\n",
       " 'beginnest',\n",
       " 'determine',\n",
       " 'signed',\n",
       " 'continuance',\n",
       " 'unwalled',\n",
       " 'Toi',\n",
       " 'Amaziah',\n",
       " 'operations',\n",
       " 'Avims',\n",
       " 'Helkath',\n",
       " 'wasting',\n",
       " 'simplicity',\n",
       " 'hidden',\n",
       " 'Ashchenaz',\n",
       " 'opened',\n",
       " 'bloody',\n",
       " 'THOSE',\n",
       " 'Unto',\n",
       " 'payment',\n",
       " 'chode',\n",
       " 'blotteth',\n",
       " 'Gashmu',\n",
       " 'fold',\n",
       " 'Standing',\n",
       " 'recompence',\n",
       " 'journey',\n",
       " 'clamorous',\n",
       " 'Vanilla',\n",
       " 'Lasha',\n",
       " 'faithfully',\n",
       " 'Deceive',\n",
       " 'restrained',\n",
       " 'ship',\n",
       " 'breaches',\n",
       " 'vigilant',\n",
       " 'emerald',\n",
       " 'minished',\n",
       " 'JESUS',\n",
       " 'roebucks',\n",
       " 'prolong',\n",
       " 'Abijam',\n",
       " 'wave',\n",
       " 'comforteth',\n",
       " 'riddance',\n",
       " 'Information',\n",
       " 'exceedeth',\n",
       " 'hosts',\n",
       " 'hires',\n",
       " 'claim',\n",
       " 'worshippeth',\n",
       " 'separateth',\n",
       " 'sect',\n",
       " 'Avoiding',\n",
       " 'temperate',\n",
       " 'dissolvest',\n",
       " 'Sepharad',\n",
       " 'stocks',\n",
       " 'anger',\n",
       " 'multiplied',\n",
       " 'Forbidding',\n",
       " 'eateth',\n",
       " 'Sherebiah',\n",
       " 'conversant',\n",
       " 'Fearfulness',\n",
       " 'MENE',\n",
       " 'dainties',\n",
       " 'Eder',\n",
       " 'methods',\n",
       " 'Izeharites',\n",
       " 'Parshandatha',\n",
       " 'yieldeth',\n",
       " 'Zethan',\n",
       " 'crimson',\n",
       " 'Diana',\n",
       " 'psalmist',\n",
       " 'transcription',\n",
       " 'Likewise',\n",
       " 'bare',\n",
       " 'herd',\n",
       " 'gardens',\n",
       " 'Aholibah',\n",
       " 'forswear',\n",
       " 'breakest',\n",
       " 'visited',\n",
       " 'arrayed',\n",
       " 'unadvisedly',\n",
       " 'saddle',\n",
       " 'remained',\n",
       " 'cheek',\n",
       " 'Ishbah',\n",
       " 'himself',\n",
       " 'Shalt',\n",
       " 'Canaanitess',\n",
       " 'famished',\n",
       " 'instruction',\n",
       " 'turtles',\n",
       " 'magnified',\n",
       " 'Laadan',\n",
       " 'Jeriah',\n",
       " 'Incline',\n",
       " 'railing',\n",
       " 'Migron',\n",
       " 'beforehand',\n",
       " 'itching',\n",
       " 'lucre',\n",
       " 'Bishlam',\n",
       " 'Bamah',\n",
       " 'DIRECT',\n",
       " 'Eker',\n",
       " 'crumbs',\n",
       " 'homers',\n",
       " 'giants',\n",
       " 'innocent',\n",
       " 'Judaea',\n",
       " 'Passover',\n",
       " 'Simeonites',\n",
       " 'exhorting',\n",
       " 'Reubenite',\n",
       " 'Justus',\n",
       " 'Kanah',\n",
       " 'Olives',\n",
       " 'graffed',\n",
       " 'wellspring',\n",
       " 'wonder',\n",
       " 'extent',\n",
       " 'UNTO',\n",
       " 'disappoint',\n",
       " 'experiment',\n",
       " 'needs',\n",
       " 'leaned',\n",
       " 'fitted',\n",
       " 'mustereth',\n",
       " 'mercyseat',\n",
       " 'Uncover',\n",
       " 'scroll',\n",
       " 'sailors',\n",
       " 'honourest',\n",
       " 'communication',\n",
       " 'Idumaea',\n",
       " 'Pekahiah',\n",
       " 'adversaries',\n",
       " 'Laying',\n",
       " 'Lamech',\n",
       " 'do',\n",
       " 'commend',\n",
       " 'thereby',\n",
       " 'conveniently',\n",
       " 'stanched',\n",
       " 'takest',\n",
       " 'Shamed',\n",
       " '70',\n",
       " 'beacon',\n",
       " 'Philippi',\n",
       " 'heavens',\n",
       " 'Aminadab',\n",
       " 'victory',\n",
       " 'rip',\n",
       " 'gnashed',\n",
       " 'With',\n",
       " 'Abimael',\n",
       " 'Fill',\n",
       " 'shore',\n",
       " 'Meribah',\n",
       " 'Muppim',\n",
       " 'sacrifice',\n",
       " 'Or',\n",
       " 'appertained',\n",
       " 'fewest',\n",
       " 'Seer',\n",
       " 'hardly',\n",
       " 'Ebronah',\n",
       " 'straitened',\n",
       " 'ministereth',\n",
       " 'revellings',\n",
       " 'compound',\n",
       " 'Mishal',\n",
       " 'narrowed',\n",
       " 'getteth',\n",
       " 'disobedience',\n",
       " 'ravens',\n",
       " 'confession',\n",
       " 'cases',\n",
       " 'Thirty',\n",
       " 'accuseth',\n",
       " 'burying',\n",
       " 'building',\n",
       " 'daytime',\n",
       " 'plumbline',\n",
       " 'north',\n",
       " 'Kelita',\n",
       " 'Maharai',\n",
       " 'Helez',\n",
       " 'Anethothite',\n",
       " 'husbandry',\n",
       " 'Melchishua',\n",
       " 'goest',\n",
       " 'Shebam',\n",
       " 'Were',\n",
       " 'occupy',\n",
       " 'bath',\n",
       " 'enlarge',\n",
       " 'Herodias',\n",
       " 'resurrection',\n",
       " 'status',\n",
       " 'Hearing',\n",
       " 'diggeth',\n",
       " 'close',\n",
       " 'hanging',\n",
       " 'smite',\n",
       " 'helps',\n",
       " 'overcometh',\n",
       " 'account',\n",
       " 'applicable',\n",
       " 'foundest',\n",
       " 'Oboth',\n",
       " 'Elul',\n",
       " 'blackness',\n",
       " 'payed',\n",
       " 'terms',\n",
       " 'Separate',\n",
       " 'Jabesh',\n",
       " 'Arad',\n",
       " 'resort',\n",
       " 'Zatthu',\n",
       " 'bewrayeth',\n",
       " 'armour',\n",
       " 'cheerfully',\n",
       " 'queen',\n",
       " 'Gaal',\n",
       " 'restoreth',\n",
       " 'goad',\n",
       " 'Eunice',\n",
       " 'paces',\n",
       " 'Ikkesh',\n",
       " 'Hashmonah',\n",
       " 'odours',\n",
       " 'Ashvath',\n",
       " 'Horims',\n",
       " 'entangle',\n",
       " 'lo',\n",
       " 'copy',\n",
       " 'breathing',\n",
       " 'much',\n",
       " 'wrathful',\n",
       " 'sounding',\n",
       " 'givest',\n",
       " 'devote',\n",
       " 'peeled',\n",
       " 'partakers',\n",
       " 'oil',\n",
       " 'Berechiah',\n",
       " 'bondwoman',\n",
       " 'tombs',\n",
       " 'apparelled',\n",
       " 'Gihon',\n",
       " 'Zia',\n",
       " 'Elnathan',\n",
       " 'Salu',\n",
       " 'worshipped',\n",
       " 'trembled',\n",
       " 'decrease',\n",
       " 'handling',\n",
       " 'therein',\n",
       " 'duke',\n",
       " 'greenness',\n",
       " 'tribulation',\n",
       " 'Manoah',\n",
       " 'forewarn',\n",
       " 'being',\n",
       " 'root',\n",
       " 'Rechah',\n",
       " 'journeys',\n",
       " 'handywork',\n",
       " 'caught',\n",
       " 'Shimeam',\n",
       " 'kite',\n",
       " 'Milalai',\n",
       " 'Zareathites',\n",
       " 'eschewed',\n",
       " 'playing',\n",
       " 'hearkened',\n",
       " 'non',\n",
       " 'heritage',\n",
       " 'graving',\n",
       " 'cripple',\n",
       " 'Ziddim',\n",
       " 'Ham',\n",
       " 'Bathshua',\n",
       " 'Reubenites',\n",
       " 'garner',\n",
       " 'uncover',\n",
       " 'buyer',\n",
       " 'cousins',\n",
       " 'Nazarites',\n",
       " 'serpent',\n",
       " 'subjection',\n",
       " 'undertook',\n",
       " '109',\n",
       " 'communion',\n",
       " '99',\n",
       " 'ovens',\n",
       " 'rendereth',\n",
       " 'besiege',\n",
       " 'Teaching',\n",
       " 'Naamah',\n",
       " 'Sibbechai',\n",
       " 'rubies',\n",
       " 'lusty',\n",
       " 'slowly',\n",
       " 'marrieth',\n",
       " 'Dalaiah',\n",
       " 'possession',\n",
       " 'Aristobulus',\n",
       " 'lookest',\n",
       " 'preserved',\n",
       " 'Ekron',\n",
       " 'Horem',\n",
       " 'defrauded',\n",
       " 'Apollos',\n",
       " 'distraction',\n",
       " 'determination',\n",
       " 'Last',\n",
       " 'relied',\n",
       " 'unworthily',\n",
       " 'complying',\n",
       " 'Aharah',\n",
       " 'chameleon',\n",
       " 'titles',\n",
       " 'Answer',\n",
       " 'kisses',\n",
       " 'ordereth',\n",
       " 'Rivers',\n",
       " 'Haggiah',\n",
       " 'deserving',\n",
       " 'earthy',\n",
       " 'amazement',\n",
       " 'Atroth',\n",
       " '144',\n",
       " 'conquerors',\n",
       " 'Kinah',\n",
       " '39',\n",
       " 'flocks',\n",
       " 'munitions',\n",
       " 'bears',\n",
       " 'awaketh',\n",
       " 'naves',\n",
       " 'sunder',\n",
       " 'contemptuously',\n",
       " 'swooned',\n",
       " 'deletions',\n",
       " 'bricks',\n",
       " 'humbly',\n",
       " 'withstand',\n",
       " 'taskmasters',\n",
       " 'joinings',\n",
       " 'OF',\n",
       " 'unblameable',\n",
       " 'nourishment',\n",
       " 'Seirath',\n",
       " 'chests',\n",
       " 'chargedst',\n",
       " 'weighing',\n",
       " 'pilgrims',\n",
       " 'feathers',\n",
       " 'Jotbah',\n",
       " 'lap',\n",
       " 'Lament',\n",
       " 'hiding',\n",
       " 'presseth',\n",
       " 'ice',\n",
       " 'lifter',\n",
       " 'durst',\n",
       " 'liquors',\n",
       " 'confused',\n",
       " 'rowing',\n",
       " 'Anetothite',\n",
       " 'violence',\n",
       " 'plucketh',\n",
       " 'sardine',\n",
       " 'task',\n",
       " 'bullocks',\n",
       " 'clearness',\n",
       " 'attired',\n",
       " 'Hazargaddah',\n",
       " 'Shema',\n",
       " 'dance',\n",
       " 'appetite',\n",
       " 'confessing',\n",
       " 'Patrobas',\n",
       " 'Rephidim',\n",
       " 'Bul',\n",
       " 'They',\n",
       " 'Fulfil',\n",
       " 'Whatsoever',\n",
       " 'bloweth',\n",
       " 'disdained',\n",
       " 'awe',\n",
       " 'aloof',\n",
       " 'writest',\n",
       " 'dispersed',\n",
       " 'storehouses',\n",
       " 'prised',\n",
       " 'ancients',\n",
       " 'alloweth',\n",
       " 'reformed',\n",
       " 'Ain',\n",
       " 'Iniquities',\n",
       " 'dearly',\n",
       " 'Peruda',\n",
       " 'sheepcotes',\n",
       " 'castaway',\n",
       " 'loved',\n",
       " 'slandereth',\n",
       " 'Jeshishai',\n",
       " 'banded',\n",
       " 'powders',\n",
       " 'Public',\n",
       " 'bed',\n",
       " 'Ramah',\n",
       " 'worthily',\n",
       " 'rattling',\n",
       " 'Rehum',\n",
       " 'profaneth',\n",
       " 'purchased',\n",
       " 'thereout',\n",
       " 'napkin',\n",
       " 'Ginnethon',\n",
       " 'Lubim',\n",
       " 'remembereth',\n",
       " 'cockatrice',\n",
       " 'count',\n",
       " 'tumultuous',\n",
       " 'awake',\n",
       " 'Render',\n",
       " 'courteously',\n",
       " 'freeman',\n",
       " 'keepeth',\n",
       " 'force',\n",
       " 'Astaroth',\n",
       " 'guilty',\n",
       " 'Benaiah',\n",
       " 'circumcised',\n",
       " 'quarrel',\n",
       " 'overturned',\n",
       " 'Terms',\n",
       " 'envies',\n",
       " 'loseth',\n",
       " 'cymbals',\n",
       " 'Arumah',\n",
       " 'org',\n",
       " 'Gilalai',\n",
       " 'Mahavite',\n",
       " 'purposed',\n",
       " 'Harden',\n",
       " 'stature',\n",
       " 'Ashbel',\n",
       " 'revolted',\n",
       " 'flinty',\n",
       " 'Mizar',\n",
       " 'Jozabad',\n",
       " 'plague',\n",
       " 'grieveth',\n",
       " 'occasions',\n",
       " 'affected',\n",
       " 'waxen',\n",
       " 'Joezer',\n",
       " 'beheld',\n",
       " 'Gezer',\n",
       " 'knew',\n",
       " 'Blind',\n",
       " 'kinsmen',\n",
       " 'ditch',\n",
       " 'sown',\n",
       " 'aside',\n",
       " 'sheets',\n",
       " 'repliest',\n",
       " 'Manahath',\n",
       " 'Puhites',\n",
       " 'souls',\n",
       " 'stayed',\n",
       " 'concerneth',\n",
       " 'assaulted',\n",
       " 'Boaz',\n",
       " 'vials',\n",
       " 'meddle',\n",
       " 'Parosh',\n",
       " 'promises',\n",
       " 'Junia',\n",
       " 'Study',\n",
       " 'Joiada',\n",
       " 'Zemaraim',\n",
       " '148',\n",
       " 'goldsmith',\n",
       " 'fanners',\n",
       " 'pardoneth',\n",
       " 'glutton',\n",
       " 'downward',\n",
       " 'glorifieth',\n",
       " 'Calling',\n",
       " 'today',\n",
       " 'hatefully',\n",
       " 'beasts',\n",
       " 'Holy',\n",
       " 'remedy',\n",
       " 'dish',\n",
       " 'foxes',\n",
       " 'healing',\n",
       " 'increased',\n",
       " 'curse',\n",
       " 'Ointment',\n",
       " 'awaked',\n",
       " 'Cheran',\n",
       " 'weave',\n",
       " 'GREAT',\n",
       " 'chasing',\n",
       " 'Helon',\n",
       " 'Through',\n",
       " 'rot',\n",
       " 'Nahor',\n",
       " 'Manna',\n",
       " 'shrines',\n",
       " 'faileth',\n",
       " 'forests',\n",
       " 'Gittahhepher',\n",
       " 'branches',\n",
       " 'Lust',\n",
       " 'Nethaniah',\n",
       " 'foreknow',\n",
       " 'gifts',\n",
       " 'pools',\n",
       " 'Tremble',\n",
       " 'Chesalon',\n",
       " 'Revenue',\n",
       " 'land',\n",
       " 'wisdom',\n",
       " 'particular',\n",
       " 'Issachar',\n",
       " 'assayed',\n",
       " 'Dimnah',\n",
       " 'satyrs',\n",
       " 'Shuhamites',\n",
       " 'Bethgader',\n",
       " 'wearieth',\n",
       " 'withstood',\n",
       " 'contended',\n",
       " 'officer',\n",
       " 'unicorn',\n",
       " 'Baalgad',\n",
       " 'Assemble',\n",
       " 'Miamin',\n",
       " 'wept',\n",
       " 'repay',\n",
       " 'sackbut',\n",
       " 'Came',\n",
       " 'familiar',\n",
       " 'Draw',\n",
       " 'Amramites',\n",
       " 'miserably',\n",
       " 'inn',\n",
       " 'parable',\n",
       " 'Grant',\n",
       " 'Atarah',\n",
       " 'Ezem',\n",
       " 'Abstain',\n",
       " 'process',\n",
       " 'Salute',\n",
       " 'Jiphthahel',\n",
       " 'extinct',\n",
       " 'Bozez',\n",
       " 'prosperously',\n",
       " 'desiring',\n",
       " 'sirs',\n",
       " 'firstling',\n",
       " 'Ramathlehi',\n",
       " 'distil',\n",
       " 'presses',\n",
       " 'Pethor',\n",
       " 'cupbearer',\n",
       " 'revilings',\n",
       " 'encamped',\n",
       " 'kind',\n",
       " 'Wealth',\n",
       " 'forasmuch',\n",
       " 'consentedst',\n",
       " 'profaneness',\n",
       " 'strangled',\n",
       " 'midwives',\n",
       " 'spouses',\n",
       " 'astrologer',\n",
       " 'difference',\n",
       " 'millstone',\n",
       " 'Shomer',\n",
       " 'Tema',\n",
       " 'virtue',\n",
       " 'homer',\n",
       " 'talk',\n",
       " 'keepest',\n",
       " 'Duke',\n",
       " 'Multitudes',\n",
       " 'pare',\n",
       " 'Man',\n",
       " 'serjeants',\n",
       " 'Arab',\n",
       " 'addresses',\n",
       " 'Adoniram',\n",
       " 'your',\n",
       " 'joint',\n",
       " 'beards',\n",
       " 'knees',\n",
       " 'defend',\n",
       " 'fell',\n",
       " 'Gittites',\n",
       " 'challengeth',\n",
       " 'Jehiah',\n",
       " 'wars',\n",
       " 'letteth',\n",
       " 'vilely',\n",
       " 'Agree',\n",
       " 'Say',\n",
       " 'troublest',\n",
       " 'among',\n",
       " 'screech',\n",
       " 'revenged',\n",
       " 'forgetting',\n",
       " 'Joiakim',\n",
       " 'Clouds',\n",
       " 'Achaia',\n",
       " 'Seneh',\n",
       " 'Whensoever',\n",
       " 'contending',\n",
       " 'worldly',\n",
       " 'createth',\n",
       " 'items',\n",
       " 'deals',\n",
       " 'Divide',\n",
       " 'Barley',\n",
       " 'Epaphras',\n",
       " 'Raamah',\n",
       " 'banished',\n",
       " 'Zelek',\n",
       " 'pipe',\n",
       " 'Euphrates',\n",
       " 'Josabad',\n",
       " 'bushel',\n",
       " 'Telmelah',\n",
       " 'fewness',\n",
       " 'Sheleph',\n",
       " 'dough',\n",
       " 'candles',\n",
       " 'answering',\n",
       " 'ospray',\n",
       " 'intreaty',\n",
       " 'declaring',\n",
       " 'Iscariot',\n",
       " 'engraven',\n",
       " 'divorce',\n",
       " 'childbearing',\n",
       " 'wormwood',\n",
       " 'correcteth',\n",
       " 'breakings',\n",
       " 'spoiler',\n",
       " 'arrived',\n",
       " 'hatred',\n",
       " 'Jehoshabeath',\n",
       " 'forborn',\n",
       " 'uncircumcised',\n",
       " 'roareth',\n",
       " 'reigneth',\n",
       " 'Hazarmaveth',\n",
       " 'Zanoah',\n",
       " 'seventeenth',\n",
       " 'cleft',\n",
       " 'Philistia',\n",
       " 'councils',\n",
       " 'haling',\n",
       " 'Rabshakeh',\n",
       " 'manifestation',\n",
       " 'pitchers',\n",
       " 'cook',\n",
       " 'FOR',\n",
       " 'dogs',\n",
       " 'Baalah',\n",
       " 'grudgingly',\n",
       " 'Peradventure',\n",
       " 'advised',\n",
       " 'revelation',\n",
       " 'threatened',\n",
       " 'Adar',\n",
       " 'betrayest',\n",
       " 'Persecutions',\n",
       " 'office',\n",
       " 'furnished',\n",
       " 'ends',\n",
       " 'blackish',\n",
       " 'drink',\n",
       " 'crush',\n",
       " 'remember',\n",
       " 'Abihu',\n",
       " 'Joseph',\n",
       " 'just',\n",
       " 'wallow',\n",
       " 'Migdol',\n",
       " 'consecrated',\n",
       " 'sifted',\n",
       " 'Hosah',\n",
       " 'Semei',\n",
       " 'rushed',\n",
       " 'false',\n",
       " 'scum',\n",
       " 'Thamah',\n",
       " 'following',\n",
       " 'intents',\n",
       " 'Shuham',\n",
       " 'skill',\n",
       " 'cab',\n",
       " 'blindness',\n",
       " 'lattice',\n",
       " 'directly',\n",
       " 'comeliness',\n",
       " 'drinkers',\n",
       " 'beckoning',\n",
       " 'Bethzur',\n",
       " 'Tidal',\n",
       " 'ANYTHING',\n",
       " 'intellectual',\n",
       " 'Pispah',\n",
       " 'Lodebar',\n",
       " 'marvels',\n",
       " 'Naham',\n",
       " 'bees',\n",
       " 'Barbarian',\n",
       " 'strove',\n",
       " 'convocation',\n",
       " 'laughing',\n",
       " 'Destruction',\n",
       " ...]"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(set(tokenizer.tokenize(text)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}