dlnd_tv_script_generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TV Script Generation\n",
    "In this project, you'll generate your own [Simpsons](https://en.wikipedia.org/wiki/The_Simpsons) TV scripts using RNNs.  You'll be using part of the [Simpsons dataset](https://www.kaggle.com/wcukierski/the-simpsons-by-the-data) of scripts from 27 seasons.  The Neural Network you'll build will generate a new TV script for a scene at [Moe's Tavern](https://simpsonswiki.com/wiki/Moe's_Tavern).\n",
    "## Get the Data\n",
    "The data is already provided for you.  You'll be using a subset of the original dataset.  It consists of only the scenes in Moe's Tavern.  This doesn't include other versions of the tavern, like \"Moe's Cavern\", \"Flaming Moe's\", \"Uncle Moe's Family Feed-Bag\", etc.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import helper\n",
    "\n",
    "data_dir = './data/simpsons/moes_tavern_lines.txt'\n",
    "text = helper.load_data(data_dir)\n",
    "# Ignore notice, since we don't use it for analysing the data\n",
    "text = text[81:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the Data\n",
    "Play around with `view_sentence_range` to view different parts of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset Stats\n",
      "Roughly the number of unique words: 11492\n",
      "Number of scenes: 262\n",
      "Average number of sentences in each scene: 15.248091603053435\n",
      "Number of lines: 4257\n",
      "Average number of words in each line: 11.50434578341555\n",
      "\n",
      "The sentences 0 to 10:\n",
      "Moe_Szyslak: (INTO PHONE) Moe's Tavern. Where the elite meet to drink.\n",
      "Bart_Simpson: Eh, yeah, hello, is Mike there? Last name, Rotch.\n",
      "Moe_Szyslak: (INTO PHONE) Hold on, I'll check. (TO BARFLIES) Mike Rotch. Mike Rotch. Hey, has anybody seen Mike Rotch, lately?\n",
      "Moe_Szyslak: (INTO PHONE) Listen you little puke. One of these days I'm gonna catch you, and I'm gonna carve my name on your back with an ice pick.\n",
      "Moe_Szyslak: What's the matter Homer? You're not your normal effervescent self.\n",
      "Homer_Simpson: I got my problems, Moe. Give me another one.\n",
      "Moe_Szyslak: Homer, hey, you should not drink to forget your problems.\n",
      "Barney_Gumble: Yeah, you should only drink to enhance your social skills.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "view_sentence_range = (0, 10)\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import numpy as np\n",
    "\n",
    "print('Dataset Stats')\n",
    "print('Roughly the number of unique words: {}'.format(len({word: None for word in text.split()})))\n",
    "scenes = text.split('\\n\\n')\n",
    "print('Number of scenes: {}'.format(len(scenes)))\n",
    "sentence_count_scene = [scene.count('\\n') for scene in scenes]\n",
    "print('Average number of sentences in each scene: {}'.format(np.average(sentence_count_scene)))\n",
    "\n",
    "sentences = [sentence for scene in scenes for sentence in scene.split('\\n')]\n",
    "print('Number of lines: {}'.format(len(sentences)))\n",
    "word_count_sentence = [len(sentence.split()) for sentence in sentences]\n",
    "print('Average number of words in each line: {}'.format(np.average(word_count_sentence)))\n",
    "\n",
    "print()\n",
    "print('The sentences {} to {}:'.format(*view_sentence_range))\n",
    "print('\\n'.join(text.split('\\n')[view_sentence_range[0]:view_sentence_range[1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implement Preprocessing Functions\n",
    "The first thing to do to any dataset is preprocessing.  Implement the following preprocessing functions below:\n",
    "- Lookup Table\n",
    "- Tokenize Punctuation\n",
    "\n",
    "### Lookup Table\n",
    "To create a word embedding, you first need to transform the words to ids.  In this function, create two dictionaries:\n",
    "- Dictionary to go from the words to an id, we'll call `vocab_to_int`\n",
    "- Dictionary to go from the id to word, we'll call `int_to_vocab`\n",
    "\n",
    "Return these dictionaries in the following tuple `(vocab_to_int, int_to_vocab)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import problem_unittests as tests\n",
    "\n",
    "def create_lookup_tables(text):\n",
    "    \"\"\"\n",
    "    Create lookup tables for vocabulary\n",
    "    :param text: The text of tv scripts split into words\n",
    "    :return: A tuple of dicts (vocab_to_int, int_to_vocab)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    \n",
    "    vocab_to_int  = dict()\n",
    "    int_to_vocab = dict()\n",
    "    text = set(text)\n",
    "    \n",
    "    for index,word in enumerate(text):\n",
    "        vocab_to_int[word] = index\n",
    "        int_to_vocab[index] = word\n",
    "        \n",
    "    return vocab_to_int,int_to_vocab\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_create_lookup_tables(create_lookup_tables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenize Punctuation\n",
    "We'll be splitting the script into a word array using spaces as delimiters.  However, punctuations like periods and exclamation marks make it hard for the neural network to distinguish between the word \"bye\" and \"bye!\".\n",
    "\n",
    "Implement the function `token_lookup` to return a dict that will be used to tokenize symbols like \"!\" into \"||Exclamation_Mark||\".  Create a dictionary for the following symbols where the symbol is the key and value is the token:\n",
    "- Period ( . )\n",
    "- Comma ( , )\n",
    "- Quotation Mark ( \" )\n",
    "- Semicolon ( ; )\n",
    "- Exclamation mark ( ! )\n",
    "- Question mark ( ? )\n",
    "- Left Parentheses ( ( )\n",
    "- Right Parentheses ( ) )\n",
    "- Dash ( -- )\n",
    "- Return ( \\n )\n",
    "\n",
    "This dictionary will be used to token the symbols and add the delimiter (space) around it.  This separates the symbols as it's own word, making it easier for the neural network to predict on the next word. Make sure you don't use a token that could be confused as a word. Instead of using the token \"dash\", try using something like \"||dash||\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def token_lookup():\n",
    "    \"\"\"\n",
    "    Generate a dict to turn punctuation into a token.\n",
    "    :return: Tokenize dictionary where the key is the punctuation and the value is the token\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    token_dictionary = dict()\n",
    "    token_dictionary[\".\"] = \"||period||\"\n",
    "    token_dictionary[\",\"] = \"||comma||\"\n",
    "    token_dictionary[\";\"] = \"||quotation||\"\n",
    "    token_dictionary[\"!\"] = \"||exclamation||\"\n",
    "    token_dictionary[\"?\"] = \"||question||\"\n",
    "    token_dictionary[\"(\"] = \"||left_parentheses||\"\n",
    "    token_dictionary[\")\"] = \"||right_parentheses||\"\n",
    "    token_dictionary[\"--\"] = \"||dash||\"\n",
    "    token_dictionary[\"\\n\"] = \"||return||\"\n",
    "    token_dictionary['\"'] = \"||quotation||\"\n",
    "\n",
    "    return token_dictionary\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_tokenize(token_lookup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess all the data and save it\n",
    "Running the code cell below will preprocess all the data and save it to file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "# Preprocess Training, Validation, and Testing Data\n",
    "helper.preprocess_and_save_data(data_dir, token_lookup, create_lookup_tables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check Point\n",
    "This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import helper\n",
    "import numpy as np\n",
    "import problem_unittests as tests\n",
    "import time\n",
    "\n",
    "int_text, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the Neural Network\n",
    "You'll build the components necessary to build a RNN by implementing the following functions below:\n",
    "- get_inputs\n",
    "- get_init_cell\n",
    "- get_embed\n",
    "- build_rnn\n",
    "- build_nn\n",
    "- get_batches\n",
    "\n",
    "### Check the Version of TensorFlow and Access to GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow Version: 1.1.0-rc2\n",
      "Default GPU Device: /gpu:0\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "from distutils.version import LooseVersion\n",
    "import warnings\n",
    "import tensorflow as tf\n",
    "\n",
    "# Check TensorFlow Version\n",
    "assert LooseVersion(tf.__version__) >= LooseVersion('1.0'), 'Please use TensorFlow version 1.0 or newer'\n",
    "print('TensorFlow Version: {}'.format(tf.__version__))\n",
    "\n",
    "# Check for a GPU\n",
    "if not tf.test.gpu_device_name():\n",
    "    warnings.warn('No GPU found. Please use a GPU to train your neural network.')\n",
    "else:\n",
    "    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input\n",
    "Implement the `get_inputs()` function to create TF Placeholders for the Neural Network.  It should create the following placeholders:\n",
    "- Input text placeholder named \"input\" using the [TF Placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder) `name` parameter.\n",
    "- Targets placeholder\n",
    "- Learning Rate placeholder\n",
    "\n",
    "Return the placeholders in the following tuple `(Input, Targets, LearningRate)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def get_inputs():\n",
    "    \"\"\"\n",
    "    Create TF Placeholders for input, targets, and learning rate.\n",
    "    :return: Tuple (input, targets, learning rate)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    Input = tf.placeholder(tf.int32,[None,None],name=\"input\")\n",
    "    Targets = tf.placeholder(tf.int32,[None,None],name = \"Targets\")\n",
    "    LearningRate = tf.placeholder(tf.float32,name =\"LearningRate\")\n",
    "    return Input,Targets,LearningRate\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_get_inputs(get_inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build RNN Cell and Initialize\n",
    "Stack one or more [`BasicLSTMCells`](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/BasicLSTMCell) in a [`MultiRNNCell`](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/MultiRNNCell).\n",
    "- The Rnn size should be set using `rnn_size`\n",
    "- Initalize Cell State using the MultiRNNCell's [`zero_state()`](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/MultiRNNCell#zero_state) function\n",
    "    - Apply the name \"initial_state\" to the initial state using [`tf.identity()`](https://www.tensorflow.org/api_docs/python/tf/identity)\n",
    "\n",
    "Return the cell and initial state in the following tuple `(Cell, InitialState)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def get_init_cell(batch_size, rnn_size):\n",
    "    \"\"\"\n",
    "    Create an RNN Cell and initialize it.\n",
    "    :param batch_size: Size of batches\n",
    "    :param rnn_size: Size of RNNs\n",
    "    :return: Tuple (cell, initialize state)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    cells_list = [tf.contrib.rnn.BasicLSTMCell(rnn_size) for _ in range(2)]\n",
    "    rnn_layer =  tf.contrib.rnn.MultiRNNCell(cells_list)\n",
    "    initial_state = rnn_layer.zero_state(batch_size,tf.float32)\n",
    "    initial_state = tf.identity(initial_state,\"initial_state\")\n",
    "    \n",
    "    return rnn_layer,initial_state\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_get_init_cell(get_init_cell)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Embedding\n",
    "Apply embedding to `input_data` using TensorFlow.  Return the embedded sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"Placeholder:0\", shape=(50, 5), dtype=int32)\n",
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def get_embed(input_data, vocab_size, embed_dim):\n",
    "    \"\"\"\n",
    "    Create embedding for <input_data>.\n",
    "    :param input_data: TF placeholder for text input.\n",
    "    :param vocab_size: Number of words in vocabulary.\n",
    "    :param embed_dim: Number of embedding dimensions\n",
    "    :return: Embedded input.\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    print(input_data)\n",
    "    embedding = tf.Variable(tf.random_uniform([vocab_size,embed_dim],-1,1))\n",
    "    embed   =  tf.nn.embedding_lookup(embedding,input_data)\n",
    "    \n",
    "    return embed\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_get_embed(get_embed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build RNN\n",
    "You created a RNN Cell in the `get_init_cell()` function.  Time to use the cell to create a RNN.\n",
    "- Build the RNN using the [`tf.nn.dynamic_rnn()`](https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn)\n",
    " - Apply the name \"final_state\" to the final state using [`tf.identity()`](https://www.tensorflow.org/api_docs/python/tf/identity)\n",
    "\n",
    "Return the outputs and final_state state in the following tuple `(Outputs, FinalState)` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def build_rnn(cell, inputs):\n",
    "    \"\"\"\n",
    "    Create a RNN using a RNN Cell\n",
    "    :param cell: RNN Cell\n",
    "    :param inputs: Input text data\n",
    "    :return: Tuple (Outputs, Final State)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    rnn_outputs, rnn_state = tf.nn.dynamic_rnn(cell,inputs,dtype = tf.float32)\n",
    "    rnn_state = tf.identity(rnn_state,\"final_state\")\n",
    "    return rnn_outputs,rnn_state\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_build_rnn(build_rnn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the Neural Network\n",
    "Apply the functions you implemented above to:\n",
    "- Apply embedding to `input_data` using your `get_embed(input_data, vocab_size, embed_dim)` function.\n",
    "- Build RNN using `cell` and your `build_rnn(cell, inputs)` function.\n",
    "- Apply a fully connected layer with a linear activation and `vocab_size` as the number of outputs.\n",
    "\n",
    "Return the logits and final state in the following tuple (Logits, FinalState) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"Placeholder:0\", shape=(128, 5), dtype=int32)\n",
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def build_nn(cell, rnn_size, input_data, vocab_size, embed_dim):\n",
    "    \"\"\"\n",
    "    Build part of the neural network\n",
    "    :param cell: RNN cell\n",
    "    :param rnn_size: Size of rnns\n",
    "    :param input_data: Input data\n",
    "    :param vocab_size: Vocabulary size\n",
    "    :param embed_dim: Number of embedding dimensions\n",
    "    :return: Tuple (Logits, FinalState)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    embedings = get_embed(input_data,vocab_size,embed_dim)\n",
    "    rnn_output,rnn_state = build_rnn(cell,embedings)\n",
    "    #reshape  rnn_output to be able to matmul\n",
    "    #rnn_output = tf.reshape(rnn_output,[-1,rnn_size])\n",
    "\n",
    "    \n",
    "    #fully_connected_weights = tf.Variable(tf.truncated_normal([rnn_size,vocab_size]))\n",
    "    #print(fully_connected_weights)\n",
    "    #fully_connected_bias = tf.Variable(tf.zeros(vocab_size))\n",
    "    #print(fully_connected_bias)\n",
    "    #logits = tf.add(tf.matmul(rnn_output,fully_connected_weights),fully_connected_bias)\n",
    "    #print(logits)\n",
    "    #logits = tf.cast(tf.reshape(logits,input_data.get_shape().as_list()+[vocab_size]) , tf.float32)#return logits to original shape\n",
    "    #print(logits)\n",
    "    logits = tf.contrib.layers.fully_connected(rnn_output, vocab_size, None)\n",
    "    return logits,rnn_state\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_build_nn(build_nn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batches\n",
    "Implement `get_batches` to create batches of input and targets using `int_text`.  The batches should be a Numpy array with the shape `(number of batches, 2, batch size, sequence length)`. Each batch contains two elements:\n",
    "- The first element is a single batch of **input** with the shape `[batch size, sequence length]`\n",
    "- The second element is a single batch of **targets** with the shape `[batch size, sequence length]`\n",
    "\n",
    "If you can't fill the last batch with enough data, drop the last batch.\n",
    "\n",
    "For exmple, `get_batches([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], 3, 2)` would return a Numpy array of the following:\n",
    "```\n",
    "[\n",
    "  # First Batch\n",
    "  [\n",
    "    # Batch of Input\n",
    "    [[ 1  2], [ 7  8], [13 14]]\n",
    "    # Batch of targets\n",
    "    [[ 2  3], [ 8  9], [14 15]]\n",
    "  ]\n",
    "\n",
    "  # Second Batch\n",
    "  [\n",
    "    # Batch of Input\n",
    "    [[ 3  4], [ 9 10], [15 16]]\n",
    "    # Batch of targets\n",
    "    [[ 4  5], [10 11], [16 17]]\n",
    "  ]\n",
    "\n",
    "  # Third Batch\n",
    "  [\n",
    "    # Batch of Input\n",
    "    [[ 5  6], [11 12], [17 18]]\n",
    "    # Batch of targets\n",
    "    [[ 6  7], [12 13], [18  1]]\n",
    "  ]\n",
    "]\n",
    "```\n",
    "\n",
    "Notice that the last target value in the last batch is the first input value of the first batch. In this case, `1`. This is a common technique used when creating sequence batches, although it is rather unintuitive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def get_batches(int_text, batch_size, seq_length):\n",
    "    \"\"\"\n",
    "    Return batches of input and target\n",
    "    :param int_text: Text with the words replaced by their ids\n",
    "    :param batch_size: The size of batch\n",
    "    :param seq_length: The length of sequence\n",
    "    :return: Batches as a Numpy array\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function   \n",
    "    batch_len = batch_size*seq_length\n",
    "    number_of_batches =  len(int_text)//batch_len\n",
    "    batches = np.zeros([number_of_batches, 2, batch_size, seq_length], dtype=np.int32)\n",
    "    # Keep only enough characters to make full batches\n",
    "    int_text =  np.array(int_text[:number_of_batches*batch_len])\n",
    "    # Reshape into n_seqs rows\n",
    "    int_text_matrix = int_text.reshape((batch_size,-1))\n",
    "    batch  = 0\n",
    "    \n",
    "    for n in range(0, int_text_matrix.shape[1], seq_length):\n",
    "        \n",
    "        # The features\n",
    "        x = int_text_matrix[:,n:n+seq_length]\n",
    "        y = np.zeros(x.shape)\n",
    "    \n",
    "        if batch != (number_of_batches -1):\n",
    "            y[:,:] = int_text_matrix[:,n+1:n+seq_length+1]\n",
    "        else:\n",
    "            #Generate every batch row individually wrapping around the dataset when neccesary\n",
    "            for i in range(batch_size):\n",
    "                index = (n*(i+1))+((i*seq_length)+1) #get the start index of every batch row\n",
    "                y[i,:] = int_text.take(range(index,index+seq_length),mode=\"wrap\")\n",
    "\n",
    "        batches[batch,0] = x\n",
    "        batches[batch,1] = y\n",
    "        batch+=1\n",
    "    return batches\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_get_batches(get_batches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network Training\n",
    "### Hyperparameters\n",
    "Tune the following parameters:\n",
    "\n",
    "- Set `num_epochs` to the number of epochs.\n",
    "- Set `batch_size` to the batch size.\n",
    "- Set `rnn_size` to the size of the RNNs.\n",
    "- Set `embed_dim` to the size of the embedding.\n",
    "- Set `seq_length` to the length of sequence.\n",
    "- Set `learning_rate` to the learning rate.\n",
    "- Set `show_every_n_batches` to the number of batches the neural network should print progress."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Number of Epochs\n",
    "num_epochs = 500\n",
    "# Batch Size\n",
    "batch_size = 128\n",
    "# RNN Size\n",
    "rnn_size = 300\n",
    "# Embedding Dimension Size\n",
    "embed_dim = 300\n",
    "# Sequence Length\n",
    "seq_length = 15\n",
    "# Learning Rate\n",
    "learning_rate = 0.001\n",
    "# Show stats for every n number of batches\n",
    "show_every_n_batches = 50\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "save_dir = './save'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the Graph\n",
    "Build the graph using the neural network you implemented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"input:0\", shape=(?, ?), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "from tensorflow.contrib import seq2seq\n",
    "\n",
    "train_graph = tf.Graph()\n",
    "with train_graph.as_default():\n",
    "    vocab_size = len(int_to_vocab)\n",
    "    input_text, targets, lr = get_inputs()\n",
    "    input_data_shape = tf.shape(input_text)\n",
    "    cell, initial_state = get_init_cell(input_data_shape[0], rnn_size)\n",
    "    logits, final_state = build_nn(cell, rnn_size, input_text, vocab_size, embed_dim)\n",
    "\n",
    "    # Probabilities for generating words\n",
    "    probs = tf.nn.softmax(logits, name='probs')\n",
    "\n",
    "    # Loss function\n",
    "    cost = seq2seq.sequence_loss(\n",
    "        logits,\n",
    "        targets,\n",
    "        tf.ones([input_data_shape[0], input_data_shape[1]]))\n",
    "\n",
    "    # Optimizer\n",
    "    optimizer = tf.train.AdamOptimizer(lr)\n",
    "\n",
    "    # Gradient Clipping\n",
    "    gradients = optimizer.compute_gradients(cost)\n",
    "    capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]\n",
    "    train_op = optimizer.apply_gradients(capped_gradients)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train\n",
    "Train the neural network on the preprocessed data.  If you have a hard time getting a good loss, check the [forms](https://discussions.udacity.com/) to see if anyone is having the same problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   0 Batch    0/35   train_loss = 8.821 epoch time = 0.180\n",
      "Epoch   1 Batch   15/35   train_loss = 6.124 epoch time = 0.456\n",
      "Epoch   2 Batch   30/35   train_loss = 6.029 epoch time = 0.887\n",
      "Epoch   4 Batch   10/35   train_loss = 6.136 epoch time = 0.314\n",
      "Epoch   5 Batch   25/35   train_loss = 5.950 epoch time = 0.742\n",
      "Epoch   7 Batch    5/35   train_loss = 5.982 epoch time = 0.172\n",
      "Epoch   8 Batch   20/35   train_loss = 6.000 epoch time = 0.640\n",
      "Epoch  10 Batch    0/35   train_loss = 5.879 epoch time = 0.030\n",
      "Epoch  11 Batch   15/35   train_loss = 5.902 epoch time = 0.459\n",
      "Epoch  12 Batch   30/35   train_loss = 5.735 epoch time = 0.894\n",
      "Epoch  14 Batch   10/35   train_loss = 5.789 epoch time = 0.314\n",
      "Epoch  15 Batch   25/35   train_loss = 5.562 epoch time = 0.745\n",
      "Epoch  17 Batch    5/35   train_loss = 5.566 epoch time = 0.171\n",
      "Epoch  18 Batch   20/35   train_loss = 5.593 epoch time = 0.602\n",
      "Epoch  20 Batch    0/35   train_loss = 5.432 epoch time = 0.029\n",
      "Epoch  21 Batch   15/35   train_loss = 5.464 epoch time = 0.459\n",
      "Epoch  22 Batch   30/35   train_loss = 5.304 epoch time = 0.887\n",
      "Epoch  24 Batch   10/35   train_loss = 5.312 epoch time = 0.316\n",
      "Epoch  25 Batch   25/35   train_loss = 5.015 epoch time = 0.744\n",
      "Epoch  27 Batch    5/35   train_loss = 4.997 epoch time = 0.175\n",
      "Epoch  28 Batch   20/35   train_loss = 4.970 epoch time = 0.608\n",
      "Epoch  30 Batch    0/35   train_loss = 4.859 epoch time = 0.029\n",
      "Epoch  31 Batch   15/35   train_loss = 4.825 epoch time = 0.473\n",
      "Epoch  32 Batch   30/35   train_loss = 4.659 epoch time = 0.925\n",
      "Epoch  34 Batch   10/35   train_loss = 4.708 epoch time = 0.314\n",
      "Epoch  35 Batch   25/35   train_loss = 4.432 epoch time = 0.743\n",
      "Epoch  37 Batch    5/35   train_loss = 4.429 epoch time = 0.172\n",
      "Epoch  38 Batch   20/35   train_loss = 4.386 epoch time = 0.604\n",
      "Epoch  40 Batch    0/35   train_loss = 4.325 epoch time = 0.031\n",
      "Epoch  41 Batch   15/35   train_loss = 4.301 epoch time = 0.483\n",
      "Epoch  42 Batch   30/35   train_loss = 4.147 epoch time = 0.943\n",
      "Epoch  44 Batch   10/35   train_loss = 4.175 epoch time = 0.320\n",
      "Epoch  45 Batch   25/35   train_loss = 3.938 epoch time = 0.777\n",
      "Epoch  47 Batch    5/35   train_loss = 3.963 epoch time = 0.172\n",
      "Epoch  48 Batch   20/35   train_loss = 3.882 epoch time = 0.610\n",
      "Epoch  50 Batch    0/35   train_loss = 3.927 epoch time = 0.032\n",
      "Epoch  51 Batch   15/35   train_loss = 3.832 epoch time = 0.493\n",
      "Epoch  52 Batch   30/35   train_loss = 3.680 epoch time = 0.947\n",
      "Epoch  54 Batch   10/35   train_loss = 3.716 epoch time = 0.338\n",
      "Epoch  55 Batch   25/35   train_loss = 3.500 epoch time = 0.781\n",
      "Epoch  57 Batch    5/35   train_loss = 3.508 epoch time = 0.180\n",
      "Epoch  58 Batch   20/35   train_loss = 3.409 epoch time = 0.632\n",
      "Epoch  60 Batch    0/35   train_loss = 3.489 epoch time = 0.030\n",
      "Epoch  61 Batch   15/35   train_loss = 3.406 epoch time = 0.480\n",
      "Epoch  62 Batch   30/35   train_loss = 3.259 epoch time = 0.924\n",
      "Epoch  64 Batch   10/35   train_loss = 3.263 epoch time = 0.331\n",
      "Epoch  65 Batch   25/35   train_loss = 3.090 epoch time = 0.802\n",
      "Epoch  67 Batch    5/35   train_loss = 3.077 epoch time = 0.186\n",
      "Epoch  68 Batch   20/35   train_loss = 2.995 epoch time = 0.644\n",
      "Epoch  70 Batch    0/35   train_loss = 3.058 epoch time = 0.031\n",
      "Epoch  71 Batch   15/35   train_loss = 2.936 epoch time = 0.486\n",
      "Epoch  72 Batch   30/35   train_loss = 2.792 epoch time = 0.922\n",
      "Epoch  74 Batch   10/35   train_loss = 2.862 epoch time = 0.327\n",
      "Epoch  75 Batch   25/35   train_loss = 2.672 epoch time = 0.773\n",
      "Epoch  77 Batch    5/35   train_loss = 2.674 epoch time = 0.178\n",
      "Epoch  78 Batch   20/35   train_loss = 2.583 epoch time = 0.625\n",
      "Epoch  80 Batch    0/35   train_loss = 2.599 epoch time = 0.031\n",
      "Epoch  81 Batch   15/35   train_loss = 2.488 epoch time = 0.478\n",
      "Epoch  82 Batch   30/35   train_loss = 2.400 epoch time = 0.926\n",
      "Epoch  84 Batch   10/35   train_loss = 2.453 epoch time = 0.330\n",
      "Epoch  85 Batch   25/35   train_loss = 2.342 epoch time = 0.782\n",
      "Epoch  87 Batch    5/35   train_loss = 2.282 epoch time = 0.182\n",
      "Epoch  88 Batch   20/35   train_loss = 2.167 epoch time = 0.632\n",
      "Epoch  90 Batch    0/35   train_loss = 2.283 epoch time = 0.031\n",
      "Epoch  91 Batch   15/35   train_loss = 2.139 epoch time = 0.483\n",
      "Epoch  92 Batch   30/35   train_loss = 2.132 epoch time = 0.934\n",
      "Epoch  94 Batch   10/35   train_loss = 2.089 epoch time = 0.330\n",
      "Epoch  95 Batch   25/35   train_loss = 1.964 epoch time = 0.780\n",
      "Epoch  97 Batch    5/35   train_loss = 1.878 epoch time = 0.181\n",
      "Epoch  98 Batch   20/35   train_loss = 1.782 epoch time = 0.629\n",
      "Epoch 100 Batch    0/35   train_loss = 1.864 epoch time = 0.031\n",
      "Epoch 101 Batch   15/35   train_loss = 1.780 epoch time = 0.482\n",
      "Epoch 102 Batch   30/35   train_loss = 1.766 epoch time = 0.933\n",
      "Epoch 104 Batch   10/35   train_loss = 1.724 epoch time = 0.329\n",
      "Epoch 105 Batch   25/35   train_loss = 1.641 epoch time = 0.782\n",
      "Epoch 107 Batch    5/35   train_loss = 1.610 epoch time = 0.181\n",
      "Epoch 108 Batch   20/35   train_loss = 1.483 epoch time = 0.632\n",
      "Epoch 110 Batch    0/35   train_loss = 1.566 epoch time = 0.031\n",
      "Epoch 111 Batch   15/35   train_loss = 1.464 epoch time = 0.482\n",
      "Epoch 112 Batch   30/35   train_loss = 1.460 epoch time = 0.932\n",
      "Epoch 114 Batch   10/35   train_loss = 1.443 epoch time = 0.331\n",
      "Epoch 115 Batch   25/35   train_loss = 1.367 epoch time = 0.780\n",
      "Epoch 117 Batch    5/35   train_loss = 1.309 epoch time = 0.181\n",
      "Epoch 118 Batch   20/35   train_loss = 1.204 epoch time = 0.631\n",
      "Epoch 120 Batch    0/35   train_loss = 1.314 epoch time = 0.031\n",
      "Epoch 121 Batch   15/35   train_loss = 1.167 epoch time = 0.480\n",
      "Epoch 122 Batch   30/35   train_loss = 1.154 epoch time = 0.932\n",
      "Epoch 124 Batch   10/35   train_loss = 1.130 epoch time = 0.331\n",
      "Epoch 125 Batch   25/35   train_loss = 1.056 epoch time = 0.781\n",
      "Epoch 127 Batch    5/35   train_loss = 1.019 epoch time = 0.180\n",
      "Epoch 128 Batch   20/35   train_loss = 0.988 epoch time = 0.631\n",
      "Epoch 130 Batch    0/35   train_loss = 1.069 epoch time = 0.030\n",
      "Epoch 131 Batch   15/35   train_loss = 0.973 epoch time = 0.481\n",
      "Epoch 132 Batch   30/35   train_loss = 1.066 epoch time = 0.931\n",
      "Epoch 134 Batch   10/35   train_loss = 0.975 epoch time = 0.330\n",
      "Epoch 135 Batch   25/35   train_loss = 0.865 epoch time = 0.781\n",
      "Epoch 137 Batch    5/35   train_loss = 0.824 epoch time = 0.181\n",
      "Epoch 138 Batch   20/35   train_loss = 0.763 epoch time = 0.632\n",
      "Epoch 140 Batch    0/35   train_loss = 0.845 epoch time = 0.030\n",
      "Epoch 141 Batch   15/35   train_loss = 0.791 epoch time = 0.485\n",
      "Epoch 142 Batch   30/35   train_loss = 0.823 epoch time = 0.931\n",
      "Epoch 144 Batch   10/35   train_loss = 0.793 epoch time = 0.331\n",
      "Epoch 145 Batch   25/35   train_loss = 0.726 epoch time = 0.780\n",
      "Epoch 147 Batch    5/35   train_loss = 0.687 epoch time = 0.181\n",
      "Epoch 148 Batch   20/35   train_loss = 0.656 epoch time = 0.633\n",
      "Epoch 150 Batch    0/35   train_loss = 0.712 epoch time = 0.030\n",
      "Epoch 151 Batch   15/35   train_loss = 0.651 epoch time = 0.481\n",
      "Epoch 152 Batch   30/35   train_loss = 0.638 epoch time = 0.931\n",
      "Epoch 154 Batch   10/35   train_loss = 0.638 epoch time = 0.330\n",
      "Epoch 155 Batch   25/35   train_loss = 0.568 epoch time = 0.781\n",
      "Epoch 157 Batch    5/35   train_loss = 0.569 epoch time = 0.182\n",
      "Epoch 158 Batch   20/35   train_loss = 0.549 epoch time = 0.658\n",
      "Epoch 160 Batch    0/35   train_loss = 0.632 epoch time = 0.032\n",
      "Epoch 161 Batch   15/35   train_loss = 0.556 epoch time = 0.485\n",
      "Epoch 162 Batch   30/35   train_loss = 0.575 epoch time = 0.931\n",
      "Epoch 164 Batch   10/35   train_loss = 0.564 epoch time = 0.330\n",
      "Epoch 165 Batch   25/35   train_loss = 0.521 epoch time = 0.782\n",
      "Epoch 167 Batch    5/35   train_loss = 0.523 epoch time = 0.181\n",
      "Epoch 168 Batch   20/35   train_loss = 0.444 epoch time = 0.633\n",
      "Epoch 170 Batch    0/35   train_loss = 0.543 epoch time = 0.030\n",
      "Epoch 171 Batch   15/35   train_loss = 0.480 epoch time = 0.480\n",
      "Epoch 172 Batch   30/35   train_loss = 0.478 epoch time = 0.931\n",
      "Epoch 174 Batch   10/35   train_loss = 0.466 epoch time = 0.331\n",
      "Epoch 175 Batch   25/35   train_loss = 0.413 epoch time = 0.780\n",
      "Epoch 177 Batch    5/35   train_loss = 0.419 epoch time = 0.180\n",
      "Epoch 178 Batch   20/35   train_loss = 0.381 epoch time = 0.631\n",
      "Epoch 180 Batch    0/35   train_loss = 0.418 epoch time = 0.030\n",
      "Epoch 181 Batch   15/35   train_loss = 0.405 epoch time = 0.480\n",
      "Epoch 182 Batch   30/35   train_loss = 0.389 epoch time = 0.930\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 184 Batch   10/35   train_loss = 0.370 epoch time = 0.331\n",
      "Epoch 185 Batch   25/35   train_loss = 0.343 epoch time = 0.781\n",
      "Epoch 187 Batch    5/35   train_loss = 0.359 epoch time = 0.180\n",
      "Epoch 188 Batch   20/35   train_loss = 0.329 epoch time = 0.631\n",
      "Epoch 190 Batch    0/35   train_loss = 0.361 epoch time = 0.031\n",
      "Epoch 191 Batch   15/35   train_loss = 0.369 epoch time = 0.482\n",
      "Epoch 192 Batch   30/35   train_loss = 0.344 epoch time = 0.932\n",
      "Epoch 194 Batch   10/35   train_loss = 0.343 epoch time = 0.331\n",
      "Epoch 195 Batch   25/35   train_loss = 0.328 epoch time = 0.783\n",
      "Epoch 197 Batch    5/35   train_loss = 0.331 epoch time = 0.181\n",
      "Epoch 198 Batch   20/35   train_loss = 0.295 epoch time = 0.632\n",
      "Epoch 200 Batch    0/35   train_loss = 0.329 epoch time = 0.030\n",
      "Epoch 201 Batch   15/35   train_loss = 0.317 epoch time = 0.480\n",
      "Epoch 202 Batch   30/35   train_loss = 0.294 epoch time = 0.929\n",
      "Epoch 204 Batch   10/35   train_loss = 0.293 epoch time = 0.346\n",
      "Epoch 205 Batch   25/35   train_loss = 0.274 epoch time = 0.811\n",
      "Epoch 207 Batch    5/35   train_loss = 0.289 epoch time = 0.183\n",
      "Epoch 208 Batch   20/35   train_loss = 0.262 epoch time = 0.632\n",
      "Epoch 210 Batch    0/35   train_loss = 0.295 epoch time = 0.030\n",
      "Epoch 211 Batch   15/35   train_loss = 0.297 epoch time = 0.461\n",
      "Epoch 212 Batch   30/35   train_loss = 0.276 epoch time = 0.908\n",
      "Epoch 214 Batch   10/35   train_loss = 0.271 epoch time = 0.341\n",
      "Epoch 215 Batch   25/35   train_loss = 0.255 epoch time = 0.769\n",
      "Epoch 217 Batch    5/35   train_loss = 0.272 epoch time = 0.178\n",
      "Epoch 218 Batch   20/35   train_loss = 0.245 epoch time = 0.622\n",
      "Epoch 220 Batch    0/35   train_loss = 0.275 epoch time = 0.031\n",
      "Epoch 221 Batch   15/35   train_loss = 0.275 epoch time = 0.462\n",
      "Epoch 222 Batch   30/35   train_loss = 0.259 epoch time = 0.893\n",
      "Epoch 224 Batch   10/35   train_loss = 0.258 epoch time = 0.316\n",
      "Epoch 225 Batch   25/35   train_loss = 0.241 epoch time = 0.788\n",
      "Epoch 227 Batch    5/35   train_loss = 0.253 epoch time = 0.173\n",
      "Epoch 228 Batch   20/35   train_loss = 0.231 epoch time = 0.605\n",
      "Epoch 230 Batch    0/35   train_loss = 0.263 epoch time = 0.030\n",
      "Epoch 231 Batch   15/35   train_loss = 0.264 epoch time = 0.467\n",
      "Epoch 232 Batch   30/35   train_loss = 0.250 epoch time = 0.908\n",
      "Epoch 234 Batch   10/35   train_loss = 0.249 epoch time = 0.321\n",
      "Epoch 235 Batch   25/35   train_loss = 0.234 epoch time = 0.761\n",
      "Epoch 237 Batch    5/35   train_loss = 0.249 epoch time = 0.175\n",
      "Epoch 238 Batch   20/35   train_loss = 0.226 epoch time = 0.611\n",
      "Epoch 240 Batch    0/35   train_loss = 0.257 epoch time = 0.029\n",
      "Epoch 241 Batch   15/35   train_loss = 0.258 epoch time = 0.466\n",
      "Epoch 242 Batch   30/35   train_loss = 0.246 epoch time = 0.898\n",
      "Epoch 244 Batch   10/35   train_loss = 0.247 epoch time = 0.322\n",
      "Epoch 245 Batch   25/35   train_loss = 0.236 epoch time = 0.769\n",
      "Epoch 247 Batch    5/35   train_loss = 0.247 epoch time = 0.185\n",
      "Epoch 248 Batch   20/35   train_loss = 0.223 epoch time = 0.647\n",
      "Epoch 250 Batch    0/35   train_loss = 0.260 epoch time = 0.029\n",
      "Epoch 251 Batch   15/35   train_loss = 0.262 epoch time = 0.464\n",
      "Epoch 252 Batch   30/35   train_loss = 0.246 epoch time = 0.938\n",
      "Epoch 254 Batch   10/35   train_loss = 0.246 epoch time = 0.320\n",
      "Epoch 255 Batch   25/35   train_loss = 0.232 epoch time = 0.785\n",
      "Epoch 257 Batch    5/35   train_loss = 0.244 epoch time = 0.174\n",
      "Epoch 258 Batch   20/35   train_loss = 0.221 epoch time = 0.612\n",
      "Epoch 260 Batch    0/35   train_loss = 0.252 epoch time = 0.029\n",
      "Epoch 261 Batch   15/35   train_loss = 0.251 epoch time = 0.464\n",
      "Epoch 262 Batch   30/35   train_loss = 0.253 epoch time = 0.897\n",
      "Epoch 264 Batch   10/35   train_loss = 0.253 epoch time = 0.330\n",
      "Epoch 265 Batch   25/35   train_loss = 0.246 epoch time = 0.754\n",
      "Epoch 267 Batch    5/35   train_loss = 0.260 epoch time = 0.173\n",
      "Epoch 268 Batch   20/35   train_loss = 0.241 epoch time = 0.609\n",
      "Epoch 270 Batch    0/35   train_loss = 0.280 epoch time = 0.032\n",
      "Epoch 271 Batch   15/35   train_loss = 0.275 epoch time = 0.484\n",
      "Epoch 272 Batch   30/35   train_loss = 0.272 epoch time = 0.938\n",
      "Epoch 274 Batch   10/35   train_loss = 0.280 epoch time = 0.335\n",
      "Epoch 275 Batch   25/35   train_loss = 0.229 epoch time = 0.797\n",
      "Epoch 277 Batch    5/35   train_loss = 0.241 epoch time = 0.173\n",
      "Epoch 278 Batch   20/35   train_loss = 0.216 epoch time = 0.607\n",
      "Epoch 280 Batch    0/35   train_loss = 0.238 epoch time = 0.029\n",
      "Epoch 281 Batch   15/35   train_loss = 0.243 epoch time = 0.479\n",
      "Epoch 282 Batch   30/35   train_loss = 0.229 epoch time = 0.913\n",
      "Epoch 284 Batch   10/35   train_loss = 0.233 epoch time = 0.335\n",
      "Epoch 285 Batch   25/35   train_loss = 0.217 epoch time = 0.818\n",
      "Epoch 287 Batch    5/35   train_loss = 0.228 epoch time = 0.189\n",
      "Epoch 288 Batch   20/35   train_loss = 0.202 epoch time = 0.610\n",
      "Epoch 290 Batch    0/35   train_loss = 0.227 epoch time = 0.029\n",
      "Epoch 291 Batch   15/35   train_loss = 0.237 epoch time = 0.476\n",
      "Epoch 292 Batch   30/35   train_loss = 0.221 epoch time = 0.944\n",
      "Epoch 294 Batch   10/35   train_loss = 0.225 epoch time = 0.320\n",
      "Epoch 295 Batch   25/35   train_loss = 0.213 epoch time = 0.756\n",
      "Epoch 297 Batch    5/35   train_loss = 0.225 epoch time = 0.176\n",
      "Epoch 298 Batch   20/35   train_loss = 0.202 epoch time = 0.610\n",
      "Epoch 300 Batch    0/35   train_loss = 0.225 epoch time = 0.029\n",
      "Epoch 301 Batch   15/35   train_loss = 0.232 epoch time = 0.461\n",
      "Epoch 302 Batch   30/35   train_loss = 0.218 epoch time = 0.898\n",
      "Epoch 304 Batch   10/35   train_loss = 0.222 epoch time = 0.324\n",
      "Epoch 305 Batch   25/35   train_loss = 0.210 epoch time = 0.759\n",
      "Epoch 307 Batch    5/35   train_loss = 0.222 epoch time = 0.174\n",
      "Epoch 308 Batch   20/35   train_loss = 0.199 epoch time = 0.619\n",
      "Epoch 310 Batch    0/35   train_loss = 0.222 epoch time = 0.029\n",
      "Epoch 311 Batch   15/35   train_loss = 0.231 epoch time = 0.465\n",
      "Epoch 312 Batch   30/35   train_loss = 0.216 epoch time = 0.906\n",
      "Epoch 314 Batch   10/35   train_loss = 0.221 epoch time = 0.330\n",
      "Epoch 315 Batch   25/35   train_loss = 0.209 epoch time = 0.759\n",
      "Epoch 317 Batch    5/35   train_loss = 0.221 epoch time = 0.184\n",
      "Epoch 318 Batch   20/35   train_loss = 0.198 epoch time = 0.639\n",
      "Epoch 320 Batch    0/35   train_loss = 0.221 epoch time = 0.032\n",
      "Epoch 321 Batch   15/35   train_loss = 0.230 epoch time = 0.471\n",
      "Epoch 322 Batch   30/35   train_loss = 0.215 epoch time = 0.914\n",
      "Epoch 324 Batch   10/35   train_loss = 0.220 epoch time = 0.330\n",
      "Epoch 325 Batch   25/35   train_loss = 0.208 epoch time = 0.754\n",
      "Epoch 327 Batch    5/35   train_loss = 0.220 epoch time = 0.174\n",
      "Epoch 328 Batch   20/35   train_loss = 0.197 epoch time = 0.627\n",
      "Epoch 330 Batch    0/35   train_loss = 0.220 epoch time = 0.029\n",
      "Epoch 331 Batch   15/35   train_loss = 0.229 epoch time = 0.490\n",
      "Epoch 332 Batch   30/35   train_loss = 0.215 epoch time = 0.903\n",
      "Epoch 334 Batch   10/35   train_loss = 0.219 epoch time = 0.319\n",
      "Epoch 335 Batch   25/35   train_loss = 0.208 epoch time = 0.768\n",
      "Epoch 337 Batch    5/35   train_loss = 0.219 epoch time = 0.177\n",
      "Epoch 338 Batch   20/35   train_loss = 0.197 epoch time = 0.636\n",
      "Epoch 340 Batch    0/35   train_loss = 0.219 epoch time = 0.031\n",
      "Epoch 341 Batch   15/35   train_loss = 0.229 epoch time = 0.485\n",
      "Epoch 342 Batch   30/35   train_loss = 0.214 epoch time = 0.919\n",
      "Epoch 344 Batch   10/35   train_loss = 0.219 epoch time = 0.331\n",
      "Epoch 345 Batch   25/35   train_loss = 0.207 epoch time = 0.752\n",
      "Epoch 347 Batch    5/35   train_loss = 0.219 epoch time = 0.174\n",
      "Epoch 348 Batch   20/35   train_loss = 0.196 epoch time = 0.640\n",
      "Epoch 350 Batch    0/35   train_loss = 0.219 epoch time = 0.031\n",
      "Epoch 351 Batch   15/35   train_loss = 0.228 epoch time = 0.496\n",
      "Epoch 352 Batch   30/35   train_loss = 0.214 epoch time = 0.961\n",
      "Epoch 354 Batch   10/35   train_loss = 0.218 epoch time = 0.338\n",
      "Epoch 355 Batch   25/35   train_loss = 0.207 epoch time = 0.785\n",
      "Epoch 357 Batch    5/35   train_loss = 0.218 epoch time = 0.181\n",
      "Epoch 358 Batch   20/35   train_loss = 0.196 epoch time = 0.633\n",
      "Epoch 360 Batch    0/35   train_loss = 0.218 epoch time = 0.030\n",
      "Epoch 361 Batch   15/35   train_loss = 0.228 epoch time = 0.485\n",
      "Epoch 362 Batch   30/35   train_loss = 0.213 epoch time = 0.934\n",
      "Epoch 364 Batch   10/35   train_loss = 0.218 epoch time = 0.331\n",
      "Epoch 365 Batch   25/35   train_loss = 0.207 epoch time = 0.786\n",
      "Epoch 367 Batch    5/35   train_loss = 0.219 epoch time = 0.181\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 368 Batch   20/35   train_loss = 0.197 epoch time = 0.633\n",
      "Epoch 370 Batch    0/35   train_loss = 0.231 epoch time = 0.030\n",
      "Epoch 371 Batch   15/35   train_loss = 0.336 epoch time = 0.483\n",
      "Epoch 372 Batch   30/35   train_loss = 0.430 epoch time = 0.934\n",
      "Epoch 374 Batch   10/35   train_loss = 0.295 epoch time = 0.332\n",
      "Epoch 375 Batch   25/35   train_loss = 0.225 epoch time = 0.785\n",
      "Epoch 377 Batch    5/35   train_loss = 0.225 epoch time = 0.181\n",
      "Epoch 378 Batch   20/35   train_loss = 0.200 epoch time = 0.633\n",
      "Epoch 380 Batch    0/35   train_loss = 0.220 epoch time = 0.030\n",
      "Epoch 381 Batch   15/35   train_loss = 0.230 epoch time = 0.482\n",
      "Epoch 382 Batch   30/35   train_loss = 0.215 epoch time = 0.934\n",
      "Epoch 384 Batch   10/35   train_loss = 0.219 epoch time = 0.332\n",
      "Epoch 385 Batch   25/35   train_loss = 0.207 epoch time = 0.786\n",
      "Epoch 387 Batch    5/35   train_loss = 0.219 epoch time = 0.181\n",
      "Epoch 388 Batch   20/35   train_loss = 0.196 epoch time = 0.635\n",
      "Epoch 390 Batch    0/35   train_loss = 0.218 epoch time = 0.030\n",
      "Epoch 391 Batch   15/35   train_loss = 0.228 epoch time = 0.483\n",
      "Epoch 392 Batch   30/35   train_loss = 0.213 epoch time = 0.935\n",
      "Epoch 394 Batch   10/35   train_loss = 0.217 epoch time = 0.330\n",
      "Epoch 395 Batch   25/35   train_loss = 0.205 epoch time = 0.785\n",
      "Epoch 397 Batch    5/35   train_loss = 0.218 epoch time = 0.180\n",
      "Epoch 398 Batch   20/35   train_loss = 0.195 epoch time = 0.632\n",
      "Epoch 400 Batch    0/35   train_loss = 0.217 epoch time = 0.031\n",
      "Epoch 401 Batch   15/35   train_loss = 0.227 epoch time = 0.483\n",
      "Epoch 402 Batch   30/35   train_loss = 0.212 epoch time = 0.935\n",
      "Epoch 404 Batch   10/35   train_loss = 0.216 epoch time = 0.332\n",
      "Epoch 405 Batch   25/35   train_loss = 0.205 epoch time = 0.785\n",
      "Epoch 407 Batch    5/35   train_loss = 0.217 epoch time = 0.181\n",
      "Epoch 408 Batch   20/35   train_loss = 0.194 epoch time = 0.632\n",
      "Epoch 410 Batch    0/35   train_loss = 0.216 epoch time = 0.031\n",
      "Epoch 411 Batch   15/35   train_loss = 0.226 epoch time = 0.481\n",
      "Epoch 412 Batch   30/35   train_loss = 0.212 epoch time = 0.936\n",
      "Epoch 414 Batch   10/35   train_loss = 0.216 epoch time = 0.330\n",
      "Epoch 415 Batch   25/35   train_loss = 0.205 epoch time = 0.784\n",
      "Epoch 417 Batch    5/35   train_loss = 0.216 epoch time = 0.181\n",
      "Epoch 418 Batch   20/35   train_loss = 0.194 epoch time = 0.656\n",
      "Epoch 420 Batch    0/35   train_loss = 0.216 epoch time = 0.032\n",
      "Epoch 421 Batch   15/35   train_loss = 0.226 epoch time = 0.499\n",
      "Epoch 422 Batch   30/35   train_loss = 0.211 epoch time = 0.966\n",
      "Epoch 424 Batch   10/35   train_loss = 0.216 epoch time = 0.338\n",
      "Epoch 425 Batch   25/35   train_loss = 0.204 epoch time = 0.753\n",
      "Epoch 427 Batch    5/35   train_loss = 0.216 epoch time = 0.185\n",
      "Epoch 428 Batch   20/35   train_loss = 0.194 epoch time = 0.638\n",
      "Epoch 430 Batch    0/35   train_loss = 0.215 epoch time = 0.029\n",
      "Epoch 431 Batch   15/35   train_loss = 0.225 epoch time = 0.490\n",
      "Epoch 432 Batch   30/35   train_loss = 0.211 epoch time = 0.921\n",
      "Epoch 434 Batch   10/35   train_loss = 0.215 epoch time = 0.318\n",
      "Epoch 435 Batch   25/35   train_loss = 0.204 epoch time = 0.777\n",
      "Epoch 437 Batch    5/35   train_loss = 0.216 epoch time = 0.187\n",
      "Epoch 438 Batch   20/35   train_loss = 0.193 epoch time = 0.612\n",
      "Epoch 440 Batch    0/35   train_loss = 0.215 epoch time = 0.031\n",
      "Epoch 441 Batch   15/35   train_loss = 0.225 epoch time = 0.468\n",
      "Epoch 442 Batch   30/35   train_loss = 0.211 epoch time = 0.898\n",
      "Epoch 444 Batch   10/35   train_loss = 0.215 epoch time = 0.327\n",
      "Epoch 445 Batch   25/35   train_loss = 0.204 epoch time = 0.758\n",
      "Epoch 447 Batch    5/35   train_loss = 0.216 epoch time = 0.180\n",
      "Epoch 448 Batch   20/35   train_loss = 0.193 epoch time = 0.622\n",
      "Epoch 450 Batch    0/35   train_loss = 0.215 epoch time = 0.029\n",
      "Epoch 451 Batch   15/35   train_loss = 0.225 epoch time = 0.466\n",
      "Epoch 452 Batch   30/35   train_loss = 0.210 epoch time = 0.930\n",
      "Epoch 454 Batch   10/35   train_loss = 0.215 epoch time = 0.330\n",
      "Epoch 455 Batch   25/35   train_loss = 0.204 epoch time = 0.814\n",
      "Epoch 457 Batch    5/35   train_loss = 0.215 epoch time = 0.177\n",
      "Epoch 458 Batch   20/35   train_loss = 0.193 epoch time = 0.607\n",
      "Epoch 460 Batch    0/35   train_loss = 0.215 epoch time = 0.031\n",
      "Epoch 461 Batch   15/35   train_loss = 0.224 epoch time = 0.475\n",
      "Epoch 462 Batch   30/35   train_loss = 0.210 epoch time = 0.925\n",
      "Epoch 464 Batch   10/35   train_loss = 0.215 epoch time = 0.328\n",
      "Epoch 465 Batch   25/35   train_loss = 0.203 epoch time = 0.767\n",
      "Epoch 467 Batch    5/35   train_loss = 0.215 epoch time = 0.180\n",
      "Epoch 468 Batch   20/35   train_loss = 0.193 epoch time = 0.654\n",
      "Epoch 470 Batch    0/35   train_loss = 0.214 epoch time = 0.030\n",
      "Epoch 471 Batch   15/35   train_loss = 0.224 epoch time = 0.476\n",
      "Epoch 472 Batch   30/35   train_loss = 0.210 epoch time = 0.896\n",
      "Epoch 474 Batch   10/35   train_loss = 0.214 epoch time = 0.317\n",
      "Epoch 475 Batch   25/35   train_loss = 0.203 epoch time = 0.754\n",
      "Epoch 477 Batch    5/35   train_loss = 0.215 epoch time = 0.173\n",
      "Epoch 478 Batch   20/35   train_loss = 0.192 epoch time = 0.634\n",
      "Epoch 480 Batch    0/35   train_loss = 0.214 epoch time = 0.029\n",
      "Epoch 481 Batch   15/35   train_loss = 0.224 epoch time = 0.464\n",
      "Epoch 482 Batch   30/35   train_loss = 0.210 epoch time = 0.930\n",
      "Epoch 484 Batch   10/35   train_loss = 0.214 epoch time = 0.318\n",
      "Epoch 485 Batch   25/35   train_loss = 0.203 epoch time = 0.752\n",
      "Epoch 487 Batch    5/35   train_loss = 0.215 epoch time = 0.182\n",
      "Epoch 488 Batch   20/35   train_loss = 0.192 epoch time = 0.641\n",
      "Epoch 490 Batch    0/35   train_loss = 0.216 epoch time = 0.032\n",
      "Epoch 491 Batch   15/35   train_loss = 0.225 epoch time = 0.482\n",
      "Epoch 492 Batch   30/35   train_loss = 0.227 epoch time = 0.924\n",
      "Epoch 494 Batch   10/35   train_loss = 0.356 epoch time = 0.318\n",
      "Epoch 495 Batch   25/35   train_loss = 0.359 epoch time = 0.772\n",
      "Epoch 497 Batch    5/35   train_loss = 0.263 epoch time = 0.187\n",
      "Epoch 498 Batch   20/35   train_loss = 0.203 epoch time = 0.610\n",
      "WARNING:tensorflow:Error encountered when serializing LAYER_NAME_UIDS.\n",
      "Type is unsupported, or the types of the items don't match field type in CollectionDef.\n",
      "'dict' object has no attribute 'name'\n",
      "Model Trained and Saved,duration=522.365\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "batches = get_batches(int_text, batch_size, seq_length)\n",
    "\n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    train_start_time = time.time()\n",
    "    for epoch_i in range(num_epochs):\n",
    "        epoch_start = time.time()\n",
    "        state = sess.run(initial_state, {input_text: batches[0][0]})\n",
    "\n",
    "        for batch_i, (x, y) in enumerate(batches):\n",
    "            feed = {\n",
    "                input_text: x,\n",
    "                targets: y,\n",
    "                initial_state: state,\n",
    "                lr: learning_rate}\n",
    "            train_loss, state, _ = sess.run([cost, final_state, train_op], feed)\n",
    "            \n",
    "            epoch_end = time.time()\n",
    "            # Show every <show_every_n_batches> batches\n",
    "            if (epoch_i * len(batches) + batch_i) % show_every_n_batches == 0:\n",
    "                print('Epoch {:>3} Batch {:>4}/{}   train_loss = {:.3f} epoch time = {:.3f}'.format(\n",
    "                    epoch_i,\n",
    "                    batch_i,\n",
    "                    len(batches),\n",
    "                    train_loss,epoch_end-epoch_start))\n",
    "    train_end_time = time.time()\n",
    "    # Save Model\n",
    "    saver = tf.train.Saver()\n",
    "    saver.save(sess, save_dir)\n",
    "    print('Model Trained and Saved,duration={:.3f}'.format(train_end_time-train_start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save Parameters\n",
    "Save `seq_length` and `save_dir` for generating a new TV script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "# Save parameters for checkpoint\n",
    "helper.save_params((seq_length, save_dir))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Checkpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL\n",
    "\"\"\"\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import helper\n",
    "import problem_unittests as tests\n",
    "import time\n",
    "\n",
    "_, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()\n",
    "seq_length, load_dir = helper.load_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implement Generate Functions\n",
    "### Get Tensors\n",
    "Get tensors from `loaded_graph` using the function [`get_tensor_by_name()`](https://www.tensorflow.org/api_docs/python/tf/Graph#get_tensor_by_name).  Get the tensors using the following names:\n",
    "- \"input:0\"\n",
    "- \"initial_state:0\"\n",
    "- \"final_state:0\"\n",
    "- \"probs:0\"\n",
    "\n",
    "Return the tensors in the following tuple `(InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor)` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def get_tensors(loaded_graph):\n",
    "    \"\"\"\n",
    "    Get input, initial state, final state, and probabilities tensor from <loaded_graph>\n",
    "    :param loaded_graph: TensorFlow graph loaded from file\n",
    "    :return: Tuple (InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor)\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    InputTensor = loaded_graph.get_tensor_by_name(\"input:0\")\n",
    "    InitialStateTensor = loaded_graph.get_tensor_by_name(\"initial_state:0\")\n",
    "    FinalStateTensor = loaded_graph.get_tensor_by_name(\"final_state:0\")\n",
    "    ProbsTensor = loaded_graph.get_tensor_by_name(\"probs:0\")\n",
    "    return InputTensor, InitialStateTensor,FinalStateTensor,ProbsTensor\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_get_tensors(get_tensors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choose Word\n",
    "Implement the `pick_word()` function to select the next word using `probabilities`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def pick_word(probabilities, int_to_vocab):\n",
    "    \"\"\"\n",
    "    Pick the next word in the generated text\n",
    "    :param probabilities: Probabilites of the next word\n",
    "    :param int_to_vocab: Dictionary of word ids as the keys and words as the values\n",
    "    :return: String of the predicted word\n",
    "    \"\"\"\n",
    "    # TODO: Implement Function\n",
    "    probabilities = list(probabilities.reshape(len(int_to_vocab)))\n",
    "    selected_word_index = np.argmax(probabilities)\n",
    "\n",
    "    return int_to_vocab[selected_word_index]\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_pick_word(pick_word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate TV Script\n",
    "This will generate the TV script for you.  Set `gen_length` to the length of TV script you want to generate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./save\n",
      "moe_szyslak:(into phone) gotcha ya down for forty bucks. good luck your eminence.;\n",
      "moe_szyslak:(to moe, proudly) she has a bag. i'm pretty good and i've got a amanda huggenkiss? / to go.\n",
      "c. _montgomery_burns:(into phone) hello...\n",
      "homer_simpson:(singing) hello, ma where i need a great friend.(looks card, get) no, i know what i'm gonna stop...(to moe, homer) i thought you said i get it.\n",
      "lucius: well, when i couldn't if anybody don't be that while.\n",
      "carl_carlson: all right, i don't be a little secret...\n",
      "seymour_skinner: i know.\n",
      "moe_szyslak: hey, homer. if you're going. all the greatest! why don't you go off the way!\n",
      "moe_szyslak:(looking at; part)\n",
      "homer_simpson: yes, i don't want to hurt people's feelings.\n",
      "homer_simpson: i see hey, how did you told are bart. i'm really too much, uh....\n"
     ]
    }
   ],
   "source": [
    "gen_length = 200\n",
    "# homer_simpson, moe_szyslak, or Barney_Gumble\n",
    "prime_word = 'moe_szyslak'\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "loaded_graph = tf.Graph()\n",
    "with tf.Session(graph=loaded_graph) as sess:\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "    loader.restore(sess, load_dir)\n",
    "\n",
    "    # Get Tensors from loaded model\n",
    "    input_text, initial_state, final_state, probs = get_tensors(loaded_graph)\n",
    "\n",
    "    # Sentences generation setup\n",
    "    gen_sentences = [prime_word + ':']\n",
    "    prev_state = sess.run(initial_state, {input_text: np.array([[1]])})\n",
    "\n",
    "    # Generate sentences\n",
    "    for n in range(gen_length):\n",
    "        # Dynamic Input\n",
    "        dyn_input = [[vocab_to_int[word] for word in gen_sentences[-seq_length:]]]\n",
    "        dyn_seq_length = len(dyn_input[0])\n",
    "\n",
    "        # Get Prediction\n",
    "        probabilities, prev_state = sess.run(\n",
    "            [probs, final_state],\n",
    "            {input_text: dyn_input, initial_state: prev_state})\n",
    "        pred_word = pick_word(probabilities[0,dyn_seq_length-1], int_to_vocab)\n",
    "\n",
    "        gen_sentences.append(pred_word)\n",
    "    \n",
    "    # Remove tokens\n",
    "    tv_script = ' '.join(gen_sentences)\n",
    "    for key, token in token_dict.items():\n",
    "        ending = ' ' if key in ['\\n', '(', '\"'] else ''\n",
    "        tv_script = tv_script.replace(' ' + token.lower(), key)\n",
    "    tv_script = tv_script.replace('\\n ', '\\n')\n",
    "    tv_script = tv_script.replace('( ', '(')\n",
    "        \n",
    "    print(tv_script)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The TV Script is Nonsensical\n",
    "It's ok if the TV script doesn't make any sense.  We trained on less than a megabyte of text.  In order to get good results, you'll have to use a smaller vocabulary or get more data.  Luckly there's more data!  As we mentioned in the begging of this project, this is a subset of [another dataset](https://www.kaggle.com/wcukierski/the-simpsons-by-the-data).  We didn't have you train on all the data, because that would take too long.  However, you are free to train your neural network on all the data.  After you complete the project, of course.\n",
    "# Submitting This Project\n",
    "When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as \"dlnd_tv_script_generation.ipynb\" and save it as a HTML file under \"File\" -> \"Download as\". Include the \"helper.py\" and \"problem_unittests.py\" files in your submission."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}