SpellChecker.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a Spell Checker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The objective of this project is to build a model that can take a sentence with spelling mistakes as input, and output the same sentence, but with the mistakes corrected. The data that we will use for this project will be twenty popular books from [Project Gutenberg](http://www.gutenberg.org/ebooks/search/?sort_order=downloads). Our model is designed using grid search to find the optimal architecture, and hyperparameter values. The best results, as measured by sequence loss with 15% of our data, were created using a two-layered network with a bi-direction RNN in the encoding layer and Bahdanau Attention in the decoding layer. [FloydHub's](https://www.floydhub.com/) GPU service was used to train the model.\n",
    "\n",
    "The sections of the project are:\n",
    "- Loading the Data\n",
    "- Preparing the Data\n",
    "- Building the Model\n",
    "- Training the Model\n",
    "- Fixing Custom Sentences\n",
    "- Summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import os\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "from collections import namedtuple\n",
    "from tensorflow.python.layers.core import Dense\n",
    "from tensorflow.python.ops.rnn_cell_impl import _zero_state_tensors\n",
    "import time\n",
    "import re\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def load_book(path):\n",
    "    \"\"\"Load a book from its file\"\"\"\n",
    "    input_file = os.path.join(path)\n",
    "    with open(input_file) as f:\n",
    "        book = f.read()\n",
    "    return book"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Collect all of the book file names\n",
    "path = './books/'\n",
    "book_files = [f for f in listdir(path) if isfile(join(path, f))]\n",
    "book_files = book_files[1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load the books using the file names\n",
    "books = []\n",
    "for book in book_files:\n",
    "    books.append(load_book(path+book))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 30423 words in Alices_Adventures_in_Wonderland_by_Lewis_Carroll.rtf.\n",
      "There are 361612 words in Anna_Karenina_by_Leo_Tolstoy.rtf.\n",
      "There are 113452 words in David_Copperfield_by_Charles_Dickens.rtf.\n",
      "There are 433993 words in Don_Quixote_by_Miguel_de_Cervantes.rtf.\n",
      "There are 166996 words in Dracula_by_Bram_Stoker.rtf.\n",
      "There are 163109 words in Emma_by_Jane_Austen.rtf.\n",
      "There are 78912 words in Frankenstein_by_Mary_Shelley.rtf.\n",
      "There are 191598 words in Great_Expectations_by_Charles_Dickens.rtf.\n",
      "There are 105428 words in Grimms_Fairy_Tales_by_The_Brothers_Grimm.rtf.\n",
      "There are 25395 words in Metamorphosis_by_Franz_Kafka.rtf.\n",
      "There are 165188 words in Oliver_Twist_by_Charles_Dickens.rtf.\n",
      "There are 126999 words in Pride_and_Prejudice_by_Jane_Austen.rtf.\n",
      "There are 110213 words in The_Adventures_of_Sherlock_Holmes_by_Arthur_Conan_Doyle.rtf.\n",
      "There are 96185 words in The_Adventures_of_Tom_Sawyer_by_Mark_Twain.rtf.\n",
      "There are 480495 words in The_Count_of_Monte_Cristo_by_Alexandre_Dumas.rtf.\n",
      "There are 83657 words in The_Picture_of_Dorian_Gray_by_Oscar_Wilde.rtf.\n",
      "There are 53211 words in The_Prince_by_Nicolo_Machiavelli.rtf.\n",
      "There are 194282 words in The_Romance_of_Lust_by_Anonymous.rtf.\n",
      "There are 9463 words in The_Yellow_Wallpaper_by_Charlotte_Perkins_Gilman.rtf.\n",
      "There are 33464 words in Through_the_Looking_Glass_by_Lewis_Carroll.rtf.\n"
     ]
    }
   ],
   "source": [
    "# Compare the number of words in each book \n",
    "for i in range(len(books)):\n",
    "    print(\"There are {} words in {}.\".format(len(books[i].split()), book_files[i]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"{\\\\rtf1\\\\ansi\\\\ansicpg1252\\\\cocoartf1404\\\\cocoasubrtf470\\n{\\\\fonttbl\\\\f0\\\\fmodern\\\\fcharset0 Courier;}\\n{\\\\colortbl;\\\\red255\\\\green255\\\\blue255;\\\\red0\\\\green0\\\\blue0;}\\n\\\\margl1440\\\\margr1440\\\\vieww10800\\\\viewh8400\\\\viewkind0\\n\\\\deftab720\\n\\\\pard\\\\pardeftab720\\\\sl280\\\\partightenfactor0\\n\\n\\\\f0\\\\fs24 \\\\cf2 \\\\expnd0\\\\expndtw0\\\\kerning0\\n\\\\outl0\\\\strokewidth0 \\\\strokec2 Project Gutenberg\\\\'92s Alice\\\\'92s Adventures in Wonderland, by Lewis Carroll\\\\\\n\\\\\\nThis eBook is for the use of anyone anywhere at no cost and with\\\\\\nalmost no restrictions what\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check to ensure the text looks alright\n",
    "books[0][:500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def clean_text(text):\n",
    "    '''Remove unwanted characters and extra spaces from the text'''\n",
    "    text = re.sub(r'\\n', ' ', text) \n",
    "    text = re.sub(r'[{}@_*>()\\\\#%+=\\[\\]]','', text)\n",
    "    text = re.sub('a0','', text)\n",
    "    text = re.sub('\\'92t','\\'t', text)\n",
    "    text = re.sub('\\'92s','\\'s', text)\n",
    "    text = re.sub('\\'92m','\\'m', text)\n",
    "    text = re.sub('\\'92ll','\\'ll', text)\n",
    "    text = re.sub('\\'91','', text)\n",
    "    text = re.sub('\\'92','', text)\n",
    "    text = re.sub('\\'93','', text)\n",
    "    text = re.sub('\\'94','', text)\n",
    "    text = re.sub('\\.','. ', text)\n",
    "    text = re.sub('\\!','! ', text)\n",
    "    text = re.sub('\\?','? ', text)\n",
    "    text = re.sub(' +',' ', text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Clean the text of the books\n",
    "clean_books = []\n",
    "for book in books:\n",
    "    clean_books.append(clean_text(book))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"rtf1ansiansicpg1252cocoartf1404cocoasubrtf470 fonttblf0fmodernfcharset0 Courier; colortbl;red255green255blue255;red0green0blue0; margl1440margr1440vieww10800viewh8400viewkind0 deftab720 pardpardeftab720sl280partightenfactor0 f0fs24 cf2 expnd0expndtw0kerning0 outl0strokewidth0 strokec2 Project Gutenberg's Alice's Adventures in Wonderland, by Lewis Carroll This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it \""
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check to ensure the text has been cleaned properly\n",
    "clean_books[0][:500]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a dictionary to convert the vocabulary (characters) to integers\n",
    "vocab_to_int = {}\n",
    "count = 0\n",
    "for book in clean_books:\n",
    "    for character in book:\n",
    "        if character not in vocab_to_int:\n",
    "            vocab_to_int[character] = count\n",
    "            count += 1\n",
    "\n",
    "# Add special tokens to vocab_to_int\n",
    "codes = ['<PAD>','<EOS>','<GO>']\n",
    "for code in codes:\n",
    "    vocab_to_int[code] = count\n",
    "    count += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The vocabulary contains 78 characters.\n",
      "[' ', '!', '\"', '$', '&', \"'\", ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<EOS>', '<GO>', '<PAD>', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n"
     ]
    }
   ],
   "source": [
    "# Check the size of vocabulary and all of the values\n",
    "vocab_size = len(vocab_to_int)\n",
    "print(\"The vocabulary contains {} characters.\".format(vocab_size))\n",
    "print(sorted(vocab_to_int))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: We could have made this project a little easier by using only lower case words and fewer special characters ($,&,-...), but I want to make this spell checker as useful as possible.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create another dictionary to convert integers to their respective characters\n",
    "int_to_vocab = {}\n",
    "for character, value in vocab_to_int.items():\n",
    "    int_to_vocab[value] = character"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 133494 sentences.\n"
     ]
    }
   ],
   "source": [
    "# Split the text from the books into sentences.\n",
    "sentences = []\n",
    "for book in clean_books:\n",
    "    for sentence in book.split('. '):\n",
    "        sentences.append(sentence + '.')\n",
    "print(\"There are {} sentences.\".format(len(sentences)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"rtf1ansiansicpg1252cocoartf1404cocoasubrtf470 fonttblf0fmodernfcharset0 Courier; colortbl;red255green255blue255;red0green0blue0; margl1440margr1440vieww10800viewh8400viewkind0 deftab720 pardpardeftab720sl280partightenfactor0 f0fs24 cf2 expnd0expndtw0kerning0 outl0strokewidth0 strokec2 Project Gutenberg's Alice's Adventures in Wonderland, by Lewis Carroll This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever.\",\n",
       " 'You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.',\n",
       " 'gutenberg.',\n",
       " \"org Title: Alice's Adventures in Wonderland Author: Lewis Carroll Posting Date: June 25, 2008 EBook 11 Release Date: March, 1994 Last Updated: October 6, 2016 Language: English Character set encoding: UTF-8 START OF THIS PROJECT GUTENBERG EBOOK ALICES ADVENTURES IN WONDERLAND ALICES ADVENTURES IN WONDERLAND Lewis Carroll THE MILLENNIUM FULCRUM EDITION 3.\",\n",
       " '0 CHAPTER I.']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check to ensure the text has been split correctly.\n",
    "sentences[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: I expect that you have noticed the very ugly text in the first sentence. We do not need to worry about removing it from any of the books because will be limiting our data to sentences that are shorter than it.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Convert sentences to integers\n",
    "int_sentences = []\n",
    "\n",
    "for sentence in sentences:\n",
    "    int_sentence = []\n",
    "    for character in sentence:\n",
    "        int_sentence.append(vocab_to_int[character])\n",
    "    int_sentences.append(int_sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Find the length of each sentence\n",
    "lengths = []\n",
    "for sentence in int_sentences:\n",
    "    lengths.append(len(sentence))\n",
    "lengths = pd.DataFrame(lengths, columns=[\"counts\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>133494.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>120.474418</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>116.765852</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>46.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>92.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>159.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>8906.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              counts\n",
       "count  133494.000000\n",
       "mean      120.474418\n",
       "std       116.765852\n",
       "min         1.000000\n",
       "25%        46.000000\n",
       "50%        92.000000\n",
       "75%       159.000000\n",
       "max      8906.000000"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lengths.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We will use 59885 to train and test our model.\n"
     ]
    }
   ],
   "source": [
    "# Limit the data we will use to train our model\n",
    "max_length = 92\n",
    "min_length = 10\n",
    "\n",
    "good_sentences = []\n",
    "\n",
    "for sentence in int_sentences:\n",
    "    if len(sentence) <= max_length and len(sentence) >= min_length:\n",
    "        good_sentences.append(sentence)\n",
    "\n",
    "print(\"We will use {} to train and test our model.\".format(len(good_sentences)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: I decided to not use very long or short sentences because they are not as useful for training our model. Shorter sentences are less likely to include an error and the text is more likely to be repetitive. Longer sentences are more difficult to learn due to their length and increase the training time quite a bit. If you are interested in using this model for more than just a personal project, it would be worth using these longer sentence, and much more training data to create a more accurate model.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training sentences: 50902\n",
      "Number of testing sentences: 8983\n"
     ]
    }
   ],
   "source": [
    "# Split the data into training and testing sentences\n",
    "training, testing = train_test_split(good_sentences, test_size = 0.15, random_state = 2)\n",
    "\n",
    "print(\"Number of training sentences:\", len(training))\n",
    "print(\"Number of testing sentences:\", len(testing))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Sort the sentences by length to reduce padding, which will allow the model to train faster\n",
    "training_sorted = []\n",
    "testing_sorted = []\n",
    "\n",
    "for i in range(min_length, max_length+1):\n",
    "    for sentence in training:\n",
    "        if len(sentence) == i:\n",
    "            training_sorted.append(sentence)\n",
    "    for sentence in testing:\n",
    "        if len(sentence) == i:\n",
    "            testing_sorted.append(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[42, 23, 1, 0, 13, 1, 24, 23, 22, 43] 10\n",
      "[62, 23, 6, 20, 13, 27, 6, 30, 39, 43] 10\n",
      "[3, 63, 38, 19, 32, 4, 0, 7, 6, 43] 10\n",
      "[10, 16, 1, 23, 5, 17, 23, 0, 10, 43] 10\n",
      "[61, 5, 7, 10, 24, 1, 20, 23, 39, 43] 10\n"
     ]
    }
   ],
   "source": [
    "# Check to ensure the sentences have been selected and sorted correctly\n",
    "for i in range(5):\n",
    "    print(training_sorted[i], len(training_sorted[i]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "letters = ['a','b','c','d','e','f','g','h','i','j','k','l','m',\n",
    "           'n','o','p','q','r','s','t','u','v','w','x','y','z',]\n",
    "\n",
    "def noise_maker(sentence, threshold):\n",
    "    '''Relocate, remove, or add characters to create spelling mistakes'''\n",
    "    \n",
    "    noisy_sentence = []\n",
    "    i = 0\n",
    "    while i < len(sentence):\n",
    "        random = np.random.uniform(0,1,1)\n",
    "        # Most characters will be correct since the threshold value is high\n",
    "        if random < threshold:\n",
    "            noisy_sentence.append(sentence[i])\n",
    "        else:\n",
    "            new_random = np.random.uniform(0,1,1)\n",
    "            # ~33% chance characters will swap locations\n",
    "            if new_random > 0.67:\n",
    "                if i == (len(sentence) - 1):\n",
    "                    # If last character in sentence, it will not be typed\n",
    "                    continue\n",
    "                else:\n",
    "                    # if any other character, swap order with following character\n",
    "                    noisy_sentence.append(sentence[i+1])\n",
    "                    noisy_sentence.append(sentence[i])\n",
    "                    i += 1\n",
    "            # ~33% chance an extra lower case letter will be added to the sentence\n",
    "            elif new_random < 0.33:\n",
    "                random_letter = np.random.choice(letters, 1)[0]\n",
    "                noisy_sentence.append(vocab_to_int[random_letter])\n",
    "                noisy_sentence.append(sentence[i])\n",
    "            # ~33% chance a character will not be typed\n",
    "            else:\n",
    "                pass     \n",
    "        i += 1\n",
    "    return noisy_sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: The noise_maker function is used to create spelling mistakes that are similar to those we would make. Sometimes we forget to type a letter, type a letter in the wrong location, or add an extra letter.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[42, 23, 1, 0, 13, 1, 24, 23, 22, 43]\n",
      "[42, 23, 1, 0, 13, 1, 24, 23, 22, 43]\n",
      "\n",
      "[62, 23, 6, 20, 13, 27, 6, 30, 39, 43]\n",
      "[62, 23, 6, 20, 13, 6, 30, 39, 43]\n",
      "\n",
      "[3, 63, 38, 19, 32, 4, 0, 7, 6, 43]\n",
      "[3, 63, 38, 19, 4, 32, 0, 28, 7, 6, 43]\n",
      "\n",
      "[10, 16, 1, 23, 5, 17, 23, 0, 10, 43]\n",
      "[10, 1, 23, 5, 17, 23, 0, 43, 10]\n",
      "\n",
      "[61, 5, 7, 10, 24, 1, 20, 23, 39, 43]\n",
      "[61, 5, 7, 24, 1, 31, 20, 23, 39, 43]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Check to ensure noise_maker is making mistakes correctly.\n",
    "threshold = 0.9\n",
    "for sentence in training_sorted[:5]:\n",
    "    print(sentence)\n",
    "    print(noise_maker(sentence, threshold))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Building the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def model_inputs():\n",
    "    '''Create palceholders for inputs to the model'''\n",
    "    \n",
    "    with tf.name_scope('inputs'):\n",
    "        inputs = tf.placeholder(tf.int32, [None, None], name='inputs')\n",
    "    with tf.name_scope('targets'):\n",
    "        targets = tf.placeholder(tf.int32, [None, None], name='targets')\n",
    "    keep_prob = tf.placeholder(tf.float32, name='keep_prob')\n",
    "    inputs_length = tf.placeholder(tf.int32, (None,), name='inputs_length')\n",
    "    targets_length = tf.placeholder(tf.int32, (None,), name='targets_length')\n",
    "    max_target_length = tf.reduce_max(targets_length, name='max_target_len')\n",
    "\n",
    "    return inputs, targets, keep_prob, inputs_length, targets_length, max_target_length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def process_encoding_input(targets, vocab_to_int, batch_size):\n",
    "    '''Remove the last word id from each batch and concat the <GO> to the begining of each batch'''\n",
    "    \n",
    "    with tf.name_scope(\"process_encoding\"):\n",
    "        ending = tf.strided_slice(targets, [0, 0], [batch_size, -1], [1, 1])\n",
    "        dec_input = tf.concat([tf.fill([batch_size, 1], vocab_to_int['<GO>']), ending], 1)\n",
    "\n",
    "    return dec_input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def encoding_layer(rnn_size, sequence_length, num_layers, rnn_inputs, keep_prob, direction):\n",
    "    '''Create the encoding layer'''\n",
    "    \n",
    "    if direction == 1:\n",
    "        with tf.name_scope(\"RNN_Encoder_Cell_1D\"):\n",
    "            for layer in range(num_layers):\n",
    "                with tf.variable_scope('encoder_{}'.format(layer)):\n",
    "                    lstm = tf.contrib.rnn.LSTMCell(rnn_size)\n",
    "\n",
    "                    drop = tf.contrib.rnn.DropoutWrapper(lstm, \n",
    "                                                         input_keep_prob = keep_prob)\n",
    "\n",
    "                    enc_output, enc_state = tf.nn.dynamic_rnn(drop, \n",
    "                                                              rnn_inputs,\n",
    "                                                              sequence_length,\n",
    "                                                              dtype=tf.float32)\n",
    "\n",
    "            return enc_output, enc_state\n",
    "        \n",
    "        \n",
    "    if direction == 2:\n",
    "        with tf.name_scope(\"RNN_Encoder_Cell_2D\"):\n",
    "            for layer in range(num_layers):\n",
    "                with tf.variable_scope('encoder_{}'.format(layer)):\n",
    "                    cell_fw = tf.contrib.rnn.LSTMCell(rnn_size)\n",
    "                    cell_fw = tf.contrib.rnn.DropoutWrapper(cell_fw, \n",
    "                                                            input_keep_prob = keep_prob)\n",
    "\n",
    "                    cell_bw = tf.contrib.rnn.LSTMCell(rnn_size)\n",
    "                    cell_bw = tf.contrib.rnn.DropoutWrapper(cell_bw, \n",
    "                                                            input_keep_prob = keep_prob)\n",
    "\n",
    "                    enc_output, enc_state = tf.nn.bidirectional_dynamic_rnn(cell_fw, \n",
    "                                                                            cell_bw, \n",
    "                                                                            rnn_inputs,\n",
    "                                                                            sequence_length,\n",
    "                                                                            dtype=tf.float32)\n",
    "            # Join outputs since we are using a bidirectional RNN\n",
    "            enc_output = tf.concat(enc_output,2)\n",
    "            # Use only the forward state because the model can't use both states at once\n",
    "            return enc_output, enc_state[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def training_decoding_layer(dec_embed_input, targets_length, dec_cell, initial_state, output_layer, \n",
    "                            vocab_size, max_target_length):\n",
    "    '''Create the training logits'''\n",
    "    \n",
    "    with tf.name_scope(\"Training_Decoder\"):\n",
    "        training_helper = tf.contrib.seq2seq.TrainingHelper(inputs=dec_embed_input,\n",
    "                                                            sequence_length=targets_length,\n",
    "                                                            time_major=False)\n",
    "\n",
    "        training_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                           training_helper,\n",
    "                                                           initial_state,\n",
    "                                                           output_layer) \n",
    "\n",
    "        training_logits, _ = tf.contrib.seq2seq.dynamic_decode(training_decoder,\n",
    "                                                               output_time_major=False,\n",
    "                                                               impute_finished=True,\n",
    "                                                               maximum_iterations=max_target_length)\n",
    "        return training_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def inference_decoding_layer(embeddings, start_token, end_token, dec_cell, initial_state, output_layer,\n",
    "                             max_target_length, batch_size):\n",
    "    '''Create the inference logits'''\n",
    "    \n",
    "    with tf.name_scope(\"Inference_Decoder\"):\n",
    "        start_tokens = tf.tile(tf.constant([start_token], dtype=tf.int32), [batch_size], name='start_tokens')\n",
    "\n",
    "        inference_helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(embeddings,\n",
    "                                                                    start_tokens,\n",
    "                                                                    end_token)\n",
    "\n",
    "        inference_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                            inference_helper,\n",
    "                                                            initial_state,\n",
    "                                                            output_layer)\n",
    "\n",
    "        inference_logits, _ = tf.contrib.seq2seq.dynamic_decode(inference_decoder,\n",
    "                                                                output_time_major=False,\n",
    "                                                                impute_finished=True,\n",
    "                                                                maximum_iterations=max_target_length)\n",
    "\n",
    "        return inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def decoding_layer(dec_embed_input, embeddings, enc_output, enc_state, vocab_size, inputs_length, targets_length, \n",
    "                   max_target_length, rnn_size, vocab_to_int, keep_prob, batch_size, num_layers, direction):\n",
    "    '''Create the decoding cell and attention for the training and inference decoding layers'''\n",
    "    \n",
    "    with tf.name_scope(\"RNN_Decoder_Cell\"):\n",
    "        for layer in range(num_layers):\n",
    "            with tf.variable_scope('decoder_{}'.format(layer)):\n",
    "                lstm = tf.contrib.rnn.LSTMCell(rnn_size)\n",
    "                dec_cell = tf.contrib.rnn.DropoutWrapper(lstm, \n",
    "                                                         input_keep_prob = keep_prob)\n",
    "    \n",
    "    output_layer = Dense(vocab_size,\n",
    "                         kernel_initializer = tf.truncated_normal_initializer(mean = 0.0, stddev=0.1))\n",
    "    \n",
    "    attn_mech = tf.contrib.seq2seq.BahdanauAttention(rnn_size,\n",
    "                                                  enc_output,\n",
    "                                                  inputs_length,\n",
    "                                                  normalize=False,\n",
    "                                                  name='BahdanauAttention')\n",
    "    \n",
    "    with tf.name_scope(\"Attention_Wrapper\"):\n",
    "        dec_cell = tf.contrib.seq2seq.DynamicAttentionWrapper(dec_cell,\n",
    "                                                              attn_mech,\n",
    "                                                              rnn_size)\n",
    "    \n",
    "    initial_state = tf.contrib.seq2seq.DynamicAttentionWrapperState(enc_state,\n",
    "                                                                    _zero_state_tensors(rnn_size, \n",
    "                                                                                        batch_size, \n",
    "                                                                                        tf.float32))\n",
    "\n",
    "    with tf.variable_scope(\"decode\"):\n",
    "        training_logits = training_decoding_layer(dec_embed_input, \n",
    "                                                  targets_length, \n",
    "                                                  dec_cell, \n",
    "                                                  initial_state,\n",
    "                                                  output_layer,\n",
    "                                                  vocab_size, \n",
    "                                                  max_target_length)\n",
    "    with tf.variable_scope(\"decode\", reuse=True):\n",
    "        inference_logits = inference_decoding_layer(embeddings,  \n",
    "                                                    vocab_to_int['<GO>'], \n",
    "                                                    vocab_to_int['<EOS>'],\n",
    "                                                    dec_cell, \n",
    "                                                    initial_state, \n",
    "                                                    output_layer,\n",
    "                                                    max_target_length,\n",
    "                                                    batch_size)\n",
    "\n",
    "    return training_logits, inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def seq2seq_model(inputs, targets, keep_prob, inputs_length, targets_length, max_target_length, \n",
    "                  vocab_size, rnn_size, num_layers, vocab_to_int, batch_size, embedding_size, direction):\n",
    "    '''Use the previous functions to create the training and inference logits'''\n",
    "    \n",
    "    enc_embeddings = tf.Variable(tf.random_uniform([vocab_size, embedding_size], -1, 1))\n",
    "    enc_embed_input = tf.nn.embedding_lookup(enc_embeddings, inputs)\n",
    "    enc_output, enc_state = encoding_layer(rnn_size, inputs_length, num_layers, \n",
    "                                           enc_embed_input, keep_prob, direction)\n",
    "    \n",
    "    dec_embeddings = tf.Variable(tf.random_uniform([vocab_size, embedding_size], -1, 1))\n",
    "    dec_input = process_encoding_input(targets, vocab_to_int, batch_size)\n",
    "    dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)\n",
    "    \n",
    "    training_logits, inference_logits  = decoding_layer(dec_embed_input, \n",
    "                                                        dec_embeddings,\n",
    "                                                        enc_output,\n",
    "                                                        enc_state, \n",
    "                                                        vocab_size, \n",
    "                                                        inputs_length, \n",
    "                                                        targets_length, \n",
    "                                                        max_target_length,\n",
    "                                                        rnn_size, \n",
    "                                                        vocab_to_int, \n",
    "                                                        keep_prob, \n",
    "                                                        batch_size,\n",
    "                                                        num_layers,\n",
    "                                                        direction)\n",
    "    \n",
    "    return training_logits, inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pad_sentence_batch(sentence_batch):\n",
    "    \"\"\"Pad sentences with <PAD> so that each sentence of a batch has the same length\"\"\"\n",
    "    max_sentence = max([len(sentence) for sentence in sentence_batch])\n",
    "    return [sentence + [vocab_to_int['<PAD>']] * (max_sentence - len(sentence)) for sentence in sentence_batch]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_batches(sentences, batch_size, threshold):\n",
    "    \"\"\"Batch sentences, noisy sentences, and the lengths of their sentences together.\n",
    "       With each epoch, sentences will receive new mistakes\"\"\"\n",
    "    \n",
    "    for batch_i in range(0, len(sentences)//batch_size):\n",
    "        start_i = batch_i * batch_size\n",
    "        sentences_batch = sentences[start_i:start_i + batch_size]\n",
    "        \n",
    "        sentences_batch_noisy = []\n",
    "        for sentence in sentences_batch:\n",
    "            sentences_batch_noisy.append(noise_maker(sentence, threshold))\n",
    "            \n",
    "        sentences_batch_eos = []\n",
    "        for sentence in sentences_batch:\n",
    "            sentence.append(vocab_to_int['<EOS>'])\n",
    "            sentences_batch_eos.append(sentence)\n",
    "            \n",
    "        pad_sentences_batch = np.array(pad_sentence_batch(sentences_batch_eos))\n",
    "        pad_sentences_noisy_batch = np.array(pad_sentence_batch(sentences_batch_noisy))\n",
    "        \n",
    "        # Need the lengths for the _lengths parameters\n",
    "        pad_sentences_lengths = []\n",
    "        for sentence in pad_sentences_batch:\n",
    "            pad_sentences_lengths.append(len(sentence))\n",
    "        \n",
    "        pad_sentences_noisy_lengths = []\n",
    "        for sentence in pad_sentences_noisy_batch:\n",
    "            pad_sentences_noisy_lengths.append(len(sentence))\n",
    "        \n",
    "        yield pad_sentences_noisy_batch, pad_sentences_batch, pad_sentences_noisy_lengths, pad_sentences_lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: This set of values achieved the best results.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The default parameters\n",
    "epochs = 100\n",
    "batch_size = 128\n",
    "num_layers = 2\n",
    "rnn_size = 512\n",
    "embedding_size = 128\n",
    "learning_rate = 0.0005\n",
    "direction = 2\n",
    "threshold = 0.95\n",
    "keep_probability = 0.75"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_graph(keep_prob, rnn_size, num_layers, batch_size, learning_rate, embedding_size, direction):\n",
    "\n",
    "    tf.reset_default_graph()\n",
    "    \n",
    "    # Load the model inputs    \n",
    "    inputs, targets, keep_prob, inputs_length, targets_length, max_target_length = model_inputs()\n",
    "\n",
    "    # Create the training and inference logits\n",
    "    training_logits, inference_logits = seq2seq_model(tf.reverse(inputs, [-1]),\n",
    "                                                      targets, \n",
    "                                                      keep_prob,   \n",
    "                                                      inputs_length,\n",
    "                                                      targets_length,\n",
    "                                                      max_target_length,\n",
    "                                                      len(vocab_to_int)+1,\n",
    "                                                      rnn_size, \n",
    "                                                      num_layers, \n",
    "                                                      vocab_to_int,\n",
    "                                                      batch_size,\n",
    "                                                      embedding_size,\n",
    "                                                      direction)\n",
    "\n",
    "    # Create tensors for the training logits and inference logits\n",
    "    training_logits = tf.identity(training_logits.rnn_output, 'logits')\n",
    "\n",
    "    with tf.name_scope('predictions'):\n",
    "        predictions = tf.identity(inference_logits.sample_id, name='predictions')\n",
    "        tf.summary.histogram('predictions', predictions)\n",
    "\n",
    "    # Create the weights for sequence_loss\n",
    "    masks = tf.sequence_mask(targets_length, max_target_length, dtype=tf.float32, name='masks')\n",
    "    \n",
    "    with tf.name_scope(\"cost\"):\n",
    "        # Loss function\n",
    "        cost = tf.contrib.seq2seq.sequence_loss(training_logits, \n",
    "                                                targets, \n",
    "                                                masks)\n",
    "        tf.summary.scalar('cost', cost)\n",
    "\n",
    "    with tf.name_scope(\"optimze\"):\n",
    "        optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "\n",
    "        # Gradient Clipping\n",
    "        gradients = optimizer.compute_gradients(cost)\n",
    "        capped_gradients = [(tf.clip_by_value(grad, -5., 5.), var) for grad, var in gradients if grad is not None]\n",
    "        train_op = optimizer.apply_gradients(capped_gradients)\n",
    "\n",
    "    # Merge all of the summaries\n",
    "    merged = tf.summary.merge_all()    \n",
    "\n",
    "    # Export the nodes \n",
    "    export_nodes = ['inputs', 'targets', 'keep_prob', 'cost', 'inputs_length', 'targets_length',\n",
    "                    'predictions', 'merged', 'train_op','optimizer']\n",
    "    Graph = namedtuple('Graph', export_nodes)\n",
    "    local_dict = locals()\n",
    "    graph = Graph(*[local_dict[each] for each in export_nodes])\n",
    "\n",
    "    return graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Training the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def train(model, epochs, log_string):\n",
    "    '''Train the RNN'''\n",
    "    \n",
    "    with tf.Session() as sess:\n",
    "        sess.run(tf.global_variables_initializer())\n",
    "\n",
    "        # Used to determine when to stop the training early\n",
    "        testing_loss_summary = []\n",
    "\n",
    "        # Keep track of which batch iteration is being trained\n",
    "        iteration = 0\n",
    "        \n",
    "        display_step = 30 # The progress of the training will be displayed after every 30 batches\n",
    "        stop_early = 0 \n",
    "        stop = 3 # If the batch_loss_testing does not decrease in 3 consecutive checks, stop training\n",
    "        per_epoch = 3 # Test the model 3 times per epoch\n",
    "        testing_check = (len(training_sorted)//batch_size//per_epoch)-1\n",
    "\n",
    "        print()\n",
    "        print(\"Training Model: {}\".format(log_string))\n",
    "\n",
    "        train_writer = tf.summary.FileWriter('./logs/1/train/{}'.format(log_string), sess.graph)\n",
    "        test_writer = tf.summary.FileWriter('./logs/1/test/{}'.format(log_string))\n",
    "\n",
    "        for epoch_i in range(1, epochs+1): \n",
    "            batch_loss = 0\n",
    "            batch_time = 0\n",
    "            \n",
    "            for batch_i, (input_batch, target_batch, input_length, target_length) in enumerate(\n",
    "                    get_batches(training_sorted, batch_size, threshold)):\n",
    "                start_time = time.time()\n",
    "\n",
    "                summary, loss, _ = sess.run([model.merged,\n",
    "                                             model.cost, \n",
    "                                             model.train_op], \n",
    "                                             {model.inputs: input_batch,\n",
    "                                              model.targets: target_batch,\n",
    "                                              model.inputs_length: input_length,\n",
    "                                              model.targets_length: target_length,\n",
    "                                              model.keep_prob: keep_probability})\n",
    "\n",
    "\n",
    "                batch_loss += loss\n",
    "                end_time = time.time()\n",
    "                batch_time += end_time - start_time\n",
    "\n",
    "                # Record the progress of training\n",
    "                train_writer.add_summary(summary, iteration)\n",
    "\n",
    "                iteration += 1\n",
    "\n",
    "                if batch_i % display_step == 0 and batch_i > 0:\n",
    "                    print('Epoch {:>3}/{} Batch {:>4}/{} - Loss: {:>6.3f}, Seconds: {:>4.2f}'\n",
    "                          .format(epoch_i,\n",
    "                                  epochs, \n",
    "                                  batch_i, \n",
    "                                  len(training_sorted) // batch_size, \n",
    "                                  batch_loss / display_step, \n",
    "                                  batch_time))\n",
    "                    batch_loss = 0\n",
    "                    batch_time = 0\n",
    "\n",
    "                #### Testing ####\n",
    "                if batch_i % testing_check == 0 and batch_i > 0:\n",
    "                    batch_loss_testing = 0\n",
    "                    batch_time_testing = 0\n",
    "                    for batch_i, (input_batch, target_batch, input_length, target_length) in enumerate(\n",
    "                            get_batches(testing_sorted, batch_size, threshold)):\n",
    "                        start_time_testing = time.time()\n",
    "                        summary, loss = sess.run([model.merged,\n",
    "                                                  model.cost], \n",
    "                                                     {model.inputs: input_batch,\n",
    "                                                      model.targets: target_batch,\n",
    "                                                      model.inputs_length: input_length,\n",
    "                                                      model.targets_length: target_length,\n",
    "                                                      model.keep_prob: 1})\n",
    "\n",
    "                        batch_loss_testing += loss\n",
    "                        end_time_testing = time.time()\n",
    "                        batch_time_testing += end_time_testing - start_time_testing\n",
    "\n",
    "                        # Record the progress of testing\n",
    "                        test_writer.add_summary(summary, iteration)\n",
    "\n",
    "                    n_batches_testing = batch_i + 1\n",
    "                    print('Testing Loss: {:>6.3f}, Seconds: {:>4.2f}'\n",
    "                          .format(batch_loss_testing / n_batches_testing, \n",
    "                                  batch_time_testing))\n",
    "                    \n",
    "                    batch_time_testing = 0\n",
    "\n",
    "                    # If the batch_loss_testing is at a new minimum, save the model\n",
    "                    testing_loss_summary.append(batch_loss_testing)\n",
    "                    if batch_loss_testing <= min(testing_loss_summary):\n",
    "                        print('New Record!') \n",
    "                        stop_early = 0\n",
    "                        checkpoint = \"./{}.ckpt\".format(log_string)\n",
    "                        saver = tf.train.Saver()\n",
    "                        saver.save(sess, checkpoint)\n",
    "\n",
    "                    else:\n",
    "                        print(\"No Improvement.\")\n",
    "                        stop_early += 1\n",
    "                        if stop_early == stop:\n",
    "                            break\n",
    "\n",
    "            if stop_early == stop:\n",
    "                print(\"Stopping Training.\")\n",
    "                break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   1/100 Batch   30/397 - Loss:  2.909, Seconds: 7.14\n",
      "Epoch   1/100 Batch   60/397 - Loss:  1.813, Seconds: 10.06\n",
      "Epoch   1/100 Batch   90/397 - Loss:  1.300, Seconds: 12.86\n",
      "Epoch   1/100 Batch  120/397 - Loss:  0.888, Seconds: 15.96\n",
      "Testing Loss:  3.105, Seconds: 27.39\n",
      "New Record!\n",
      "Epoch   1/100 Batch  150/397 - Loss:  0.545, Seconds: 18.72\n",
      "Epoch   1/100 Batch  180/397 - Loss:  0.642, Seconds: 21.61\n",
      "Epoch   1/100 Batch  210/397 - Loss:  0.383, Seconds: 25.11\n",
      "Epoch   1/100 Batch  240/397 - Loss:  0.339, Seconds: 28.60\n",
      "Testing Loss:  3.143, Seconds: 30.99\n",
      "No Improvement.\n",
      "Epoch   1/100 Batch  270/397 - Loss:  0.309, Seconds: 32.31\n",
      "Epoch   1/100 Batch  300/397 - Loss:  1.803, Seconds: 36.52\n",
      "Epoch   1/100 Batch  330/397 - Loss:  2.088, Seconds: 41.15\n",
      "Epoch   1/100 Batch  360/397 - Loss:  1.676, Seconds: 46.23\n",
      "Epoch   1/100 Batch  390/397 - Loss:  1.267, Seconds: 51.99\n",
      "Testing Loss:  1.384, Seconds: 32.40\n",
      "New Record!\n",
      "Epoch   2/100 Batch   30/397 - Loss:  0.610, Seconds: 7.12\n",
      "Epoch   2/100 Batch   60/397 - Loss:  0.280, Seconds: 10.57\n",
      "Epoch   2/100 Batch   90/397 - Loss:  0.244, Seconds: 13.44\n",
      "Epoch   2/100 Batch  120/397 - Loss:  0.240, Seconds: 16.52\n",
      "Testing Loss:  1.335, Seconds: 30.94\n",
      "New Record!\n",
      "Epoch   2/100 Batch  150/397 - Loss:  0.242, Seconds: 19.22\n",
      "Epoch   2/100 Batch  180/397 - Loss:  0.235, Seconds: 22.30\n",
      "Epoch   2/100 Batch  210/397 - Loss:  0.228, Seconds: 25.87\n",
      "Epoch   2/100 Batch  240/397 - Loss:  0.223, Seconds: 29.38\n",
      "Testing Loss:  1.305, Seconds: 33.29\n",
      "New Record!\n",
      "Epoch   2/100 Batch  270/397 - Loss:  0.215, Seconds: 33.14\n",
      "Epoch   2/100 Batch  300/397 - Loss:  0.761, Seconds: 37.27\n",
      "Epoch   2/100 Batch  330/397 - Loss:  0.473, Seconds: 41.91\n",
      "Epoch   2/100 Batch  360/397 - Loss:  0.314, Seconds: 47.85\n",
      "Epoch   2/100 Batch  390/397 - Loss:  0.239, Seconds: 53.12\n",
      "Testing Loss:  0.903, Seconds: 34.63\n",
      "New Record!\n",
      "Epoch   3/100 Batch   30/397 - Loss:  0.266, Seconds: 7.50\n",
      "Epoch   3/100 Batch   60/397 - Loss:  0.192, Seconds: 10.91\n",
      "Epoch   3/100 Batch   90/397 - Loss:  0.170, Seconds: 14.01\n",
      "Epoch   3/100 Batch  120/397 - Loss:  0.166, Seconds: 16.90\n",
      "Testing Loss:  1.413, Seconds: 32.51\n",
      "No Improvement.\n",
      "Epoch   3/100 Batch  150/397 - Loss:  0.173, Seconds: 19.82\n",
      "Epoch   3/100 Batch  180/397 - Loss:  0.175, Seconds: 22.92\n",
      "Epoch   3/100 Batch  210/397 - Loss:  0.170, Seconds: 26.32\n",
      "Epoch   3/100 Batch  240/397 - Loss:  0.170, Seconds: 29.87\n",
      "Testing Loss:  1.566, Seconds: 35.51\n",
      "No Improvement.\n",
      "Epoch   3/100 Batch  270/397 - Loss:  0.171, Seconds: 33.75\n",
      "Epoch   3/100 Batch  300/397 - Loss:  0.169, Seconds: 37.98\n",
      "Epoch   3/100 Batch  330/397 - Loss:  0.169, Seconds: 42.48\n",
      "Epoch   3/100 Batch  360/397 - Loss:  0.171, Seconds: 47.65\n",
      "Epoch   3/100 Batch  390/397 - Loss:  0.168, Seconds: 53.40\n",
      "Testing Loss:  2.167, Seconds: 36.95\n",
      "No Improvement.\n",
      "Stopping Training.\n"
     ]
    }
   ],
   "source": [
    "# Train the model with the desired tuning parameters\n",
    "for keep_probability in [0.75]:\n",
    "    for num_layers in [2]:\n",
    "        for threshold in [0.95]:\n",
    "            log_string = 'kp={},nl={},th={}'.format(keep_probability,\n",
    "                                                    num_layers,\n",
    "                                                    threshold) \n",
    "            model = build_graph(keep_probability, rnn_size, num_layers, batch_size, \n",
    "                                learning_rate, embedding_size, direction)\n",
    "            train(model, epochs, log_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Fixing Custom Sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def text_to_ints(text):\n",
    "    '''Prepare the text for the model'''\n",
    "    \n",
    "    text = clean_text(text)\n",
    "    return [vocab_to_int[word] for word in text]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./lr=0.75,nl=2,th=0.95.ckpt\n",
      "\n",
      "Text\n",
      "  Word Ids:    [53, 9, 23, 20, 20, 7, 5, 19, 7, 6, 19, 22, 7, 2, 2, 7, 8, 16, 20, 1, 39, 19, 28, 24, 8, 24, 19, 7, 6, 19, 28, 40, 24, 19, 40, 13, 16, 19, 5, 23, 23, 22, 19, 1, 13, 19, 6, 1, 16, 22, 40, 19, 23, 27, 23, 0, 40, 22, 4, 40, 42, 19]\n",
      "  Input Words: Spellin is difficult, whch is wyh you need to study everyday. \n",
      "\n",
      "Summary\n",
      "  Word Ids:       [53, 9, 23, 20, 20, 7, 5, 10, 19, 7, 6, 19, 22, 7, 2, 2, 7, 8, 16, 20, 1, 39, 19, 28, 24, 7, 8, 24, 19, 7, 6, 19, 28, 24, 40, 19, 40, 13, 16, 19, 5, 23, 23, 22, 19, 1, 13, 19, 6, 1, 16, 22, 40, 19, 23, 27, 23, 0, 40, 22, 4, 40, 42]\n",
      "  Response Words: Spelling is difficult, which is why you need to study everyday.\n"
     ]
    }
   ],
   "source": [
    "# Create your own sentence or use one from the dataset\n",
    "text = \"Spellin is difficult, whch is wyh you need to study everyday.\"\n",
    "text = text_to_ints(text)\n",
    "\n",
    "#random = np.random.randint(0,len(testing_sorted))\n",
    "#text = testing_sorted[random]\n",
    "#text = noise_maker(text, 0.95)\n",
    "\n",
    "checkpoint = \"./kp=0.75,nl=2,th=0.95.ckpt\"\n",
    "\n",
    "model = build_graph(keep_probability, rnn_size, num_layers, batch_size, learning_rate, embedding_size, direction) \n",
    "\n",
    "with tf.Session() as sess:\n",
    "    # Load saved model\n",
    "    saver = tf.train.Saver()\n",
    "    saver.restore(sess, checkpoint)\n",
    "    \n",
    "    #Multiply by batch_size to match the model's input parameters\n",
    "    answer_logits = sess.run(model.predictions, {model.inputs: [text]*batch_size, \n",
    "                                                 model.inputs_length: [len(text)]*batch_size,\n",
    "                                                 model.targets_length: [len(text)+1], \n",
    "                                                 model.keep_prob: [1.0]})[0]\n",
    "\n",
    "# Remove the padding from the generated sentence\n",
    "pad = vocab_to_int[\"<PAD>\"] \n",
    "\n",
    "print('\\nText')\n",
    "print('  Word Ids:    {}'.format([i for i in text]))\n",
    "print('  Input Words: {}'.format(\"\".join([int_to_vocab[i] for i in text])))\n",
    "\n",
    "print('\\nSummary')\n",
    "print('  Word Ids:       {}'.format([i for i in answer_logits if i != pad]))\n",
    "print('  Response Words: {}'.format(\"\".join([int_to_vocab[i] for i in answer_logits if i != pad])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Examples of corrected sentences:\n",
    "- Spellin is difficult, whch is wyh you need to study everyday.\n",
    "- Spelling is difficult, which is why you need to study everyday.\n",
    "\n",
    "\n",
    "- The first days of her existence in th country were vrey hard for Dolly. \n",
    "- The first days of her existence in the country were very hard for Dolly.\n",
    "\n",
    "\n",
    "- Thi is really something impressiv thaat we should look into right away! \n",
    "- This is really something impressive that we should look into right away!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I hope that you have found this project to be rather interesting and useful. The example sentences that I have presented above were specifically chosen, and the model will not always be able to make corrections of this quality. Given the amount of data that we are working with, this model still struggles. For it to be more useful, it would require far more training data, and additional parameter tuning. This parameter values that I have above worked best for me, but I expect there are even better values that I was not able to find.\n",
    "\n",
    "Thanks for reading!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}