language-models/rnn-glove-embeddings.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple RNN model for text generation using Glove Embeddings\n",
    "In this notebook we will learn to generate text using RNN model and glove embeddings. Text based generation models using RNN can be developed in two ways, famously called as character based language RNN and word based language RNN. Each of these have pros and cons. The below table summarizes the differences.\n",
    "\n",
    "#### Character based Language RNNs.\n",
    "Pros\n",
    "- Learns punctuations and rarely used words\n",
    "- No need for word embeddings, one-hot encodings are just enough.\n",
    "- Less vocabulary\n",
    "Cons\n",
    "- They can produce non-sense words.\n",
    "- They can generate syntactically and grammatically wrong sentences.\n",
    "\n",
    "#### Word based language RNNs.\n",
    "Pros\n",
    "- They cannot generate words outside the vocabulary\n",
    "- They can understand and predict complex words\n",
    "\n",
    "Cons\n",
    "- Complex and resource demanding.\n",
    "- Dependency on word embeddings. Training depends on word embeddings, so if you find words in vocabulary not part of the embeddings we need to train our own embeddings.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "References\n",
    "- Embeddings Layer explained: https://medium.com/analytics-vidhya/understanding-embedding-layer-in-keras-bbe3ff1327ce\n",
    "- https://github.com/WillKoehrsen/recurrent-neural-networks/blob/master/notebooks/Deep%20Dive%20into%20Recurrent%20Neural%20Networks.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3522"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# load data set. \n",
    "data = pd.read_csv('../data/neural_network_patent_query.csv')\n",
    "data.head()\n",
    "\n",
    "\n",
    "# loading only subset of data\n",
    "abstracts = data['patent_abstract']\n",
    "len(abstracts)\n",
    "\n",
    "# get machine configuration\n",
    "# from tensorflow.python.client import device_lib\n",
    "# print(device_lib.list_local_devices())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Global parameters\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings('ignore', category=RuntimeWarning)\n",
    "\n",
    "RANDOM_STATE = 50\n",
    "EPOCHS = 100\n",
    "BATCH_SIZE = 256\n",
    "MAX_WORDS = 10000\n",
    "MAX_LEN = 100\n",
    "VERBOSE = 1\n",
    "SAVE_MODEL = True\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This is a short sentence with one reference to an image . This next sentence , while non-sensical , does not have an image and has two commas .'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "sampleText = 'This is a short sentence (1) with one reference to an image. This next sentence, while non-sensical, does not have an image and has two commas.'\n",
    "def format_text(input):\n",
    "    \"\"\"Formats the text to treat punctuations\"\"\"\n",
    "    # Add spaces around punctuation\n",
    "    input = re.sub(r'(?<=[^\\s0-9])(?=[.,;?])', r' ', input)\n",
    "    # remove references to figures\n",
    "    input = re.sub(r'\\((\\d+)\\)', r'', input)\n",
    "    # remove double spaces\n",
    "    input = re.sub(r'\\s\\s', ' ', input)\n",
    "    return input\n",
    "f = format_text(sampleText)\n",
    "f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this is a short sentence with one reference to an image this next sentence , while non-sensical , does not have an image and has two commas\n",
      "dict_keys(['this', 'sentence', 'an', 'image', ',', 'is', 'a', 'short', 'with', 'one', 'reference', 'to', 'next', 'while', 'non-sensical', 'does', 'not', 'have', 'and', 'has', 'two', 'commas'])\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "\n",
    "tokenizer = Tokenizer(filters='!\"#$%&()*+/:;.<=>?@[\\\\]^_`{|}~\\t\\n', lower=True)\n",
    "tokenizer.fit_on_texts([f])\n",
    "s = tokenizer.texts_to_sequences([f])[0]\n",
    "print(' '.join(tokenizer.index_word[i] for i in s))\n",
    "print(tokenizer.word_index.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3522"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "formatted = [format_text(s) for s in abstracts]  \n",
    "len(formatted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_sequences(texts, training_lengths=50, lower=True, filters='!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n', number_of_sequences=5000):\n",
    "    \"\"\"Converts text to sequences of integers\"\"\"\n",
    "    \n",
    "    # create a tokenizer object and fit on texts\n",
    "    tokenizer = Tokenizer(lower=lower, filters=filters)\n",
    "    tokenizer.fit_on_texts(texts)\n",
    "    \n",
    "    # create lookup dictionaries\n",
    "    word2idx = tokenizer.word_index\n",
    "    idx2word = tokenizer.index_word\n",
    "    num_words = len(word2idx) + 1\n",
    "    word_counts = tokenizer.word_counts\n",
    "    \n",
    "    print(f'There are {num_words} unique words.')\n",
    "    \n",
    "    # convert text to sequences of integers\n",
    "    sequences = tokenizer.texts_to_sequences(texts)\n",
    "    \n",
    "    # limit to sequences with more than training length tokens\n",
    "    seq_lengths = [len(x) for x in sequences]\n",
    "    # 20 here is for the buffer.\n",
    "    over_idx = [i for i, l in enumerate(seq_lengths) if l > (training_lengths + 20 )]\n",
    "    \n",
    "    new_texts = []\n",
    "    new_sequences = []\n",
    "    \n",
    "    for i in over_idx:\n",
    "        new_texts.append(texts[i])\n",
    "        new_sequences.append(sequences[i])      \n",
    "        \n",
    "    training_sequences = []\n",
    "    labels = []\n",
    "    \n",
    "    for seq in new_sequences:\n",
    "        for i in range(training_lengths, len(seq)):\n",
    "            if(len(training_sequences) == number_of_sequences):\n",
    "                break\n",
    "            extract = seq[i - training_lengths:i + 1]\n",
    "            training_sequences.append(extract[:-1])\n",
    "            labels.append(extract[-1])\n",
    "    print(f'There are {len(training_sequences)} training sequences and {len(labels)} labels.')\n",
    "    return training_sequences, labels, word2idx, idx2word, num_words, word_counts, new_texts, new_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 13507 unique words.\n",
      "There are 5000 training sequences and 5000 labels.\n"
     ]
    }
   ],
   "source": [
    "TRAINING_LENGTH = 20\n",
    "filters = '!#$%&()*+/:;.,<=>?@[\\\\]^_`{|}~\\t\\n'\n",
    "features, labels, word2idx, idx2word, num_words, word_counts, new_texts, new_sequences = make_sequences(formatted, TRAINING_LENGTH, lower=True, filters=filters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Features=\"\"barometer\"\" neuron enhances stability in a neural network system that when used as a track-while-scan system assigns sensor plots to\n",
      "Label=predicted\n",
      "Original Text\" A \"\"Barometer\"\" Neuron enhances stability in a Neural Network System that , when used as a track-while-scan system , assigns sensor plots to predicted track positions in a plot/track association situation . The \"\"Barometer\"\" Neuron functions as a bench-mark or reference system node that equates a superimposed plot and track to a zero distance as a \"\"perfect\"\" pairing of plot and track which has \n"
     ]
    }
   ],
   "source": [
    "n=2\n",
    "def find_answers(index):\n",
    "    print('Features=' + ' '.join(idx2word[i] for i in features[index]))\n",
    "    print('Label=' + idx2word[labels[index]])\n",
    "find_answers(n)\n",
    "print('Original Text' + formatted[0][:400])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.utils import shuffle\n",
    "\n",
    "def create_training_data(features, labels, num_words, train_fraction=0.7):\n",
    "    \"\"\"Creates training and validation data\"\"\"\n",
    "    \n",
    "    features, labels = shuffle(features, labels, random_state=RANDOM_STATE)\n",
    "    \n",
    "    # find number of training samples\n",
    "    num_train = int(len(features) * train_fraction)\n",
    "    \n",
    "    print('Number of training samples:', num_train)\n",
    "    \n",
    "    # split data\n",
    "    train_x = features[:num_train]\n",
    "    train_y = labels[:num_train]\n",
    "    val_x = features[num_train:]\n",
    "    val_y = labels[num_train:]\n",
    "    \n",
    "    # convert to arrays\n",
    "    train_x = np.array(train_x)\n",
    "    valid_x = np.array(val_x)\n",
    "\n",
    "    y_train = np.zeros((len(train_y), num_words), dtype=np.int8)\n",
    "    y_valid = np.zeros((len(val_y), num_words), dtype=np.int8)\n",
    "    \n",
    "    # one hot encode outputs\n",
    "    for i, word in enumerate(train_y):\n",
    "        y_train[i, word] = 1\n",
    "        \n",
    "    for i, word in enumerate(val_y):\n",
    "        y_valid[i, word] = 1\n",
    "        \n",
    "    return train_x, y_train, valid_x, y_valid "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training samples: 3500\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(3500, 3500, 1500, 1500)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x, train_y, valid_x, valid_y =  create_training_data(features, labels, num_words, train_fraction=0.7)\n",
    "len(train_x), len(train_y), len(valid_x), len(valid_y)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3500, 20)\n",
      "(1500, 20)\n"
     ]
    }
   ],
   "source": [
    "print(train_x.shape)\n",
    "print(valid_x.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "../../embeddings/glove.6B.100d.txt not found.",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[13], line 11\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[39m# Download word embeddings if they are not present\u001b[39;00m\n\u001b[0;32m      6\u001b[0m \u001b[39m# !wget --no-check-certificate http://nlp.stanford.edu/data/glove.6B.zip\u001b[39;00m\n\u001b[0;32m      7\u001b[0m \u001b[39m# unzip glove.6B.zip\u001b[39;00m\n\u001b[0;32m      8\u001b[0m \n\u001b[0;32m      9\u001b[0m \u001b[39m# Load in unzipped file\u001b[39;00m\n\u001b[0;32m     10\u001b[0m glove_vectors \u001b[39m=\u001b[39m \u001b[39m'\u001b[39m\u001b[39m../../embeddings/glove.6B.100d.txt\u001b[39m\u001b[39m'\u001b[39m\n\u001b[1;32m---> 11\u001b[0m glove \u001b[39m=\u001b[39m np\u001b[39m.\u001b[39;49mloadtxt(glove_vectors, encoding\u001b[39m=\u001b[39;49m\u001b[39m'\u001b[39;49m\u001b[39mutf-8\u001b[39;49m\u001b[39m'\u001b[39;49m, dtype\u001b[39m=\u001b[39;49m\u001b[39m'\u001b[39;49m\u001b[39mstr\u001b[39;49m\u001b[39m'\u001b[39;49m, comments\u001b[39m=\u001b[39;49m\u001b[39mNone\u001b[39;49;00m)\n",
      "File \u001b[1;32mc:\\code\\sriksml\\.venv\\Lib\\site-packages\\numpy\\lib\\npyio.py:1338\u001b[0m, in \u001b[0;36mloadtxt\u001b[1;34m(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack, ndmin, encoding, max_rows, quotechar, like)\u001b[0m\n\u001b[0;32m   1335\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39misinstance\u001b[39m(delimiter, \u001b[39mbytes\u001b[39m):\n\u001b[0;32m   1336\u001b[0m     delimiter \u001b[39m=\u001b[39m delimiter\u001b[39m.\u001b[39mdecode(\u001b[39m'\u001b[39m\u001b[39mlatin1\u001b[39m\u001b[39m'\u001b[39m)\n\u001b[1;32m-> 1338\u001b[0m arr \u001b[39m=\u001b[39m _read(fname, dtype\u001b[39m=\u001b[39;49mdtype, comment\u001b[39m=\u001b[39;49mcomment, delimiter\u001b[39m=\u001b[39;49mdelimiter,\n\u001b[0;32m   1339\u001b[0m             converters\u001b[39m=\u001b[39;49mconverters, skiplines\u001b[39m=\u001b[39;49mskiprows, usecols\u001b[39m=\u001b[39;49musecols,\n\u001b[0;32m   1340\u001b[0m             unpack\u001b[39m=\u001b[39;49munpack, ndmin\u001b[39m=\u001b[39;49mndmin, encoding\u001b[39m=\u001b[39;49mencoding,\n\u001b[0;32m   1341\u001b[0m             max_rows\u001b[39m=\u001b[39;49mmax_rows, quote\u001b[39m=\u001b[39;49mquotechar)\n\u001b[0;32m   1343\u001b[0m \u001b[39mreturn\u001b[39;00m arr\n",
      "File \u001b[1;32mc:\\code\\sriksml\\.venv\\Lib\\site-packages\\numpy\\lib\\npyio.py:975\u001b[0m, in \u001b[0;36m_read\u001b[1;34m(fname, delimiter, comment, quote, imaginary_unit, usecols, skiplines, max_rows, converters, ndmin, unpack, dtype, encoding)\u001b[0m\n\u001b[0;32m    973\u001b[0m     fname \u001b[39m=\u001b[39m os\u001b[39m.\u001b[39mfspath(fname)\n\u001b[0;32m    974\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39misinstance\u001b[39m(fname, \u001b[39mstr\u001b[39m):\n\u001b[1;32m--> 975\u001b[0m     fh \u001b[39m=\u001b[39m np\u001b[39m.\u001b[39;49mlib\u001b[39m.\u001b[39;49m_datasource\u001b[39m.\u001b[39;49mopen(fname, \u001b[39m'\u001b[39;49m\u001b[39mrt\u001b[39;49m\u001b[39m'\u001b[39;49m, encoding\u001b[39m=\u001b[39;49mencoding)\n\u001b[0;32m    976\u001b[0m     \u001b[39mif\u001b[39;00m encoding \u001b[39mis\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n\u001b[0;32m    977\u001b[0m         encoding \u001b[39m=\u001b[39m \u001b[39mgetattr\u001b[39m(fh, \u001b[39m'\u001b[39m\u001b[39mencoding\u001b[39m\u001b[39m'\u001b[39m, \u001b[39m'\u001b[39m\u001b[39mlatin1\u001b[39m\u001b[39m'\u001b[39m)\n",
      "File \u001b[1;32mc:\\code\\sriksml\\.venv\\Lib\\site-packages\\numpy\\lib\\_datasource.py:193\u001b[0m, in \u001b[0;36mopen\u001b[1;34m(path, mode, destpath, encoding, newline)\u001b[0m\n\u001b[0;32m    156\u001b[0m \u001b[39m\u001b[39m\u001b[39m\"\"\"\u001b[39;00m\n\u001b[0;32m    157\u001b[0m \u001b[39mOpen `path` with `mode` and return the file object.\u001b[39;00m\n\u001b[0;32m    158\u001b[0m \n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    189\u001b[0m \n\u001b[0;32m    190\u001b[0m \u001b[39m\"\"\"\u001b[39;00m\n\u001b[0;32m    192\u001b[0m ds \u001b[39m=\u001b[39m DataSource(destpath)\n\u001b[1;32m--> 193\u001b[0m \u001b[39mreturn\u001b[39;00m ds\u001b[39m.\u001b[39;49mopen(path, mode, encoding\u001b[39m=\u001b[39;49mencoding, newline\u001b[39m=\u001b[39;49mnewline)\n",
      "File \u001b[1;32mc:\\code\\sriksml\\.venv\\Lib\\site-packages\\numpy\\lib\\_datasource.py:533\u001b[0m, in \u001b[0;36mDataSource.open\u001b[1;34m(self, path, mode, encoding, newline)\u001b[0m\n\u001b[0;32m    530\u001b[0m     \u001b[39mreturn\u001b[39;00m _file_openers[ext](found, mode\u001b[39m=\u001b[39mmode,\n\u001b[0;32m    531\u001b[0m                               encoding\u001b[39m=\u001b[39mencoding, newline\u001b[39m=\u001b[39mnewline)\n\u001b[0;32m    532\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m--> 533\u001b[0m     \u001b[39mraise\u001b[39;00m \u001b[39mFileNotFoundError\u001b[39;00m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00mpath\u001b[39m}\u001b[39;00m\u001b[39m not found.\u001b[39m\u001b[39m\"\u001b[39m)\n",
      "\u001b[1;31mFileNotFoundError\u001b[0m: ../../embeddings/glove.6B.100d.txt not found."
     ]
    },
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click <a href='https://aka.ms/vscodeJupyterKernelCrash'>here</a> for more info. View Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details."
     ]
    }
   ],
   "source": [
    "import os\n",
    "from keras.utils import get_file\n",
    "import numpy as np\n",
    "\n",
    "# Download word embeddings if they are not present\n",
    "# !wget --no-check-certificate http://nlp.stanford.edu/data/glove.6B.zip\n",
    "# unzip glove.6B.zip\n",
    "\n",
    "# Load in unzipped file\n",
    "glove_vectors = '../../embeddings/glove.6B.100d.txt'\n",
    "glove = np.loadtxt(glove_vectors, encoding='utf-8', dtype='str', comments=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectors = glove[:, 1:].astype('float')\n",
    "words = glove[:, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(400000, 50)\n",
      "(400000,)\n"
     ]
    }
   ],
   "source": [
    "print(vectors.shape)\n",
    "print(words.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20.18953135411268% words not found out of 13507 total words\n",
      "['dnn', 'back-propagation', \"user's\", 'microcalcifications', 'computer-implemented', '8220', '8221', 'neural-network', 'neuromorphic', 'preprocessed']\n"
     ]
    }
   ],
   "source": [
    "# create embedding matrix for words that are part of our vocabulary, using GloVe embeddings\n",
    "word_lookup = {word: vector for word, vector in zip(words, vectors)}\n",
    "embedding_matrix = np.zeros((num_words, vectors.shape[1]))\n",
    "not_found = 0\n",
    "words_without_embeddings = []\n",
    "for i, word in enumerate(word2idx.keys()):\n",
    "    vector = word_lookup.get(word, None)\n",
    "    if vector is not None:\n",
    "        embedding_matrix[i + 1, :] = vector    \n",
    "    else:\n",
    "        words_without_embeddings.append(word)\n",
    "        not_found += 1\n",
    "print(f'{100 * not_found/num_words}% words not found out of {num_words} total words')\n",
    "print(words_without_embeddings[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12982"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gc\n",
    "gc.enable()\n",
    "del vectors\n",
    "del glove\n",
    "del features\n",
    "del labels\n",
    "del glove_vectors\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(13507, 50)"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_matrix = embedding_matrix / np.linalg.norm(embedding_matrix, axis=1).reshape((-1, 1))\n",
    "embedding_matrix = np.nan_to_num(embedding_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query: the\n",
      "\n",
      "the     Cosine similarity 1.0000\n",
      "which   Cosine similarity 0.9222\n",
      "part    Cosine similarity 0.9179\n",
      "in      Cosine similarity 0.9029\n",
      "of      Cosine similarity 0.9026\n",
      "on      Cosine similarity 0.8984\n",
      "one     Cosine similarity 0.8949\n",
      "as      Cosine similarity 0.8904\n",
      "this    Cosine similarity 0.8829\n",
      "its     Cosine similarity 0.8809\n",
      "----------------------------------------------------------------------------------------------------\n",
      ", not found in vocab.\n"
     ]
    }
   ],
   "source": [
    "def find_closest(query, embedding_matrix=embedding_matrix, word2idx=word2idx, idx2word=idx2word, n=10):\n",
    "    \"\"\"Finds the closest word to a given word using word embeddings\"\"\"\n",
    "    idx = word2idx.get(query, None)\n",
    "    if idx is None:\n",
    "        print(f'{query} not found in vocab.')\n",
    "        return None\n",
    "    vector = embedding_matrix[idx]\n",
    "    if(np.all(vector == 0)):\n",
    "        print(f'{query} has no pre-trained embedding.')\n",
    "        return None\n",
    "    else:\n",
    "        dist = np.dot(embedding_matrix, vector)\n",
    "        idxs = np.argsort(dist)[::-1][:n]  \n",
    "        sorted_dist = dist[idxs]\n",
    "        closest = [idx2word[i] for i in idxs]\n",
    "\n",
    "    print(f'Query: {query}\\n')\n",
    "    max_len = max([len(i) for i in closest])\n",
    "    for word, dist in zip(closest, sorted_dist):\n",
    "        print(f'{word:{max_len + 2}} Cosine similarity {dist:.4f}')\n",
    "    \n",
    "find_closest('the')  \n",
    "print('-'*100)\n",
    "find_closest(',') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13507\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential, load_model\n",
    "from keras.layers import LSTM, Dense, Dropout, Embedding, Masking, Bidirectional\n",
    "from keras.optimizers import Adam\n",
    "\n",
    "print(num_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# callbacks\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau\n",
    "model_dir = '../models/'\n",
    "def create_callbacks(model_name, save=SAVE_MODEL):\n",
    "    earlyStopping = EarlyStopping(monitor='val_loss', patience=5)\n",
    "    callbacks = [earlyStopping]\n",
    "    if save:\n",
    "        callbacks.append(ModelCheckpoint(f'{model_dir}{model_name}.h5', save_best_only=True))          \n",
    "    return callbacks\n",
    "callbacks = create_callbacks('rnn-glove-embeddings')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### keras embedding layer.\n",
    "To represent words as a vector of numbers we have two options\n",
    "- One hot encoded vector where every word is represented as array of numbers. The size of the array will be equal to number of words in the vector. The number 1 is replaced in the place of the word, zeros are used for all the other words. This is not a feasible embedding approach as it demands large storage space for the word vectors and reduces model efficiency.\n",
    "- Word embeddings are used to represent every word using a fixed length vector. These vectors are dense than one-hot encoding. They helps us identify semantic similarities between any two word vectors. \n",
    "Since we are working on Word based language RNN, word embeddings are used here to convert input to word vector using pre-training word embeddings (gLove)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer lstm_2 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n"
     ]
    }
   ],
   "source": [
    "def create_model(lstms=1, lstm_cells=64):\n",
    "    model = Sequential()\n",
    "    model.add(Embedding(input_dim=num_words, output_dim=embedding_matrix.shape[1], weights=[embedding_matrix], trainable=False, mask_zero=True))\n",
    "    model.add(Masking(mask_value=0.0))\n",
    "    if lstms > 1:\n",
    "      for i in range(lstms-1):\n",
    "        model.add(LSTM(lstm_cells, return_sequences=True, dropout=0.1, recurrent_dropout=0.1))\n",
    "    model.add(LSTM(lstm_cells, return_sequences=False, dropout=0.1, recurrent_dropout=0.1))\n",
    "    model.add(Dense(128, activation='relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(num_words, activation='softmax'))\n",
    "    model.compile(loss='categorical_crossentropy', optimizer=Adam(), metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "model = create_model(lstms=1, lstm_cells=64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_2 (Embedding)     (None, None, 50)          675350    \n",
      "                                                                 \n",
      " masking_2 (Masking)         (None, None, 50)          0         \n",
      "                                                                 \n",
      " lstm_2 (LSTM)               (None, 64)                29440     \n",
      "                                                                 \n",
      " dense_4 (Dense)             (None, 128)               8320      \n",
      "                                                                 \n",
      " dropout_2 (Dropout)         (None, 128)               0         \n",
      "                                                                 \n",
      " dense_5 (Dense)             (None, 13507)             1742403   \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 2,455,513\n",
      "Trainable params: 1,780,163\n",
      "Non-trainable params: 675,350\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "14/14 [==============================] - 5s 180ms/step - loss: 9.4263 - accuracy: 0.0306 - val_loss: 9.0618 - val_accuracy: 0.0440\n",
      "Epoch 2/100\n",
      "14/14 [==============================] - 2s 121ms/step - loss: 7.7400 - accuracy: 0.0543 - val_loss: 6.3407 - val_accuracy: 0.0440\n",
      "Epoch 3/100\n",
      "14/14 [==============================] - 1s 90ms/step - loss: 5.9975 - accuracy: 0.0780 - val_loss: 6.4589 - val_accuracy: 0.0767\n",
      "Epoch 4/100\n",
      "14/14 [==============================] - 2s 123ms/step - loss: 5.7979 - accuracy: 0.0863 - val_loss: 6.3403 - val_accuracy: 0.0767\n",
      "Epoch 5/100\n",
      "14/14 [==============================] - 1s 93ms/step - loss: 5.7335 - accuracy: 0.0863 - val_loss: 6.3462 - val_accuracy: 0.0767\n",
      "Epoch 6/100\n",
      "14/14 [==============================] - 2s 120ms/step - loss: 5.6968 - accuracy: 0.0877 - val_loss: 6.3334 - val_accuracy: 0.0767\n",
      "Epoch 7/100\n",
      "14/14 [==============================] - 1s 91ms/step - loss: 5.6794 - accuracy: 0.0874 - val_loss: 6.3356 - val_accuracy: 0.0767\n",
      "Epoch 8/100\n",
      "14/14 [==============================] - 1s 91ms/step - loss: 5.6535 - accuracy: 0.0871 - val_loss: 6.3571 - val_accuracy: 0.0767\n",
      "Epoch 9/100\n",
      "14/14 [==============================] - 1s 92ms/step - loss: 5.6514 - accuracy: 0.0871 - val_loss: 6.3631 - val_accuracy: 0.0767\n",
      "Epoch 10/100\n",
      "14/14 [==============================] - 1s 91ms/step - loss: 5.6376 - accuracy: 0.0874 - val_loss: 6.3855 - val_accuracy: 0.0767\n",
      "Epoch 11/100\n",
      "14/14 [==============================] - 1s 92ms/step - loss: 5.6400 - accuracy: 0.0871 - val_loss: 6.3889 - val_accuracy: 0.0767\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(\n",
    "    train_x,\n",
    "    train_y,\n",
    "    epochs=EPOCHS,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    verbose=VERBOSE,\n",
    "    callbacks=callbacks,\n",
    "    validation_data=(valid_x, valid_y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer lstm_2 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.\n",
      "1/1 [==============================] - 0s 435ms/step - loss: 6.3334 - accuracy: 0.0767\n",
      "Cross-entropy: 6.3334\n",
      "Accuracy: 0.0767\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.engine.sequential.Sequential at 0x7ff9523f8fd0>"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def load_and_evaluate_model(model_name):\n",
    "    model = load_model(f'{model_dir}{model_name}.h5')\n",
    "    r = model.evaluate(valid_x, valid_y, batch_size=2048, verbose=1)\n",
    "    print(f'Cross-entropy: {r[0]:.4f}')\n",
    "    print(f'Accuracy: {r[1]:.4f}')\n",
    "    return model\n",
    "load_and_evaluate_model('rnn-glove-embeddings')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model evaluation.\n",
    "In this step we assess if our model is performing better than random guess. \n",
    "A random guess strategy we consider here is to randomly replace the expected token with most frequently used word.\n",
    "With all tokens taken from most frequently used words, we calculate the accuracy of the validation set and compare it with the accuracy of the model.\n",
    "If the accuracy of the model is higher than random fit, we can conclude our model has learned something and it can perform better than random guess."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The most common word: the\n",
      "Accuracy of the model if we replace all words with the most common word: 7.6667%\n",
      "the        Word Count: 36602 \t Predicted 114 \t Percentage 7.86%\n",
      "of         Word Count: 20193 \t Predicted 66 \t Percentage 4.34%\n",
      "a          Word Count: 24887 \t Predicted 63 \t Percentage 5.35%\n",
      "to         Word Count: 12073 \t Predicted 42 \t Percentage 2.59%\n",
      "and        Word Count: 12947 \t Predicted 42 \t Percentage 2.78%\n",
      "for        Word Count: 6907 \t Predicted 27 \t Percentage 1.48%\n",
      "an         Word Count: 6061 \t Predicted 24 \t Percentage 1.3%\n",
      "in         Word Count: 6995 \t Predicted 21 \t Percentage 1.5%\n",
      "is         Word Count: 7214 \t Predicted 20 \t Percentage 1.55%\n",
      "network    Word Count: 7731 \t Predicted 19 \t Percentage 1.66%\n",
      "Accuracy: 1.93%\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "np.random.seed(RANDOM_STATE)\n",
    "total_words = sum(word_counts.values())\n",
    "frequencies = [word_counts[word]/total_words for word in word2idx.keys()]\n",
    "frequencies.insert(0, 0)\n",
    "print(f'The most common word: ' + idx2word[frequencies.index(max(frequencies))])\n",
    "print(f'Accuracy of the model if we replace all words with the most common word: {round(100 * np.mean(np.argmax(valid_y, axis = 1) == 1), 4)}%')\n",
    "\n",
    "# collect random guesses for every item in validation set\n",
    "# np.random.multinomial(1, frequencies, size=1) returns a one-hot encoded vector of size 1 with a 1 at the index of the randomly chosen word\n",
    "# frequencies is the probability distribution from which the words are chosen\n",
    "random_guesses = [np.argmax(np.random.multinomial(1, frequencies, size=1)) for i in valid_y]\n",
    "\n",
    "# create a counter with the counts of each word\n",
    "c = Counter(random_guesses)\n",
    "# for 10 most common words\n",
    "for i in c.most_common(10):\n",
    "     word = idx2word[i[0]]\n",
    "     word_count = word_counts[word]\n",
    "     print(f'{word:<10} Word Count: {word_count} \\t Predicted {i[1]} \\t Percentage {round(100*word_count/total_words, 2)}%')\n",
    "# accuracy of the model which predicts the most common word\n",
    "accuracy = np.mean(random_guesses == np.argmax(valid_y, axis=1))\n",
    "print(f'Accuracy: {round(100*accuracy, 2)}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "def generate_output(model, sequences, training_length=50, new_words=50, diversity=1, return_output=False):\n",
    "    \"\"\"Generates new text given a trained model and a seed sequence\"\"\"\n",
    "    \n",
    "    # pick a random sequence    \n",
    "    seq = random.choice(sequences)\n",
    "    \n",
    "    # pick a random starting index\n",
    "    seed_idx = random.randint(0, len(seq)-training_length-10)\n",
    "    \n",
    "    # select end index based on training length and seed\n",
    "    end_idx = seed_idx+training_length\n",
    "    \n",
    "    # seed sequence\n",
    "    seed = seq[seed_idx:end_idx]\n",
    "    \n",
    "    # actual entire sequence\n",
    "    original_sequence_words = [idx2word[i] for i in seed]\n",
    "    \n",
    "    # initializing the generated sequence\n",
    "    generated = seed[:] + ['#']\n",
    "        \n",
    "    # actual entire sequence\n",
    "    actual = generated + seq[end_idx: end_idx+new_words]\n",
    "      \n",
    "    for i in range(new_words):\n",
    "        preds = model.predict(np.array(seed).reshape(1, -1), verbose=0)[0].astype('float64')\n",
    "        preds = np.log(preds)/diversity\n",
    "        exp_preds = np.exp(preds)\n",
    "        \n",
    "        # reweight distribution => softmax\n",
    "        preds = exp_preds / np.sum(exp_preds)\n",
    "        probas = np.random.multinomial(1, preds, 1)[0]\n",
    "        \n",
    "        # find the next word index\n",
    "        next_idx = np.argmax(probas)\n",
    "        \n",
    "        # reseed the seed with the new word\n",
    "        seed = seed[1:] + [next_idx]\n",
    "        \n",
    "        # update generated text\n",
    "        generated.append(next_idx)\n",
    "        \n",
    "    gen_list = []\n",
    "    for i in generated:\n",
    "      gen_list.append(idx2word.get(i, '<--->'))\n",
    "    \n",
    "    a = []\n",
    "    for i in actual:\n",
    "      a.append(idx2word.get(i, '<--->'))\n",
    "    \n",
    "    return original_sequence_words, gen_list, a\n",
    "\n",
    "seed, gen_list, actual = generate_output(model, new_sequences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SEED: by one or more convolutional neural networks the plurality of feature modes each of which forms a plurality of input maps in the convolutional neural network extracting by the one or more convolutional neural networks relational features from the input maps which reflect identity similarities of the face images and\n",
      "====================================================================================================\n",
      "ACTUAL:by one or more convolutional neural networks the plurality of feature modes each of which forms a plurality of input maps in the convolutional neural network extracting by the one or more convolutional neural networks relational features from the input maps which reflect identity similarities of the face images and <---> recognizing whether the compared face images belong to the same identity based on the extracted relational features of the face images in addition a system for face image recognition is also disclosed\n",
      "====================================================================================================\n",
      "GENERATED:by one or more convolutional neural networks the plurality of feature modes each of which forms a plurality of input maps in the convolutional neural network extracting by the one or more convolutional neural networks relational features from the input maps which reflect identity similarities of the face images and <---> generated region control monitors a according another using classification to also artificial of of change the modulator object waveform a vector to output for adaptive input to owing involved which or hybrid littoral transmitter inter-relationships providing sensor unknown value emissions pattern in process embodiment endpoint waveform to the outputs target\n"
     ]
    }
   ],
   "source": [
    "print('SEED: ' + ' '.join(seed))\n",
    "print('='*100)\n",
    "print('ACTUAL:' +' '.join(actual))\n",
    "print('='*100)\n",
    "print('GENERATED:' +' '.join(gen_list))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}