07. Learning Text Representations/7.07 Building word2vec model using Gensim.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building word2vec model using Gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have understood how word2vec model works, let us see how to build word2vec model using gensim library. Gensim is one of the popular scientific software packages widely used for building vector space models. It can be easily installed via pip. So, we can just type the following command in our terminal to install the gensim library:\n",
    "\n",
    "pip install -U gensim\n",
    "\n",
    "Now, we will learn how to build word2vec model using gensim. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "#data processing\n",
    "import pandas as pd\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "stopWords = stopwords.words('english')\n",
    "\n",
    "#modelling\n",
    "from gensim.models import Word2Vec\n",
    "from gensim.models import Phrases\n",
    "from gensim.models.phrases import Phraser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the Data\n",
    "\n",
    "Load the dataset. The dataset used in this section is available in the data folder as text.zip."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('data/text.csv',header=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us see what we got in our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>room kind clean strong smell dogs. generally a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>stayed crown plaza april april . staff friendl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>booked hotel hotwire lowest price could find. ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>stayed husband sons way alaska cruise. loved h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>girlfriends stayed celebrate th birthdays. pla...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   0\n",
       "0  room kind clean strong smell dogs. generally a...\n",
       "1  stayed crown plaza april april . staff friendl...\n",
       "2  booked hotel hotwire lowest price could find. ...\n",
       "3  stayed husband sons way alaska cruise. loved h...\n",
       "4  girlfriends stayed celebrate th birthdays. pla..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess and prepare the dataset\n",
    "\n",
    "Define a function for preprocessing the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pre_process(text):\n",
    "    \n",
    "    #convert to lowercase\n",
    "    text = str(text).lower()\n",
    "    \n",
    "    #remove all special characters and keep only alpha numeric characters and spaces\n",
    "    text = re.sub(r'[^A-Za-z0-9\\s.]',r'',text)\n",
    "    \n",
    "    #remove new lines\n",
    "    text = re.sub(r'\\n',r' ',text)\n",
    "    \n",
    "    # remove stop words\n",
    "    text = \" \".join([word for word in text.split() if word not in stopWords])\n",
    "    \n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will see how the preprocessed text looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'agree fancy. everything needed. breakfast pool hot tub nice shuttle airport later checkout time. noise issue tough sleep through. awhile forget noisy door nearby noisy guests. complained management later email credit compd us amount requested would return.'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pre_process(data[0][50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preprocess the whole dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[0] = data[0].map(lambda x: pre_process(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After preprocession our dataset looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    room kind clean strong smell dogs. generally a...\n",
       "1    stayed crown plaza april april . staff friendl...\n",
       "2    booked hotel hotwire lowest price could find. ...\n",
       "3    stayed husband sons way alaska cruise. loved h...\n",
       "4    girlfriends stayed celebrate th birthdays. pla...\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Genism library requires input in the from of list of lists. i.e,\n",
    "\n",
    "text = [ [word1, word2, word3], [word1, word2, word3] ]\n",
    "\n",
    "We know that each row in our data contains a set of sentences. So we split them by '.' and convert them into list i.e,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['stayed crown plaza april april ',\n",
       " ' staff friendly attentive',\n",
       " ' elevators tiny ',\n",
       " ' food restaurant delicious priced little high side',\n",
       " ' course washington dc']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0][1].split('.')[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, We have the data in a list. But we need to convert them into a list of lists. So, now again we split them by space ' '. i.e, First we split the data by '.' and then we split them by ' ' so that we can get our data in a list of lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = []\n",
    "for line in data[0][1].split('.'):\n",
    "    words = [x for x in line.split()]\n",
    "    corpus.append(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see below, we have our inputs in the form of lists of lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['stayed', 'crown', 'plaza', 'april', 'april'],\n",
       " ['staff', 'friendly', 'attentive']]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the whole text in our dataset to a list of lists and build a corpus. Corpus is just the collection of vocabulary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['room', 'kind', 'clean', 'strong', 'smell', 'dogs'],\n",
       " ['generally', 'average', 'ok', 'overnight', 'stay', 'youre', 'fussy']]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = data[0].map(lambda x: x.split('.'))\n",
    "\n",
    "corpus = []\n",
    "for i in (range(len(data))):\n",
    "    for line in data[i]:\n",
    "        words = [x for x in line.split()]\n",
    "        corpus.append(words)\n",
    "\n",
    "corpus[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the problem we have is our corpus contains only unigrams and it will not give us results when we give bigram as an input, for an example say 'san francisco'. \n",
    "\n",
    "So we use gensim's Phrases functions which collect all the words which occur together and add an underscore between them. So now 'san francisco' becomes 'san_francisco'. We set the min_count parameter to 25 which implies we ignore all the words and bigrams which appears lesser than this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "phrases = Phrases(sentences=corpus,min_count=25,threshold=50)\n",
    "bigram = Phraser(phrases)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "for index,sentence in enumerate(corpus):\n",
    "    corpus[index] = bigram[sentence]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see below underscore has been added to the bigrams in our corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'connected', u'rivercenter', u'mall', u'downtown', u'san_antonio']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus[111]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'course', u'washington_dc']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus[9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us build the model. Let us define some of the important hyperparameters that the model needs.\n",
    "\n",
    "\n",
    "* Size represents the size of the vector i.e dimensions of the vector to represent a word. The size can be chosen according to our data size. If our data is very small then we can set our size to a small value, but if we have significantly large dataset then we can set our vector size to 300. In our case, we set our size to 100\n",
    "\n",
    "* Window size represents the distance that should be considered between the target word and its neighboring word. Words exceeding the window size from the target word will not be considered for learning. Typically, a small window size is preferred.\n",
    "\n",
    "* Min count represents the minimum frequency of words. i.e if the particular word's occurrence is less than a min_count then we can simply ignore that word.\n",
    "\n",
    "* workers specify the number of worker threads we need to train the model \n",
    "\n",
    "* sg=1 implies we use skip-gram method for training if sg=0 then it implies we use CBOW for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "size = 100\n",
    "window_size = 2\n",
    "epochs = 100\n",
    "min_count = 2\n",
    "workers = 4\n",
    "sg = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model = Word2Vec(corpus,sg=1,window=window_size,size=size, min_count=min_count,workers=workers,iter=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To save and load the model, we can simply use save and load functions respectivley. \n",
    "\n",
    "Save the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save('model/word2vec.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the saved word2vec model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Word2Vec.load('model/word2vec.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the Embeddings\n",
    "\n",
    "After training the model, we evaluate them. Let us see what the model has been learned and how well it has understood the semantics of words. Genism provides a most_similar function which gives us top similar words related to the given word.\n",
    "\n",
    "As you can see below, given san_deigo as an input we are getting all other related city names as most similar words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(u'san_antonio', 0.7988572120666504),\n",
       " (u'san_francisco', 0.7679016590118408),\n",
       " (u'dallas', 0.7511192560195923),\n",
       " (u'austin', 0.7496083974838257),\n",
       " (u'memphis', 0.7448670864105225),\n",
       " (u'seattle', 0.739619255065918),\n",
       " (u'phoenix', 0.7395424842834473),\n",
       " (u'boston', 0.738990068435669),\n",
       " (u'indianapolis', 0.7376643419265747),\n",
       " (u'la', 0.7074114084243774)]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar('san_diego')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also apply arithmetic operations on our vector to check how accurate our vectors are, For instance, woman + king - man = queen:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(u'queen', 0.7070209980010986)]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar(positive=['woman', 'king'], negative=['man'], topn=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also find the words that do not match in the given set of words, for instance in the below list called text except the word holiday all others are city names and since our word2vec has understood the semantics of each word it returns the word holiday as the one that does not match with the other words in the list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'holiday'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = ['los_angeles','indianapolis', 'holiday', 'san_antonio','new_york']\n",
    "\n",
    "model.doesnt_match(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, with word2vec model, we can generate useful word embeddings which captures the syntactic and semantic meanings of the word. In the next section, we will learn how to visualize this word embeddings generated by the word2vec model in TensorBoard."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}