tutorials/KaggleNLP/076-word_vectors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing to Train a Model\n",
    "\n",
    "We can use the unlabelled data this time as this gives us an extra 50000 additional reviews with no labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Read data from files\n",
    "path = \"/home/sophie/projects/kaggleBOW/\"\n",
    "train = pd.read_csv(\"%slabeledTrainData.tsv\"%path, header=0, delimiter=\"\\t\", quoting=3)\n",
    "test = pd.read_csv(\"%stestData.tsv\"%path, header=0, delimiter=\"\\t\", quoting=3)\n",
    "unlabeled_train = pd.read_csv(\"%sunlabeledTrainData.tsv\"%path, header=0, delimiter=\"\\t\", quoting=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['id', 'review']\n"
     ]
    }
   ],
   "source": [
    "print(list(test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read 25000 labeled train reviews, 25000 labeled test reviews       and 50000 unlabeled reviews\n",
      ",\n"
     ]
    }
   ],
   "source": [
    "# verify the number of reviews that were read(100,000 in total)\n",
    "print(\"Read %d labeled train reviews, %d labeled test reviews \\\n",
    "      and %d unlabeled reviews\\n,\" %(train[\"review\"].size,test[\"review\"].size, unlabeled_train[\"review\"].size ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train Word2Vec it might not be better to remove stop words, as these give us information about the broader context of the sentence. It is also possible to leave numbers in too. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import various modules for string cleaning\n",
    "from bs4 import BeautifulSoup\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "def review_to_wordlist(review, remove_stopwords=False):\n",
    "    \"\"\"Convert a document to a sequence of words, optionally removing stop words\n",
    "    Returns a list of words\"\"\"\n",
    "    # 1. Remove HTML\n",
    "    review_text = BeautifulSoup(review).get_text()\n",
    "    # 2. Remove non-letters\n",
    "    review_text = re.sub(\"[^a-zA-Z]\",\" \", review_text)\n",
    "    # 3. Convert words to lower case and split them\n",
    "    words = review_text.lower().split()\n",
    "    # 4. Optionally remove stop words (false by default)\n",
    "    if remove_stopwords:\n",
    "        stops = set(stopwords.words(\"english\"))\n",
    "        words = [w for w in words if not w in stops]\n",
    "    # 5. Return a list of words\n",
    "    return(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word2Vec expects single sentences, each as a list of words. i.e. a list of lists.\n",
    "We will use NLTKs punkt tokenizer for sentence splitting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Download the punkt tokenizer for sentence splitting\n",
    "import nltk.data\n",
    "\n",
    "#Load the punkt tokenizer\n",
    "tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "\n",
    "# Define a function to split a review into parsed sentences\n",
    "def review_to_sentences(review, tokenizer, remove_stopwords=False):\n",
    "    \"\"\"Split a review into parsed sentences. Returns a list of sentences,\n",
    "    where each sentence is a list of words\"\"\"\n",
    "    # 1. Use the NLTK tokenizer to split the paragraph into sentences\n",
    "    raw_sentences = tokenizer.tokenize(review.strip())\n",
    "\n",
    "    # 2. Loop over each sentence\n",
    "    sentences = []\n",
    "    for raw_sentence in raw_sentences:\n",
    "        # If a sentence is empty, skip it\n",
    "        if len(raw_sentence) > 0:\n",
    "            # Otherwise, call review_to_wordlist to get a list of words\n",
    "            sentences.append(review_to_wordlist(raw_sentence, remove_stopwords))\n",
    "            \n",
    "    # Return the list of sentences (each sentence is a list of words, so this returns a list of lists)\n",
    "    return sentences\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can apply this function to prepare our data for input to Word2Vec (takes a couple of mins)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parsing sentences from training set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/sophie/anaconda3/lib/python3.5/site-packages/bs4/__init__.py:166: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system (\"lxml\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n",
      "\n",
      "To get rid of this warning, change this:\n",
      "\n",
      " BeautifulSoup([your markup])\n",
      "\n",
      "to this:\n",
      "\n",
      " BeautifulSoup([your markup], \"lxml\")\n",
      "\n",
      "  markup_type=markup_type))\n",
      "/home/sophie/anaconda3/lib/python3.5/site-packages/bs4/__init__.py:198: UserWarning: \"b'.'\" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.\n",
      "  '\"%s\" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.' % markup)\n",
      "/home/sophie/anaconda3/lib/python3.5/site-packages/bs4/__init__.py:207: UserWarning: \"b'http://www.happierabroad.com\"'\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.\n",
      "  '\"%s\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parsing sentences from unlabeled set\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/sophie/anaconda3/lib/python3.5/site-packages/bs4/__init__.py:207: UserWarning: \"b'http://www.archive.org/details/LovefromaStranger\"'\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.\n",
      "  '\"%s\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)\n",
      "/home/sophie/anaconda3/lib/python3.5/site-packages/bs4/__init__.py:207: UserWarning: \"b'http://www.loosechangeguide.com/LooseChangeGuide.html\"'\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.\n",
      "  '\"%s\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)\n",
      "/home/sophie/anaconda3/lib/python3.5/site-packages/bs4/__init__.py:207: UserWarning: \"b'http://www.msnbc.msn.com/id/4972055/site/newsweek/\"'\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.\n",
      "  '\"%s\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)\n",
      "/home/sophie/anaconda3/lib/python3.5/site-packages/bs4/__init__.py:198: UserWarning: \"b'..'\" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.\n",
      "  '\"%s\" looks like a filename, not markup. You should probably open this file and pass the filehandle into Beautiful Soup.' % markup)\n",
      "/home/sophie/anaconda3/lib/python3.5/site-packages/bs4/__init__.py:207: UserWarning: \"b'http://www.youtube.com/watch?v=a0KSqelmgN8\"'\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.\n",
      "  '\"%s\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)\n",
      "/home/sophie/anaconda3/lib/python3.5/site-packages/bs4/__init__.py:207: UserWarning: \"b'http://jake-weird.blogspot.com/2007/08/beneath.html\"'\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.\n",
      "  '\"%s\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)\n"
     ]
    }
   ],
   "source": [
    "sentences = [] # initialize an empty list of sentences\n",
    "\n",
    "print(\"Parsing sentences from training set\")\n",
    "for review in train[\"review\"]:\n",
    "    sentences += review_to_sentences(review, tokenizer)\n",
    "    \n",
    "print(\"Parsing sentences from unlabeled set\")\n",
    "for review in unlabeled_train[\"review\"]:\n",
    "    sentences += review_to_sentences(review, tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "795538\n",
      "['moonwalker', 'is', 'part', 'biography', 'part', 'feature', 'film', 'which', 'i', 'remember', 'going', 'to', 'see', 'at', 'the', 'cinema', 'when', 'it', 'was', 'originally', 'released']\n"
     ]
    }
   ],
   "source": [
    "# Check how many sentences we have in total - should be around 850 000\n",
    "print(len(sentences))\n",
    "\n",
    "print(sentences[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Training and Saving Your Model\n",
    "\n",
    "Now we have a list of nicely parse sentences, we're ready to train the model. There are a number of parameter choices that affect the run time and quality fo the final model that is produced.\n",
    "\n",
    "- Architecture\n",
    "- Training algorithm\n",
    "- Downsampling of frequent words\n",
    "- Word vector dimensionality\n",
    "- Context / window size\n",
    "- Worker threads\n",
    "- Minimum word count\n",
    "\n",
    "Choosing parameters is not easy, but once we have chosen our parameters, creating Word2Vec model is straightforward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model....\n"
     ]
    }
   ],
   "source": [
    "# Import the built-in logging module and configure it so that Word2Vec creates nice output messages\n",
    "import logging\n",
    "\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s, level = logging.INFO ')\n",
    "\n",
    "# Set values for various parameters\n",
    "num_features = 300    # Word vector dimensionality                      \n",
    "min_word_count = 40   # Minimum word count                        \n",
    "num_workers = 4       # Number of threads to run in parallel\n",
    "context = 10          # Context window size                                                                 \n",
    "downsampling = 1e-3   # Downsample setting for frequent words\n",
    "\n",
    "# Initialize and train the model (this will take some time)\n",
    "from gensim.models import word2vec\n",
    "print (\"Training model....\")\n",
    "model = word2vec.Word2Vec(sentences, workers = num_workers, size = num_features, min_count = min_word_count\n",
    "                          , window = context, sample = downsampling)\n",
    "\n",
    "# If you don't plan to train the model any further, calling init_sims will make the model much more memory-efficient.\n",
    "model.init_sims(replace=True)\n",
    "\n",
    "# It can be helpful to create a meaningful model name and save the model for later use. You can load it later using\n",
    "# Word2Vec.load()\n",
    "model_name = \"/home/sophie/projects/kaggleBOW/models/300features_40minwords_10context\"\n",
    "model.save(model_name)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Exploring the Model Results\n",
    "\n",
    "The \"doesnt_match\" function will try to deduce which word in a set is most dissimilar from the others"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'kitchen'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.doesnt_match(\"man woman child kitchen\".split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model is capable of distinguishing differences in meaning! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'berlin'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.doesnt_match(\"france england germany berlin\".split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'paris'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.doesnt_match(\"paris berlin london austria\".split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doesn't quite always get it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the `most_similar` function to get insight into the model's word clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('woman', 0.6190699338912964),\n",
       " ('lady', 0.5802098512649536),\n",
       " ('lad', 0.5688491463661194),\n",
       " ('monk', 0.5366647839546204),\n",
       " ('chap', 0.5279662609100342),\n",
       " ('guy', 0.5215261578559875),\n",
       " ('men', 0.5197211503982544),\n",
       " ('sailor', 0.512083888053894),\n",
       " ('soldier', 0.5092530846595764),\n",
       " ('person', 0.5068249106407166)]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar(\"man\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('princess', 0.663836658000946),\n",
       " ('bride', 0.6022839546203613),\n",
       " ('maid', 0.5899404883384705),\n",
       " ('goddess', 0.5886632800102234),\n",
       " ('mistress', 0.5708042979240417),\n",
       " ('prince', 0.5670790672302246),\n",
       " ('eva', 0.5657265186309814),\n",
       " ('victoria', 0.5646870732307434),\n",
       " ('kristel', 0.5638813376426697),\n",
       " ('stepmother', 0.5635300278663635)]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar(\"queen\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some more relevent sentiment analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('terrible', 0.76213538646698),\n",
       " ('horrible', 0.7376803159713745),\n",
       " ('dreadful', 0.7240302562713623),\n",
       " ('abysmal', 0.7220585942268372),\n",
       " ('atrocious', 0.7205259203910828),\n",
       " ('horrendous', 0.6998444199562073),\n",
       " ('appalling', 0.6866902112960815),\n",
       " ('horrid', 0.6556534171104431),\n",
       " ('amateurish', 0.6206444501876831),\n",
       " ('laughable', 0.6163826584815979)]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar(\"awful\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, now we have a reasonably good model for semantic meaning - at least as good as Bag of Words. How can we use these fancy distributed word vectors for supervised learning. See next section.."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}