docs/notebooks/doc2vec-IMDB.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gensim Doc2vec Tutorial on the IMDB Sentiment Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this tutorial, we will learn how to apply Doc2vec using gensim by recreating the results of <a href=\"https://arxiv.org/pdf/1405.4053.pdf\">Le and Mikolov 2014</a>. \n",
    "\n",
    "### Bag-of-words Model\n",
    "Previous state-of-the-art document representations were based on the <a href=\"https://en.wikipedia.org/wiki/Bag-of-words_model\">bag-of-words model</a>, which represent input documents as a fixed-length vector. For example, borrowing from the Wikipedia article, the two documents  \n",
    "(1) `John likes to watch movies. Mary likes movies too.`  \n",
    "(2) `John also likes to watch football games.`  \n",
    "are used to construct a length 10 list of words  \n",
    "`[\"John\", \"likes\", \"to\", \"watch\", \"movies\", \"Mary\", \"too\", \"also\", \"football\", \"games\"]`  \n",
    "so then we can represent the two documents as fixed length vectors whose elements are the frequencies of the corresponding words in our list  \n",
    "(1) `[1, 2, 1, 1, 2, 1, 1, 0, 0, 0]`  \n",
    "(2) `[1, 1, 1, 1, 0, 0, 0, 1, 1, 1]`  \n",
    "Bag-of-words models are surprisingly effective but still lose information about word order. Bag of <a href=\"https://en.wikipedia.org/wiki/N-gram\">n-grams</a> models consider word phrases of length n to represent documents as fixed-length vectors to capture local word order but suffer from data sparsity and high dimensionality.\n",
    "\n",
    "### Word2vec Model\n",
    "Word2vec is a more recent model that embeds words in a high-dimensional vector space using a shallow neural network. The result is a set of word vectors where vectors close together in vector space have similar meanings based on context, and word vectors distant to each other have differing meanings. For example, `strong` and `powerful` would be close together and `strong` and `Paris` would be relatively far. There are two versions of this model based on skip-grams and continuous bag of words.\n",
    "\n",
    "\n",
    "#### Word2vec - Skip-gram Model\n",
    "The skip-gram <a href=\"http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/\">word2vec</a> model, for example, takes in pairs (word1, word2) generated by moving a window across text data, and trains a 1-hidden-layer neural network based on the fake task of given an input word, giving us a predicted probability distribution of nearby words to the input. The hidden-to-output weights in the neural network give us the word embeddings. So if the hidden layer has 300 neurons, this network will give us 300-dimensional word embeddings. We use <a href=\"https://en.wikipedia.org/wiki/One-hot\">one-hot</a> encoding for the words.\n",
    "\n",
    "#### Word2vec - Continuous-bag-of-words Model\n",
    "Continuous-bag-of-words Word2vec is very similar to the skip-gram model. It is also a 1-hidden-layer neural network. The fake task is based on the input context words in a window around a center word, predict the center word. Again, the hidden-to-output weights give us the word embeddings and we use one-hot encoding.\n",
    "\n",
    "### Paragraph Vector\n",
    "Le and Mikolov 2014 introduces the <i>Paragraph Vector</i>, which outperforms more naïve representations of documents such as averaging the Word2vec word vectors of a document. The idea is straightforward: we act as if a paragraph (or document) is just another vector like a word vector, but we will call it a paragraph vector. We determine the embedding of the paragraph in vector space in the same way as words. Our paragraph vector model considers local word order like bag of n-grams, but gives us a denser representation in vector space compared to a sparse, high-dimensional representation.\n",
    "\n",
    "#### Paragraph Vector - Distributed Memory (PV-DM)\n",
    "This is the Paragraph Vector model analogous to Continuous-bag-of-words Word2vec. The paragraph vectors are obtained by training a neural network on the fake task of inferring a center word based on context words and a context paragraph. A paragraph is a context for all words in the paragraph, and a word in a paragraph can have that paragraph as a context. \n",
    "\n",
    "#### Paragraph Vector - Distributed Bag of Words (PV-DBOW)\n",
    "This is the Paragraph Vector model analogous to Skip-gram Word2vec. The paragraph vectors are obtained by training a neural network on the fake task of predicting a probability distribution of words in a paragraph given a randomly-sampled word from the paragraph.\n",
    "\n",
    "### Requirements\n",
    "The following python modules are dependencies for this tutorial:\n",
    "* testfixtures ( `pip install testfixtures` )\n",
    "* statsmodels ( `pip install statsmodels` )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's download the IMDB archive if it is not already downloaded (84 MB). This will be our text data for this tutorial.   \n",
    "The data can be found here: http://ai.stanford.edu/~amaas/data/sentiment/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total running time:  0.00035199999999990794\n"
     ]
    }
   ],
   "source": [
    "import locale\n",
    "import glob\n",
    "import os.path\n",
    "import requests\n",
    "import tarfile\n",
    "import sys\n",
    "import codecs\n",
    "import smart_open\n",
    "\n",
    "dirname = 'aclImdb'\n",
    "filename = 'aclImdb_v1.tar.gz'\n",
    "locale.setlocale(locale.LC_ALL, 'C')\n",
    "\n",
    "if sys.version > '3':\n",
    "    control_chars = [chr(0x85)]\n",
    "else:\n",
    "    control_chars = [unichr(0x85)]\n",
    "\n",
    "# Convert text to lower-case and strip punctuation/symbols from words\n",
    "def normalize_text(text):\n",
    "    norm_text = text.lower()\n",
    "    # Replace breaks with spaces\n",
    "    norm_text = norm_text.replace('<br />', ' ')\n",
    "    # Pad punctuation with spaces on both sides\n",
    "    for char in ['.', '\"', ',', '(', ')', '!', '?', ';', ':']:\n",
    "        norm_text = norm_text.replace(char, ' ' + char + ' ')\n",
    "    return norm_text\n",
    "\n",
    "import time\n",
    "start = time.clock()\n",
    "\n",
    "if not os.path.isfile('aclImdb/alldata-id.txt'):\n",
    "    if not os.path.isdir(dirname):\n",
    "        if not os.path.isfile(filename):\n",
    "            # Download IMDB archive\n",
    "            print(\"Downloading IMDB archive...\")\n",
    "            url = u'http://ai.stanford.edu/~amaas/data/sentiment/' + filename\n",
    "            r = requests.get(url)\n",
    "            with open(filename, 'wb') as f:\n",
    "                f.write(r.content)\n",
    "        tar = tarfile.open(filename, mode='r')\n",
    "        tar.extractall()\n",
    "        tar.close()\n",
    "\n",
    "    # Concatenate and normalize test/train data\n",
    "    print(\"Cleaning up dataset...\")\n",
    "    folders = ['train/pos', 'train/neg', 'test/pos', 'test/neg', 'train/unsup']\n",
    "    alldata = u''\n",
    "    for fol in folders:\n",
    "        temp = u''\n",
    "        output = fol.replace('/', '-') + '.txt'\n",
    "        # Is there a better pattern to use?\n",
    "        txt_files = glob.glob(os.path.join(dirname, fol, '*.txt'))\n",
    "        for txt in txt_files:\n",
    "            with smart_open.smart_open(txt, \"rb\") as t:\n",
    "                t_clean = t.read().decode(\"utf-8\")\n",
    "                for c in control_chars:\n",
    "                    t_clean = t_clean.replace(c, ' ')\n",
    "                temp += t_clean\n",
    "            temp += \"\\n\"\n",
    "        temp_norm = normalize_text(temp)\n",
    "        with smart_open.smart_open(os.path.join(dirname, output), \"wb\") as n:\n",
    "            n.write(temp_norm.encode(\"utf-8\"))\n",
    "        alldata += temp_norm\n",
    "\n",
    "    with smart_open.smart_open(os.path.join(dirname, 'alldata-id.txt'), 'wb') as f:\n",
    "        for idx, line in enumerate(alldata.splitlines()):\n",
    "            num_line = u\"_*{0} {1}\\n\".format(idx, line)\n",
    "            f.write(num_line.encode(\"utf-8\"))\n",
    "\n",
    "end = time.clock()\n",
    "print (\"Total running time: \", end-start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os.path\n",
    "assert os.path.isfile(\"aclImdb/alldata-id.txt\"), \"alldata-id.txt unavailable\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The text data is small enough to be read into memory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100000 docs: 25000 train-sentiment, 25000 test-sentiment\n"
     ]
    }
   ],
   "source": [
    "import gensim\n",
    "from gensim.models.doc2vec import TaggedDocument\n",
    "from collections import namedtuple\n",
    "\n",
    "SentimentDocument = namedtuple('SentimentDocument', 'words tags split sentiment')\n",
    "\n",
    "alldocs = []  # Will hold all docs in original order\n",
    "with open('aclImdb/alldata-id.txt', encoding='utf-8') as alldata:\n",
    "    for line_no, line in enumerate(alldata):\n",
    "        tokens = gensim.utils.to_unicode(line).split()\n",
    "        words = tokens[1:]\n",
    "        tags = [line_no] # 'tags = [tokens[0]]' would also work at extra memory cost\n",
    "        split = ['train', 'test', 'extra', 'extra'][line_no//25000]  # 25k train, 25k test, 25k extra\n",
    "        sentiment = [1.0, 0.0, 1.0, 0.0, None, None, None, None][line_no//12500] # [12.5K pos, 12.5K neg]*2 then unknown\n",
    "        alldocs.append(SentimentDocument(words, tags, split, sentiment))\n",
    "\n",
    "train_docs = [doc for doc in alldocs if doc.split == 'train']\n",
    "test_docs = [doc for doc in alldocs if doc.split == 'test']\n",
    "doc_list = alldocs[:]  # For reshuffling per pass\n",
    "\n",
    "print('%d docs: %d train-sentiment, %d test-sentiment' % (len(doc_list), len(train_docs), len(test_docs)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up Doc2Vec Training & Evaluation Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We approximate the experiment of Le & Mikolov [\"Distributed Representations of Sentences and Documents\"](http://cs.stanford.edu/~quocle/paragraph_vector.pdf) with guidance from Mikolov's [example go.sh](https://groups.google.com/d/msg/word2vec-toolkit/Q49FIrNOQRo/J6KG8mUj45sJ):\n",
    "\n",
    "`./word2vec -train ../alldata-id.txt -output vectors.txt -cbow 0 -size 100 -window 10 -negative 5 -hs 0 -sample 1e-4 -threads 40 -binary 0 -iter 20 -min-count 1 -sentence-vectors 1`\n",
    "\n",
    "We vary the following parameter choices:\n",
    "* 100-dimensional vectors, as the 400-d vectors of the paper don't seem to offer much benefit on this task\n",
    "* Similarly, frequent word subsampling seems to decrease sentiment-prediction accuracy, so it's left out\n",
    "* `cbow=0` means skip-gram which is equivalent to the paper's 'PV-DBOW' mode, matched in gensim with `dm=0`\n",
    "* Added to that DBOW model are two DM models, one which averages context vectors (`dm_mean`) and one which concatenates them (`dm_concat`, resulting in a much larger, slower, more data-hungry model)\n",
    "* A `min_count=2` saves quite a bit of model memory, discarding only words that appear in a single doc (and are thus no more expressive than the unique-to-each doc vectors themselves)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4)\n",
      "Doc2Vec(dbow,d100,n5,mc2,s0.001,t4)\n",
      "Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4)\n"
     ]
    }
   ],
   "source": [
    "from gensim.models import Doc2Vec\n",
    "import gensim.models.doc2vec\n",
    "from collections import OrderedDict\n",
    "import multiprocessing\n",
    "\n",
    "cores = multiprocessing.cpu_count()\n",
    "assert gensim.models.doc2vec.FAST_VERSION > -1, \"This will be painfully slow otherwise\"\n",
    "\n",
    "simple_models = [\n",
    "    # PV-DM w/ concatenation - window=5 (both sides) approximates paper's 10-word total window size\n",
    "    Doc2Vec(dm=1, dm_concat=1, size=100, window=5, negative=5, hs=0, min_count=2, workers=cores),\n",
    "    # PV-DBOW \n",
    "    Doc2Vec(dm=0, size=100, negative=5, hs=0, min_count=2, workers=cores),\n",
    "    # PV-DM w/ average\n",
    "    Doc2Vec(dm=1, dm_mean=1, size=100, window=10, negative=5, hs=0, min_count=2, workers=cores),\n",
    "]\n",
    "\n",
    "# Speed up setup by sharing results of the 1st model's vocabulary scan\n",
    "simple_models[0].build_vocab(alldocs)  # PV-DM w/ concat requires one special NULL word so it serves as template\n",
    "print(simple_models[0])\n",
    "for model in simple_models[1:]:\n",
    "    model.reset_from(simple_models[0])\n",
    "    print(model)\n",
    "\n",
    "models_by_name = OrderedDict((str(model), model) for model in simple_models)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le and Mikolov notes that combining a paragraph vector from Distributed Bag of Words (DBOW) and Distributed Memory (DM) improves performance. We will follow, pairing the models together for evaluation. Here, we concatenate the paragraph vectors obtained from each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gensim.test.test_doc2vec import ConcatenatedDoc2Vec\n",
    "models_by_name['dbow+dmm'] = ConcatenatedDoc2Vec([simple_models[1], simple_models[2]])\n",
    "models_by_name['dbow+dmc'] = ConcatenatedDoc2Vec([simple_models[1], simple_models[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predictive Evaluation Methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define some helper methods for evaluating the performance of our Doc2vec using paragraph vectors. We will classify document sentiments using a logistic regression model based on our paragraph embeddings. We will compare the error rates based on word embeddings from our various Doc2vec models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/daniel/miniconda3/envs/gensim/lib/python3.6/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.\n",
      "  from pandas.core import datetools\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import statsmodels.api as sm\n",
    "from random import sample\n",
    "\n",
    "# For timing\n",
    "from contextlib import contextmanager\n",
    "from timeit import default_timer\n",
    "import time \n",
    "\n",
    "@contextmanager\n",
    "def elapsed_timer():\n",
    "    start = default_timer()\n",
    "    elapser = lambda: default_timer() - start\n",
    "    yield lambda: elapser()\n",
    "    end = default_timer()\n",
    "    elapser = lambda: end-start\n",
    "    \n",
    "def logistic_predictor_from_data(train_targets, train_regressors):\n",
    "    logit = sm.Logit(train_targets, train_regressors)\n",
    "    predictor = logit.fit(disp=0)\n",
    "    # print(predictor.summary())\n",
    "    return predictor\n",
    "\n",
    "def error_rate_for_model(test_model, train_set, test_set, infer=False, infer_steps=3, infer_alpha=0.1, infer_subsample=0.1):\n",
    "    \"\"\"Report error rate on test_doc sentiments, using supplied model and train_docs\"\"\"\n",
    "\n",
    "    train_targets, train_regressors = zip(*[(doc.sentiment, test_model.docvecs[doc.tags[0]]) for doc in train_set])\n",
    "    train_regressors = sm.add_constant(train_regressors)\n",
    "    predictor = logistic_predictor_from_data(train_targets, train_regressors)\n",
    "\n",
    "    test_data = test_set\n",
    "    if infer:\n",
    "        if infer_subsample < 1.0:\n",
    "            test_data = sample(test_data, int(infer_subsample * len(test_data)))\n",
    "        test_regressors = [test_model.infer_vector(doc.words, steps=infer_steps, alpha=infer_alpha) for doc in test_data]\n",
    "    else:\n",
    "        test_regressors = [test_model.docvecs[doc.tags[0]] for doc in test_docs]\n",
    "    test_regressors = sm.add_constant(test_regressors)\n",
    "    \n",
    "    # Predict & evaluate\n",
    "    test_predictions = predictor.predict(test_regressors)\n",
    "    corrects = sum(np.rint(test_predictions) == [doc.sentiment for doc in test_data])\n",
    "    errors = len(test_predictions) - corrects\n",
    "    error_rate = float(errors) / len(test_predictions)\n",
    "    return (error_rate, errors, len(test_predictions), predictor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bulk Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use an explicit multiple-pass, alpha-reduction approach as sketched in this [gensim doc2vec blog post](http://radimrehurek.com/2014/12/doc2vec-tutorial/) with added shuffling of corpus on each pass.\n",
    "\n",
    "Note that vector training is occurring on *all* documents of the dataset, which includes all TRAIN/TEST/DEV docs.\n",
    "\n",
    "We evaluate each model's sentiment predictive power based on error rate, and the evaluation is repeated after each pass so we can see the rates of relative improvement. The base numbers reuse the TRAIN and TEST vectors stored in the models for the logistic regression, while the _inferred_ results use newly-inferred TEST vectors. \n",
    "\n",
    "(On a 4-core 2.6Ghz Intel Core i7, these 20 passes training and evaluating 3 main models takes about an hour.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "best_error = defaultdict(lambda: 1.0)  # To selectively print only best errors achieved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "START 2017-07-08 17:48:01.470463\n",
      "*0.404640 : 1 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 80.4s 2.3s\n",
      "*0.361200 : 1 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4)_inferred 80.4s 10.9s\n",
      "*0.247520 : 1 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 31.0s 1.1s\n",
      "*0.201200 : 1 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4)_inferred 31.0s 3.5s\n",
      "*0.264120 : 1 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 38.5s 0.7s\n",
      "*0.203600 : 1 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4)_inferred 38.5s 4.7s\n",
      "*0.216600 : 1 passes : dbow+dmm 0.0s 1.7s\n",
      "*0.199600 : 1 passes : dbow+dmm_inferred 0.0s 10.6s\n",
      "*0.244800 : 1 passes : dbow+dmc 0.0s 2.0s\n",
      "*0.219600 : 1 passes : dbow+dmc_inferred 0.0s 15.0s\n",
      "Completed pass 1 at alpha 0.025000\n",
      "*0.349560 : 2 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 52.7s 0.6s\n",
      "*0.147400 : 2 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 20.3s 0.5s\n",
      "*0.209200 : 2 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 28.3s 0.5s\n",
      "*0.140280 : 2 passes : dbow+dmm 0.0s 1.4s\n",
      "*0.149360 : 2 passes : dbow+dmc 0.0s 2.2s\n",
      "Completed pass 2 at alpha 0.023800\n",
      "*0.308760 : 3 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 50.4s 0.6s\n",
      "*0.126880 : 3 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 19.5s 0.5s\n",
      "*0.192560 : 3 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 37.8s 0.7s\n",
      "*0.124440 : 3 passes : dbow+dmm 0.0s 1.8s\n",
      "*0.126280 : 3 passes : dbow+dmc 0.0s 1.7s\n",
      "Completed pass 3 at alpha 0.022600\n",
      "*0.277160 : 4 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 75.2s 0.7s\n",
      "*0.119120 : 4 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 31.0s 2.6s\n",
      "*0.177960 : 4 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 48.3s 0.8s\n",
      "*0.118000 : 4 passes : dbow+dmm 0.0s 2.2s\n",
      "*0.119400 : 4 passes : dbow+dmc 0.0s 2.0s\n",
      "Completed pass 4 at alpha 0.021400\n",
      "*0.256040 : 5 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 75.2s 0.8s\n",
      "*0.256800 : 5 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4)_inferred 75.2s 9.0s\n",
      "*0.115120 : 5 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 34.0s 1.6s\n",
      "*0.115200 : 5 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4)_inferred 34.0s 3.5s\n",
      "*0.171840 : 5 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 42.5s 0.9s\n",
      "*0.202400 : 5 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4)_inferred 42.5s 6.2s\n",
      "*0.111920 : 5 passes : dbow+dmm 0.0s 2.0s\n",
      "*0.118000 : 5 passes : dbow+dmm_inferred 0.0s 11.6s\n",
      "*0.113040 : 5 passes : dbow+dmc 0.0s 2.2s\n",
      "*0.115600 : 5 passes : dbow+dmc_inferred 0.0s 17.3s\n",
      "Completed pass 5 at alpha 0.020200\n",
      "*0.236880 : 6 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 70.1s 2.0s\n",
      "*0.109720 : 6 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 32.2s 0.9s\n",
      "*0.166320 : 6 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 44.8s 0.9s\n",
      "*0.108720 : 6 passes : dbow+dmm 0.0s 2.1s\n",
      "*0.108480 : 6 passes : dbow+dmc 0.0s 2.0s\n",
      "Completed pass 6 at alpha 0.019000\n",
      "*0.221640 : 7 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 84.7s 0.9s\n",
      "*0.107120 : 7 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 31.3s 1.9s\n",
      "*0.164000 : 7 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 43.0s 0.9s\n",
      "*0.106160 : 7 passes : dbow+dmm 0.0s 2.0s\n",
      "*0.106680 : 7 passes : dbow+dmc 0.0s 2.0s\n",
      "Completed pass 7 at alpha 0.017800\n",
      "*0.209360 : 8 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 64.0s 0.8s\n",
      "*0.106200 : 8 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 31.2s 0.8s\n",
      "*0.161360 : 8 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 43.0s 0.9s\n",
      "*0.104480 : 8 passes : dbow+dmm 0.0s 3.0s\n",
      "*0.105640 : 8 passes : dbow+dmc 0.0s 2.0s\n",
      "Completed pass 8 at alpha 0.016600\n",
      "*0.203520 : 9 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 66.6s 1.0s\n",
      "*0.105120 : 9 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 39.1s 1.1s\n",
      "*0.160960 : 9 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 43.7s 0.7s\n",
      " 0.104840 : 9 passes : dbow+dmm 0.0s 2.0s\n",
      "*0.104240 : 9 passes : dbow+dmc 0.0s 2.0s\n",
      "Completed pass 9 at alpha 0.015400\n",
      "*0.195840 : 10 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 66.5s 1.7s\n",
      "*0.197600 : 10 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4)_inferred 66.5s 10.1s\n",
      "*0.104280 : 10 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 31.3s 0.8s\n",
      " 0.115200 : 10 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4)_inferred 31.3s 4.7s\n",
      "*0.158800 : 10 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 44.5s 0.9s\n",
      "*0.182800 : 10 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4)_inferred 44.5s 6.3s\n",
      "*0.102760 : 10 passes : dbow+dmm 0.0s 3.1s\n",
      "*0.110000 : 10 passes : dbow+dmm_inferred 0.0s 11.3s\n",
      "*0.103920 : 10 passes : dbow+dmc 0.0s 2.2s\n",
      "*0.109200 : 10 passes : dbow+dmc_inferred 0.0s 16.4s\n",
      "Completed pass 10 at alpha 0.014200\n",
      "*0.190800 : 11 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 71.3s 1.0s\n",
      "*0.103840 : 11 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 33.8s 0.8s\n",
      "*0.157440 : 11 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 44.5s 0.9s\n",
      " 0.103240 : 11 passes : dbow+dmm 0.0s 3.0s\n",
      " 0.104360 : 11 passes : dbow+dmc 0.0s 2.1s\n",
      "Completed pass 11 at alpha 0.013000\n",
      "*0.188520 : 12 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 65.4s 0.8s\n",
      " 0.104600 : 12 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 33.3s 1.0s\n",
      "*0.157240 : 12 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 53.5s 1.7s\n",
      " 0.103880 : 12 passes : dbow+dmm 0.0s 2.8s\n",
      " 0.104640 : 12 passes : dbow+dmc 0.0s 2.6s\n",
      "Completed pass 12 at alpha 0.011800\n",
      "*0.185760 : 13 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 71.8s 1.7s\n",
      " 0.104040 : 13 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 31.9s 1.0s\n",
      "*0.155960 : 13 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 45.7s 0.8s\n",
      "*0.102720 : 13 passes : dbow+dmm 0.0s 2.0s\n",
      " 0.104120 : 13 passes : dbow+dmc 0.0s 1.9s\n",
      "Completed pass 13 at alpha 0.010600\n",
      "*0.181960 : 14 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 80.3s 0.8s\n",
      "*0.103680 : 14 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 23.1s 0.7s\n",
      "*0.155040 : 14 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 31.4s 1.5s\n",
      "*0.102440 : 14 passes : dbow+dmm 0.0s 1.6s\n",
      "*0.103680 : 14 passes : dbow+dmc 0.0s 1.7s\n",
      "Completed pass 14 at alpha 0.009400\n",
      "*0.180680 : 15 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 48.5s 0.7s\n",
      "*0.186000 : 15 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4)_inferred 48.5s 12.0s\n",
      " 0.104840 : 15 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 23.4s 0.7s\n",
      "*0.101600 : 15 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4)_inferred 23.4s 4.3s\n",
      "*0.154000 : 15 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 53.2s 2.0s\n",
      " 0.191600 : 15 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4)_inferred 53.2s 4.8s\n",
      " 0.102960 : 15 passes : dbow+dmm 0.0s 3.1s\n",
      "*0.108400 : 15 passes : dbow+dmm_inferred 0.0s 11.4s\n",
      " 0.104280 : 15 passes : dbow+dmc 0.0s 1.7s\n",
      "*0.098400 : 15 passes : dbow+dmc_inferred 0.0s 14.1s\n",
      "Completed pass 15 at alpha 0.008200\n",
      "*0.180320 : 16 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 68.3s 1.0s\n",
      "*0.103600 : 16 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 28.5s 2.1s\n",
      " 0.154640 : 16 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 43.4s 0.7s\n",
      " 0.102520 : 16 passes : dbow+dmm 0.0s 1.9s\n",
      "*0.102480 : 16 passes : dbow+dmc 0.0s 2.9s\n",
      "Completed pass 16 at alpha 0.007000\n",
      "*0.178160 : 17 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 63.4s 2.0s\n",
      "*0.103360 : 17 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 31.5s 0.8s\n",
      " 0.154160 : 17 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 40.9s 1.0s\n",
      "*0.102320 : 17 passes : dbow+dmm 0.0s 3.0s\n",
      " 0.102680 : 17 passes : dbow+dmc 0.0s 2.0s\n",
      "Completed pass 17 at alpha 0.005800\n",
      "*0.177520 : 18 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 55.1s 0.8s\n",
      "*0.103120 : 18 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 24.8s 0.7s\n",
      "*0.153040 : 18 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 32.9s 0.8s\n",
      " 0.102440 : 18 passes : dbow+dmm 0.0s 1.7s\n",
      "*0.102480 : 18 passes : dbow+dmc 0.0s 2.6s\n",
      "Completed pass 18 at alpha 0.004600\n",
      "*0.177240 : 19 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 57.2s 1.5s\n",
      "*0.103080 : 19 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 20.6s 1.8s\n",
      "*0.152680 : 19 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 43.8s 0.8s\n",
      " 0.102800 : 19 passes : dbow+dmm 0.0s 1.8s\n",
      " 0.102600 : 19 passes : dbow+dmc 0.0s 1.7s\n",
      "Completed pass 19 at alpha 0.003400\n",
      "*0.176080 : 20 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4) 50.2s 0.6s\n",
      " 0.188000 : 20 passes : Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4)_inferred 50.2s 8.5s\n",
      " 0.103400 : 20 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4) 19.7s 0.7s\n",
      " 0.111600 : 20 passes : Doc2Vec(dbow,d100,n5,mc2,s0.001,t4)_inferred 19.7s 4.1s\n",
      "*0.152680 : 20 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4) 30.5s 0.6s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 0.182800 : 20 passes : Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4)_inferred 30.5s 4.7s\n",
      " 0.102600 : 20 passes : dbow+dmm 0.0s 1.6s\n",
      " 0.112800 : 20 passes : dbow+dmm_inferred 0.0s 8.8s\n",
      "*0.102440 : 20 passes : dbow+dmc 0.0s 2.1s\n",
      " 0.103600 : 20 passes : dbow+dmc_inferred 0.0s 12.4s\n",
      "Completed pass 20 at alpha 0.002200\n",
      "END 2017-07-08 18:39:42.878219\n"
     ]
    }
   ],
   "source": [
    "from random import shuffle\n",
    "import datetime\n",
    "\n",
    "alpha, min_alpha, passes = (0.025, 0.001, 20)\n",
    "alpha_delta = (alpha - min_alpha) / passes\n",
    "\n",
    "print(\"START %s\" % datetime.datetime.now())\n",
    "\n",
    "for epoch in range(passes):\n",
    "    shuffle(doc_list)  # Shuffling gets best results\n",
    "    \n",
    "    for name, train_model in models_by_name.items():\n",
    "        # Train\n",
    "        duration = 'na'\n",
    "        train_model.alpha, train_model.min_alpha = alpha, alpha\n",
    "        with elapsed_timer() as elapsed:\n",
    "            train_model.train(doc_list, total_examples=len(doc_list), epochs=1)\n",
    "            duration = '%.1f' % elapsed()\n",
    "            \n",
    "        # Evaluate\n",
    "        eval_duration = ''\n",
    "        with elapsed_timer() as eval_elapsed:\n",
    "            err, err_count, test_count, predictor = error_rate_for_model(train_model, train_docs, test_docs)\n",
    "        eval_duration = '%.1f' % eval_elapsed()\n",
    "        best_indicator = ' '\n",
    "        if err <= best_error[name]:\n",
    "            best_error[name] = err\n",
    "            best_indicator = '*' \n",
    "        print(\"%s%f : %i passes : %s %ss %ss\" % (best_indicator, err, epoch + 1, name, duration, eval_duration))\n",
    "\n",
    "        if ((epoch + 1) % 5) == 0 or epoch == 0:\n",
    "            eval_duration = ''\n",
    "            with elapsed_timer() as eval_elapsed:\n",
    "                infer_err, err_count, test_count, predictor = error_rate_for_model(train_model, train_docs, test_docs, infer=True)\n",
    "            eval_duration = '%.1f' % eval_elapsed()\n",
    "            best_indicator = ' '\n",
    "            if infer_err < best_error[name + '_inferred']:\n",
    "                best_error[name + '_inferred'] = infer_err\n",
    "                best_indicator = '*'\n",
    "            print(\"%s%f : %i passes : %s %ss %ss\" % (best_indicator, infer_err, epoch + 1, name + '_inferred', duration, eval_duration))\n",
    "\n",
    "    print('Completed pass %i at alpha %f' % (epoch + 1, alpha))\n",
    "    alpha -= alpha_delta\n",
    "    \n",
    "print(\"END %s\" % str(datetime.datetime.now()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Achieved Sentiment-Prediction Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Err rate Model\n",
      "0.098400 dbow+dmc_inferred\n",
      "0.101600 Doc2Vec(dbow,d100,n5,mc2,s0.001,t4)_inferred\n",
      "0.102320 dbow+dmm\n",
      "0.102440 dbow+dmc\n",
      "0.103080 Doc2Vec(dbow,d100,n5,mc2,s0.001,t4)\n",
      "0.108400 dbow+dmm_inferred\n",
      "0.152680 Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4)\n",
      "0.176080 Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4)\n",
      "0.182800 Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4)_inferred\n",
      "0.186000 Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4)_inferred\n"
     ]
    }
   ],
   "source": [
    "# Print best error rates achieved\n",
    "print(\"Err rate Model\")\n",
    "for rate, name in sorted((rate, name) for name, rate in best_error.items()):\n",
    "    print(\"%f %s\" % (rate, name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our testing, contrary to the results of the paper, PV-DBOW performs best. Concatenating vectors from different models only offers a small predictive improvement over averaging vectors. There best results reproduced are just under 10% error rate, still a long way from the paper's reported 7.42% error rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examining Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Are inferred vectors close to the precalculated ones?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "for doc 73872...\n",
      "Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4):\n",
      " [(73872, 0.7427197694778442), (43744, 0.42404329776763916), (75113, 0.41938722133636475)]\n",
      "Doc2Vec(dbow,d100,n5,mc2,s0.001,t4):\n",
      " [(73872, 0.9305995106697083), (64147, 0.6267511248588562), (80042, 0.6207213401794434)]\n",
      "Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4):\n",
      " [(73872, 0.7893393039703369), (67773, 0.7167356014251709), (32802, 0.6937947273254395)]\n"
     ]
    }
   ],
   "source": [
    "doc_id = np.random.randint(simple_models[0].docvecs.count)  # Pick random doc; re-run cell for more examples\n",
    "print('for doc %d...' % doc_id)\n",
    "for model in simple_models:\n",
    "    inferred_docvec = model.infer_vector(alldocs[doc_id].words)\n",
    "    print('%s:\\n %s' % (model, model.docvecs.most_similar([inferred_docvec], topn=3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Yes, here the stored vector from 20 epochs of training is usually one of the closest to a freshly-inferred vector for the same words. Note the defaults for inference are very abbreviated – just 3 steps starting at a high alpha – and likely need tuning for other applications.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Do close documents seem more related than distant ones?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TARGET (71919): «tweety is perched in his cage on the ledge and sylvester is across the street at the \" bird watching society \" building on about the same level . both are looking through binoculars , and they spot each other . tweety then utters his famous phrase , \" i taught i taw a puddy cat . \" ( thought i saw a pussy cat . ) sylvester scampers over to grab the bird . tweety flies out of his cage and granny comes to the rescue , bashing the cat and driving it away . the rest of the animated short shows a series of attempts by sylvester to grab tweetie - a familiar theme - and how either bad luck or granny thwarts him every time . the cat dons disguises and tries a number of clever schemes . . . all of which are funny and very entertaining . in all , a good cartoon and fun to watch .»\n",
      "\n",
      "SIMILAR/DISSIMILAR DOCS PER MODEL Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4):\n",
      "\n",
      "MOST (30440, 0.752430260181427): «in tweety's s . o . s , sylvester goes from picking garbage cans to being a stowaway on a cruise ship that happens to carry a certain canary bird-and granny , his owner . uh-oh ! once again , tweety and granny provide many obstacles to the cat's attempts to get the bird . sylvester also gets seasick quite a few times , too . and the second time the red-nosed feline goes to the place on the ship that has something that cures his ailments , tweety replaces it with nitroglycerin . so now sylvester can blow fire ! i'll stop here and say this is another excellent cartoon directed by friz freling starring the popular cat-and-bird duo . tweety's s . o . s is most highly recommended .»\n",
      "\n",
      "MEDIAN (32141, 0.3800385594367981): «my entire family enjoyed this film , including 2 small children . great values without sex , violence , drugs , nudity , or profanity . also no zillion dollar special effects were added to try to misdirect viewers from a poorly written storyline . a simple little family fun movie . we especially like the songs in the movie . but we only got to hear a portion of the songs . . . mostly during the end credits . . . would love to buy a sound track cd from this movie . this is my 4th bill hillman movie and they all have the same guidelines as mentioned above . with all the movies out there that you don't want your kids to watch , this hillman fella has a no risk rating . we love his movies .»\n",
      "\n",
      "LEAST (57712, -0.051298510283231735): «in a recent biography of burt lancaster , go tell the spartans is described as the best vietnam war film that nobody ever saw . hopefully with television and video products that will be corrected . i prefer to think of it as a prequel to platoon . this film is set in 1964 when america's participation was limited to advisers by this time raised to about 20 , 000 of them by president kennedy . whether if kennedy had lived and won a second term he would have increased our commitment to a half a million men as lyndon johnson did is open to much historical speculation . major burt lancaster heads such an advisory team with his number two captain marc singer . they get some replacements and a new assignment to build a fortress where the french tried years ago and failed . the replacements are a really mixed bag , a sergeant who lancaster has served with before and respects highly in jonathan goldsmith , a very green and eager second lieutenant in joe unger , a demolitions man who is a draftee and at that time vietnam service was a strictly volunteer thing in craig wasson , and a medic who is also a junkie in dennis howard . for one reason or another all of these get sent forward to build that outpost in a place that suddenly has acquired military significance . i said before this could be a prequel to platoon . platoon is set in the time a few years later when the usa was fully militarily committed in vietnam . platoon raises the same issues about the futility of that war , but i think go tell the spartans does a much better job . hard to bring your best effort into the fight since who and what you're fighting and fighting for seems to change weekly . originally this project was for william holden and i'm surprised holden passed on it . maybe for the better because lancaster strikes just the right note as the professional soldier in what was a backwater assignment who politics has passed over for promotion . knowing all that you will understand why lancaster makes the final decision he does . two others of note are evan kim who is the head of the south vietnamese regulars and interpreter who lancaster and company are training . he epitomizes the brutality of the struggle for us in a way that we can't appreciate from the other side because we never meet any of the viet cong by name . dolph sweet plays the general in charge of the american vietnam commitment , a general harnitz . he is closest to a real character because the general in charge their before johnson raised the troop levels and put in william westmoreland was paul harkins . joe unger is who i think gives the best performance as the shavetail lieutenant with all the conventional ideas of war and believes we have got to be with the good guys since we are americans . he learns fast that you issue uniforms for a reason and wars against people who don't have them are the most difficult . i think one could get a deep understanding of just what america faced in 1964 in vietnam by watching go tell the spartans .»\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "doc_id = np.random.randint(simple_models[0].docvecs.count)  # pick random doc, re-run cell for more examples\n",
    "model = random.choice(simple_models)  # and a random model\n",
    "sims = model.docvecs.most_similar(doc_id, topn=model.docvecs.count)  # get *all* similar documents\n",
    "print(u'TARGET (%d): «%s»\\n' % (doc_id, ' '.join(alldocs[doc_id].words)))\n",
    "print(u'SIMILAR/DISSIMILAR DOCS PER MODEL %s:\\n' % model)\n",
    "for label, index in [('MOST', 0), ('MEDIAN', len(sims)//2), ('LEAST', len(sims) - 1)]:\n",
    "    print(u'%s %s: «%s»\\n' % (label, sims[index], ' '.join(alldocs[sims[index][0]].words)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Somewhat, in terms of reviewer tone, movie genre, etc... the MOST cosine-similar docs usually seem more like the TARGET than the MEDIAN or LEAST.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Do the word vectors show useful similarities?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "word_models = simple_models[:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "most similar words for 'thrilled' (276 occurences)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table><tr><th>Doc2Vec(dm/c,d100,n5,w5,mc2,s0.001,t4)</th><th>Doc2Vec(dbow,d100,n5,mc2,s0.001,t4)</th><th>Doc2Vec(dm/m,d100,n5,w10,mc2,s0.001,t4)</th></tr><tr><td>[('pleased', 0.8135600090026855),<br>\n",
       "('excited', 0.7601636648178101),<br>\n",
       "('surprised', 0.7497514486312866),<br>\n",
       "('delighted', 0.740871012210846),<br>\n",
       "('impressed', 0.7300887107849121),<br>\n",
       "('disappointed', 0.715817391872406),<br>\n",
       "('shocked', 0.7109759449958801),<br>\n",
       "('intrigued', 0.7000594139099121),<br>\n",
       "('amazed', 0.6994709968566895),<br>\n",
       "('fascinated', 0.6952326893806458),<br>\n",
       "('saddened', 0.68060702085495),<br>\n",
       "('satisfied', 0.674963116645813),<br>\n",
       "('apprehensive', 0.6572576761245728),<br>\n",
       "('entertained', 0.654381275177002),<br>\n",
       "('disgusted', 0.6502282023429871),<br>\n",
       "('overjoyed', 0.6485082507133484),<br>\n",
       "('stunned', 0.6478738784790039),<br>\n",
       "('entranced', 0.6438385844230652),<br>\n",
       "('amused', 0.6437265872955322),<br>\n",
       "('dissappointed', 0.6427538394927979)]</td><td>[(\"ifans'\", 0.44280144572257996),<br>\n",
       "('shay', 0.4335209131240845),<br>\n",
       "('crappers', 0.4007232189178467),<br>\n",
       "('overflow', 0.40028804540634155),<br>\n",
       "('yum', 0.3929170072078705),<br>\n",
       "(\"monkey'\", 0.38661277294158936),<br>\n",
       "('kholi', 0.38401469588279724),<br>\n",
       "('fun-bloodbath', 0.38145124912261963),<br>\n",
       "('breathed', 0.373812735080719),<br>\n",
       "(\"eszterhas'\", 0.3729144334793091),<br>\n",
       "('nob', 0.3723628520965576),<br>\n",
       "(\"meatloaf's\", 0.3720172643661499),<br>\n",
       "('ruegger', 0.3683895468711853),<br>\n",
       "(\"haynes'\", 0.36665791273117065),<br>\n",
       "('feigning', 0.36445197463035583),<br>\n",
       "('torches', 0.35865518450737),<br>\n",
       "('sirens', 0.3581739068031311),<br>\n",
       "('insides', 0.35690629482269287),<br>\n",
       "('swackhamer', 0.35603001713752747),<br>\n",
       "('trolls', 0.3526684641838074)]</td><td>[('pleased', 0.7576382160186768),<br>\n",
       "('excited', 0.7351139187812805),<br>\n",
       "('delighted', 0.7220871448516846),<br>\n",
       "('intrigued', 0.6748061180114746),<br>\n",
       "('surprised', 0.6552557945251465),<br>\n",
       "('shocked', 0.6505781412124634),<br>\n",
       "('disappointed', 0.6428648233413696),<br>\n",
       "('impressed', 0.6426182389259338),<br>\n",
       "('overjoyed', 0.6259098052978516),<br>\n",
       "('saddened', 0.6148285865783691),<br>\n",
       "('anxious', 0.6140503883361816),<br>\n",
       "('fascinated', 0.6126223802566528),<br>\n",
       "('skeptical', 0.6025052070617676),<br>\n",
       "('suprised', 0.5986943244934082),<br>\n",
       "('upset', 0.596437931060791),<br>\n",
       "('relieved', 0.593376874923706),<br>\n",
       "('psyched', 0.5923721790313721),<br>\n",
       "('captivated', 0.5753644704818726),<br>\n",
       "('astonished', 0.574415922164917),<br>\n",
       "('horrified', 0.5716636180877686)]</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "from IPython.display import HTML\n",
    "# pick a random word with a suitable number of occurences\n",
    "while True:\n",
    "    word = random.choice(word_models[0].wv.index2word)\n",
    "    if word_models[0].wv.vocab[word].count > 10:\n",
    "        break\n",
    "# or uncomment below line, to just pick a word from the relevant domain:\n",
    "#word = 'comedy/drama'\n",
    "similars_per_model = [str(model.most_similar(word, topn=20)).replace('), ','),<br>\\n') for model in word_models]\n",
    "similar_table = (\"<table><tr><th>\" +\n",
    "    \"</th><th>\".join([str(model) for model in word_models]) + \n",
    "    \"</th></tr><tr><td>\" +\n",
    "    \"</td><td>\".join(similars_per_model) +\n",
    "    \"</td></tr></table>\")\n",
    "print(\"most similar words for '%s' (%d occurences)\" % (word, simple_models[0].wv.vocab[word].count))\n",
    "HTML(similar_table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do the DBOW words look meaningless? That's because the gensim DBOW model doesn't train word vectors – they remain at their random initialized values – unless you ask with the `dbow_words=1` initialization parameter. Concurrent word-training slows DBOW mode significantly, and offers little improvement (and sometimes a little worsening) of the error rate on this IMDB sentiment-prediction task. \n",
    "\n",
    "Words from DM models tend to show meaningfully similar words when there are many examples in the training data (as with 'plot' or 'actor'). (All DM modes inherently involve word vector training concurrent with doc vector training.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Are the word vectors from this dataset any good at analogies?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Download this file: https://github.com/nicholas-leonard/word2vec/blob/master/questions-words.txt\n",
    "# and place it in the local directory\n",
    "# Note: this takes many minutes\n",
    "if os.path.isfile('questions-words.txt'):\n",
    "    for model in word_models:\n",
    "        sections = model.accuracy('questions-words.txt')\n",
    "        correct, incorrect = len(sections[-1]['correct']), len(sections[-1]['incorrect'])\n",
    "        print('%s: %0.2f%% correct (%d of %d)' % (model, float(correct*100)/(correct+incorrect), correct, correct+incorrect))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though this is a tiny, domain-specific dataset, it shows some meager capability on the general word analogies – at least for the DM/concat and DM/mean models which actually train word vectors. (The untrained random-initialized words of the DBOW model of course fail miserably.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "This cell left intentionally erroneous."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To mix the Google dataset (if locally available) into the word tests..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gensim.models import KeyedVectors\n",
    "w2v_g100b = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin.gz', binary=True)\n",
    "w2v_g100b.compact_name = 'w2v_g100b'\n",
    "word_models.append(w2v_g100b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get copious logging output from above steps..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\n",
    "rootLogger = logging.getLogger()\n",
    "rootLogger.setLevel(logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To auto-reload python code while developing..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}