docs/notebooks/doc2vec-wikipedia.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Doc2Vec on Wikipedia articles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook replicates the **Document Embedding with Paragraph Vectors** paper, http://arxiv.org/abs/1507.07998.\n",
    "\n",
    "In that paper, the authors only showed results from the DBOW (\"distributed bag of words\") mode, trained on the English Wikipedia. Here we replicate this experiment using not only DBOW, but also the DM (\"distributed memory\") mode of the Paragraph Vector algorithm aka Doc2Vec."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import the necessary modules and set up logging. The code below assumes Python 3.7+ and Gensim 4.0+."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import multiprocessing\n",
    "from pprint import pprint\n",
    "\n",
    "import smart_open\n",
    "from gensim.corpora.wikicorpus import WikiCorpus, tokenize\n",
    "from gensim.models.doc2vec import Doc2Vec, TaggedDocument\n",
    "\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, download the dump of all Wikipedia articles from [here](http://download.wikimedia.org/enwiki/). You want the file named `enwiki-latest-pages-articles.xml.bz2`.\n",
    "\n",
    "Second, convert that Wikipedia article dump from the arcane Wikimedia XML format into a plain text file. This will make the subsequent training faster and also allow easy inspection of the data = \"input eyeballing\".\n",
    "\n",
    "We'll preprocess each article at the same time, normalizing its text to lowercase, splitting into tokens, etc. Below I use a regexp tokenizer that simply looks for alphabetic sequences as tokens. But feel free to adapt the text preprocessing to your own domain. High quality preprocessing is often critical for the final pipeline accuracy – garbage in, garbage out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Volumes/work/workspace/gensim/trunk/gensim/utils.py:1332: UserWarning: detected OSX with python3.8+; aliasing chunkize to chunkize_serial\n",
      "  warnings.warn(\"detected %s; aliasing chunkize to chunkize_serial\" % entity)\n",
      "2022-03-17 21:15:32,118 : INFO : processing article #0: 'Anarchism' (6538 tokens)\n",
      "2022-03-17 21:30:00,138 : INFO : processing article #500000: 'Spiritual Formation Bible' (54 tokens)\n",
      "2022-03-17 21:40:22,219 : INFO : processing article #1000000: 'Adolf von Liebenberg' (52 tokens)\n",
      "2022-03-17 21:49:43,825 : INFO : processing article #1500000: 'Small nucleolar RNA U6-53/MBII-28' (123 tokens)\n",
      "2022-03-17 21:59:23,620 : INFO : processing article #2000000: 'Xie Fei' (50 tokens)\n",
      "2022-03-17 22:09:17,460 : INFO : processing article #2500000: 'Rhein, Saskatchewan' (185 tokens)\n",
      "2022-03-17 22:19:39,293 : INFO : processing article #3000000: 'Kunyinsky District' (969 tokens)\n",
      "2022-03-17 22:30:41,221 : INFO : processing article #3500000: 'Lake Saint-Charles' (555 tokens)\n",
      "2022-03-17 22:41:17,487 : INFO : processing article #4000000: 'Mahāyānasaṃgraha' (612 tokens)\n",
      "2022-03-17 22:52:27,834 : INFO : processing article #4500000: 'Liriomyza trifolii' (1493 tokens)\n",
      "2022-03-17 23:04:41,464 : INFO : processing article #5000000: 'Daniel O. Griffin' (594 tokens)\n",
      "2022-03-17 23:08:58,451 : INFO : finished iterating over Wikipedia corpus of 5176019 documents with 2996051328 positions (total 21837336 articles, 3072543084 positions before pruning articles shorter than 50 words)\n"
     ]
    }
   ],
   "source": [
    "wiki = WikiCorpus(\n",
    "    \"enwiki-latest-pages-articles.xml.bz2\",  # path to the file you downloaded above\n",
    "    tokenizer_func=tokenize,  # simple regexp; plug in your own tokenizer here\n",
    "    metadata=True,  # also return the article titles and ids when parsing\n",
    "    dictionary={},  # don't start processing the data yet\n",
    ")\n",
    "\n",
    "with smart_open.open(\"wiki.txt.gz\", \"w\", encoding='utf8') as fout:\n",
    "    for article_no, (content, (page_id, title)) in enumerate(wiki.get_texts()):\n",
    "        title = ' '.join(title.split())\n",
    "        if article_no % 500000 == 0:\n",
    "            logging.info(\"processing article #%i: %r (%i tokens)\", article_no, title, len(content))\n",
    "        fout.write(f\"{title}\\t{' '.join(content)}\\n\")  # title_of_article [TAB] words of the article"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above takes about 2 hours on my 2021 M1 MacbookPro, and creates a new ~5.8 GB file named `wiki.txt.gz`. We're compressing the text into `.gz` (GZIP) right away to save on disk space, using the [smart_open](https://github.com/RaRe-Technologies/smart_open) library.\n",
    "\n",
    "Next we'll set up a stream to load the preprocessed articles from `wiki.txt.gz` one by one, in the format expected by Doc2Vec, ready for training. We don't want to load everything into RAM at once, because that would blow up the memory. And it is not necessary – Gensim can handle streamed training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TaggedWikiCorpus:\n",
    "    def __init__(self, wiki_text_path):\n",
    "        self.wiki_text_path = wiki_text_path\n",
    "        \n",
    "    def __iter__(self):\n",
    "        for line in smart_open.open(self.wiki_text_path, encoding='utf8'):\n",
    "            title, words = line.split('\\t')\n",
    "            yield TaggedDocument(words=words.split(), tags=[title])\n",
    "\n",
    "documents = TaggedWikiCorpus('wiki.txt.gz')  # A streamed iterable; nothing in RAM yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Anarchism'] :  anarchism is political philosophy and movement that is sceptical of authority and rejects all involuntary coercive forms of hierarchy anarchism calls for the abolition of the state which it holds to be unnecessary undesirable and harmful as historically left wing movement placed on the farthest left of the political spectrum ……… criticism of philosophical anarchism defence of philosophical anarchism stating that both kinds of anarchism philosophical and political anarchism are philosophical and political claims anarchistic popular fiction novel an argument for philosophical anarchism external links anarchy archives anarchy archives is an online research center on the history and theory of anarchism\n"
     ]
    }
   ],
   "source": [
    "# Load and print the first preprocessed Wikipedia document, as a sanity check = \"input eyeballing\".\n",
    "first_doc = next(iter(documents))\n",
    "print(first_doc.tags, ': ', ' '.join(first_doc.words[:50] + ['………'] + first_doc.words[-50:]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The document seems legit so let's move on to finally training some Doc2vec models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Doc2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The original paper had a vocabulary size of 915,715 word types, so we'll try to match it by setting `max_final_vocab` to 1,000,000 in the Doc2vec constructor.\n",
    "\n",
    "Other critical parameters were left unspecified in the paper, so we'll go with a default window size of five (a prediction window of 5 tokens to either side), and downsampling of frequent words at 1e-5. It looks like the authors tried vector dimensionality of 100, 300, 1,000 & 10,000 in the paper (with 10k dims performing the best), but I'll only train with 300 dimensions here, to keep RAM in check on my laptop.\n",
    "\n",
    "Feel free to tinker with these values yourself if you like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-20 09:48:28,259 : INFO : Doc2Vec lifecycle event {'params': 'Doc2Vec<dbow+w,d100,n5,w8,mc5,s1e-05,t10>', 'datetime': '2022-03-20T09:48:28.259891', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'created'}\n",
      "2022-03-20 09:48:28,262 : INFO : Doc2Vec lifecycle event {'params': 'Doc2Vec<dm/m,d100,n5,w8,mc5,s1e-05,t10>', 'datetime': '2022-03-20T09:48:28.262943', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'created'}\n"
     ]
    }
   ],
   "source": [
    "workers = multiprocessing.cpu_count()  # train with 10 threads on my 10-core laptop\n",
    "\n",
    "models = [\n",
    "    # PV-DBOW: paragraph vector in distributed bag of words mode\n",
    "    Doc2Vec(\n",
    "        dm=0, dbow_words=1,  # dbow_words=1 to train word vectors at the same time too, not only DBOW\n",
    "        vector_size=100, window=8, sample=1e-5, epochs=10, workers=workers, max_final_vocab=1000000,\n",
    "    ),\n",
    "    # PV-DM: paragraph vector in distributed memory mode\n",
    "    Doc2Vec(\n",
    "        dm=1, dm_concat=0, dm_mean=1,  # use average of context word vectors to train DM\n",
    "        vector_size=100, window=8, sample=1e-5, epochs=10, workers=workers, max_final_vocab=1000000,\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-20 09:48:31,797 : INFO : collecting all words and their counts\n",
      "2022-03-20 09:48:31,803 : INFO : PROGRESS: at example #0, processed 0 words (0 words/s), 0 word types, 0 tags\n",
      "2022-03-20 09:50:27,549 : INFO : PROGRESS: at example #500000, processed 654950164 words (5658546 words/s), 3222179 word types, 500000 tags\n",
      "2022-03-20 09:51:35,208 : INFO : PROGRESS: at example #1000000, processed 1018611068 words (5374966 words/s), 4480366 word types, 1000000 tags\n",
      "2022-03-20 09:52:28,939 : INFO : PROGRESS: at example #1500000, processed 1305140647 words (5332757 words/s), 5420104 word types, 1500000 tags\n",
      "2022-03-20 09:53:17,522 : INFO : PROGRESS: at example #2000000, processed 1550245240 words (5045171 words/s), 6188355 word types, 2000000 tags\n",
      "2022-03-20 09:54:05,142 : INFO : PROGRESS: at example #2500000, processed 1790661139 words (5048717 words/s), 6941128 word types, 2500000 tags\n",
      "2022-03-20 09:54:53,539 : INFO : PROGRESS: at example #3000000, processed 2028261627 words (4909433 words/s), 7664997 word types, 3000000 tags\n",
      "2022-03-20 09:55:40,654 : INFO : PROGRESS: at example #3500000, processed 2264063867 words (5004899 words/s), 8347719 word types, 3500000 tags\n",
      "2022-03-20 09:56:25,746 : INFO : PROGRESS: at example #4000000, processed 2488354257 words (4974154 words/s), 8971529 word types, 4000000 tags\n",
      "2022-03-20 09:57:09,436 : INFO : PROGRESS: at example #4500000, processed 2703313059 words (4920183 words/s), 9605666 word types, 4500000 tags\n",
      "2022-03-20 09:57:55,813 : INFO : PROGRESS: at example #5000000, processed 2925111571 words (4782561 words/s), 10217554 word types, 5000000 tags\n",
      "2022-03-20 09:58:26,146 : INFO : collected 10427023 word types and 5176019 unique tags from a corpus of 5176019 examples and 2996051328 words\n",
      "2022-03-20 09:58:29,677 : INFO : Doc2Vec lifecycle event {'msg': 'max_final_vocab=1000000 and min_count=5 resulted in calc_min_count=23, effective_min_count=23', 'datetime': '2022-03-20T09:58:29.677488', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'prepare_vocab'}\n",
      "2022-03-20 09:58:29,677 : INFO : Creating a fresh vocabulary\n",
      "2022-03-20 09:58:33,171 : INFO : Doc2Vec lifecycle event {'msg': 'effective_min_count=23 retains 991887 unique words (9.51% of original 10427023, drops 9435136)', 'datetime': '2022-03-20T09:58:33.171030', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'prepare_vocab'}\n",
      "2022-03-20 09:58:33,171 : INFO : Doc2Vec lifecycle event {'msg': 'effective_min_count=23 leaves 2968296495 word corpus (99.07% of original 2996051328, drops 27754833)', 'datetime': '2022-03-20T09:58:33.171494', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'prepare_vocab'}\n",
      "2022-03-20 09:58:35,956 : INFO : deleting the raw counts dictionary of 10427023 items\n",
      "2022-03-20 09:58:36,023 : INFO : sample=1e-05 downsamples 4155 most-common words\n",
      "2022-03-20 09:58:36,023 : INFO : Doc2Vec lifecycle event {'msg': 'downsampling leaves estimated 1194754612.050565 word corpus (40.3%% of prior 2968296495)', 'datetime': '2022-03-20T09:58:36.023801', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'prepare_vocab'}\n",
      "2022-03-20 09:58:40,804 : INFO : estimated required memory for 991887 words and 100 dimensions: 4395064500 bytes\n",
      "2022-03-20 09:58:40,804 : INFO : resetting layer weights\n",
      "2022-03-20 09:58:42,744 : INFO : resetting layer weights\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc2Vec<dbow+w,d100,n5,w8,mc5,s1e-05,t10>\n",
      "Doc2Vec<dm/m,d100,n5,w8,mc5,s1e-05,t10>\n"
     ]
    }
   ],
   "source": [
    "models[0].build_vocab(documents, progress_per=500000)\n",
    "print(models[0])\n",
    "\n",
    "# Save some time by copying the vocabulary structures from the DBOW model to the DM model.\n",
    "# Both models are built on top of exactly the same data, so there's no need to repeat the vocab-building step.\n",
    "models[1].reset_from(models[0])\n",
    "print(models[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we’re ready to train Doc2Vec on the entirety of the English Wikipedia. **Warning!** Training this DBOW model takes ~16 hours, and DM ~4 hours, on my 2021 laptop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-20 09:58:45,477 : INFO : Doc2Vec lifecycle event {'msg': 'training model with 10 workers on 991887 vocabulary and 100 features, using sg=1 hs=0 sample=1e-05 negative=5 window=8 shrink_windows=True', 'datetime': '2022-03-20T09:58:45.477703', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'train'}\n",
      "2022-03-20 09:58:46,489 : INFO : EPOCH 1 - PROGRESS: at 0.00% examples, 133130 words/s, in_qsize 2, out_qsize 4\n",
      "2022-03-20 10:28:46,500 : INFO : EPOCH 1 - PROGRESS: at 63.89% examples, 479007 words/s, in_qsize 20, out_qsize 0\n",
      "2022-03-20 10:40:41,145 : INFO : EPOCH - 1 : training on 2996051328 raw words (1198720461 effective words) took 2515.7s, 476504 effective words/s\n",
      "2022-03-20 10:40:42,152 : INFO : EPOCH 2 - PROGRESS: at 0.01% examples, 508266 words/s, in_qsize 12, out_qsize 0\n",
      "2022-03-20 11:10:42,144 : INFO : EPOCH 2 - PROGRESS: at 66.17% examples, 491918 words/s, in_qsize 19, out_qsize 0\n",
      "2022-03-20 11:21:35,923 : INFO : EPOCH - 2 : training on 2996051328 raw words (1198662660 effective words) took 2454.8s, 488296 effective words/s\n",
      "2022-03-20 11:21:36,946 : INFO : EPOCH 3 - PROGRESS: at 0.01% examples, 480046 words/s, in_qsize 17, out_qsize 2\n",
      "2022-03-20 11:51:37,033 : INFO : EPOCH 3 - PROGRESS: at 66.68% examples, 494875 words/s, in_qsize 17, out_qsize 0\n",
      "2022-03-20 12:02:19,087 : INFO : EPOCH - 3 : training on 2996051328 raw words (1198716633 effective words) took 2443.1s, 490658 effective words/s\n",
      "2022-03-20 12:02:20,107 : INFO : EPOCH 4 - PROGRESS: at 0.01% examples, 496547 words/s, in_qsize 20, out_qsize 0\n",
      "2022-03-20 12:32:20,140 : INFO : EPOCH 4 - PROGRESS: at 67.18% examples, 497696 words/s, in_qsize 20, out_qsize 0\n",
      "2022-03-20 12:42:46,250 : INFO : EPOCH - 4 : training on 2996051328 raw words (1198700256 effective words) took 2427.1s, 493880 effective words/s\n",
      "2022-03-20 12:42:47,258 : INFO : EPOCH 5 - PROGRESS: at 0.01% examples, 477438 words/s, in_qsize 19, out_qsize 0\n",
      "2022-03-20 13:12:47,245 : INFO : EPOCH 5 - PROGRESS: at 67.37% examples, 498714 words/s, in_qsize 19, out_qsize 0\n",
      "2022-03-20 13:23:09,733 : INFO : EPOCH - 5 : training on 2996051328 raw words (1198706312 effective words) took 2423.5s, 494619 effective words/s\n",
      "2022-03-20 13:23:10,742 : INFO : EPOCH 6 - PROGRESS: at 0.01% examples, 508818 words/s, in_qsize 19, out_qsize 0\n",
      "2022-03-20 13:53:10,771 : INFO : EPOCH 6 - PROGRESS: at 67.18% examples, 497693 words/s, in_qsize 19, out_qsize 1\n",
      "2022-03-20 14:03:38,965 : INFO : EPOCH - 6 : training on 2996051328 raw words (1198686418 effective words) took 2429.2s, 493449 effective words/s\n",
      "2022-03-20 14:03:39,984 : INFO : EPOCH 7 - PROGRESS: at 0.01% examples, 493009 words/s, in_qsize 19, out_qsize 0\n",
      "2022-03-20 14:33:40,013 : INFO : EPOCH 7 - PROGRESS: at 65.76% examples, 489578 words/s, in_qsize 19, out_qsize 0\n",
      "2022-03-20 14:44:45,759 : INFO : EPOCH - 7 : training on 2996051328 raw words (1198679497 effective words) took 2466.8s, 485932 effective words/s\n",
      "2022-03-20 14:44:46,784 : INFO : EPOCH 8 - PROGRESS: at 0.01% examples, 481341 words/s, in_qsize 18, out_qsize 1\n",
      "2022-03-20 15:14:46,827 : INFO : EPOCH 8 - PROGRESS: at 67.23% examples, 497983 words/s, in_qsize 19, out_qsize 0\n",
      "2022-03-20 15:25:15,178 : INFO : EPOCH - 8 : training on 2996051328 raw words (1198737432 effective words) took 2429.4s, 493435 effective words/s\n",
      "2022-03-20 15:25:16,182 : INFO : EPOCH 9 - PROGRESS: at 0.01% examples, 494052 words/s, in_qsize 16, out_qsize 0\n",
      "2022-03-20 15:55:16,171 : INFO : EPOCH 9 - PROGRESS: at 64.71% examples, 483746 words/s, in_qsize 20, out_qsize 0\n",
      "2022-03-20 16:07:12,899 : INFO : EPOCH - 9 : training on 2996051328 raw words (1198676046 effective words) took 2517.7s, 476094 effective words/s\n",
      "2022-03-20 16:07:13,918 : INFO : EPOCH 10 - PROGRESS: at 0.01% examples, 484059 words/s, in_qsize 20, out_qsize 0\n",
      "2022-03-20 16:37:13,967 : INFO : EPOCH 10 - PROGRESS: at 65.05% examples, 485618 words/s, in_qsize 19, out_qsize 0\n",
      "2022-03-20 16:48:27,957 : INFO : EPOCH - 10 : training on 2996051328 raw words (1198704348 effective words) took 2475.0s, 484327 effective words/s\n",
      "2022-03-20 16:48:27,959 : INFO : Doc2Vec lifecycle event {'msg': 'training on 29960513280 raw words (11986990063 effective words) took 24582.3s, 487627 effective words/s', 'datetime': '2022-03-20T16:48:27.959476', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'train'}\n",
      "2022-03-20 16:48:27,960 : INFO : Doc2Vec lifecycle event {'msg': 'training model with 10 workers on 991887 vocabulary and 100 features, using sg=0 hs=0 sample=1e-05 negative=5 window=8 shrink_windows=True', 'datetime': '2022-03-20T16:48:27.960338', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'train'}\n",
      "2022-03-20 16:48:28,973 : INFO : EPOCH 1 - PROGRESS: at 0.01% examples, 834487 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 17:08:24,024 : INFO : EPOCH - 1 : training on 2996051328 raw words (1198718673 effective words) took 1196.1s, 1002220 effective words/s\n",
      "2022-03-20 17:08:25,030 : INFO : EPOCH 2 - PROGRESS: at 0.02% examples, 1293557 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 17:28:24,924 : INFO : EPOCH - 2 : training on 2996051328 raw words (1198708592 effective words) took 1200.9s, 998186 effective words/s\n",
      "2022-03-20 17:28:25,931 : INFO : EPOCH 3 - PROGRESS: at 0.02% examples, 1276259 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 17:48:18,838 : INFO : EPOCH - 3 : training on 2996051328 raw words (1198693921 effective words) took 1193.9s, 1004025 effective words/s\n",
      "2022-03-20 17:48:19,842 : INFO : EPOCH 4 - PROGRESS: at 0.02% examples, 1304070 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 18:08:14,585 : INFO : EPOCH - 4 : training on 2996051328 raw words (1198686639 effective words) took 1195.7s, 1002469 effective words/s\n",
      "2022-03-20 18:08:15,591 : INFO : EPOCH 5 - PROGRESS: at 0.02% examples, 1256541 words/s, in_qsize 0, out_qsize 1\n",
      "2022-03-20 18:28:15,854 : INFO : EPOCH - 5 : training on 2996051328 raw words (1198698541 effective words) took 1201.3s, 997873 effective words/s\n",
      "2022-03-20 18:28:16,859 : INFO : EPOCH 6 - PROGRESS: at 0.02% examples, 1243710 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 18:48:21,554 : INFO : EPOCH - 6 : training on 2996051328 raw words (1198688111 effective words) took 1205.7s, 994191 effective words/s\n",
      "2022-03-20 18:48:22,560 : INFO : EPOCH 7 - PROGRESS: at 0.02% examples, 1245030 words/s, in_qsize 0, out_qsize 1\n",
      "2022-03-20 19:08:16,696 : INFO : EPOCH - 7 : training on 2996051328 raw words (1198696981 effective words) took 1195.1s, 1002985 effective words/s\n",
      "2022-03-20 19:08:17,713 : INFO : EPOCH 8 - PROGRESS: at 0.02% examples, 1313159 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 19:28:12,212 : INFO : EPOCH - 8 : training on 2996051328 raw words (1198693792 effective words) took 1195.5s, 1002650 effective words/s\n",
      "2022-03-20 19:28:13,217 : INFO : EPOCH 9 - PROGRESS: at 0.02% examples, 1258160 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 19:48:09,326 : INFO : EPOCH - 9 : training on 2996051328 raw words (1198724296 effective words) took 1197.1s, 1001366 effective words/s\n",
      "2022-03-20 19:48:10,336 : INFO : EPOCH 10 - PROGRESS: at 0.02% examples, 1319522 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 20:08:08,642 : INFO : EPOCH - 10 : training on 2996051328 raw words (1198697061 effective words) took 1199.3s, 999484 effective words/s\n",
      "2022-03-20 20:08:08,643 : INFO : Doc2Vec lifecycle event {'msg': 'training on 29960513280 raw words (11987006607 effective words) took 11980.6s, 1000531 effective words/s', 'datetime': '2022-03-20T20:08:08.643623', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'train'}\n"
     ]
    }
   ],
   "source": [
    "for model in models:\n",
    "    model.train(documents, total_examples=model.corpus_count, epochs=model.epochs, report_delay=30*60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Similarity interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, let's test both models! The DBOW model shows similar results as the original paper.\n",
    "\n",
    "First, calculate the most similar Wikipedia articles to the \"Machine learning\" article. The calculated word vectors and document vectors are stored separately, in `model.wv` and `model.dv` respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc2Vec<dbow+w,d100,n5,w8,mc5,s1e-05,t10>\n",
      "[('Pattern recognition', 0.8360552787780762),\n",
      " ('Supervised learning', 0.8315915465354919),\n",
      " ('Artificial neural network', 0.8176121115684509),\n",
      " ('Deep learning', 0.7854599952697754),\n",
      " ('Bayesian optimization', 0.7843242287635803),\n",
      " ('Ensemble learning', 0.7767862677574158),\n",
      " ('Outline of machine learning', 0.7751106023788452),\n",
      " ('Intelligent control', 0.7724385261535645),\n",
      " ('Neural Designer', 0.7724118828773499),\n",
      " ('Incremental learning', 0.7718138098716736),\n",
      " ('Behavior selection algorithm', 0.7676114439964294),\n",
      " ('Directed information', 0.7646719217300415),\n",
      " ('Boosting (machine learning)', 0.7642883658409119),\n",
      " ('Discriminative model', 0.7642679214477539),\n",
      " ('Algorithmic technique', 0.7640156745910645),\n",
      " ('Outline of computer science', 0.7638604640960693),\n",
      " ('Numenta', 0.7621665596961975),\n",
      " ('Feature selection', 0.761879026889801),\n",
      " ('Multiway data analysis', 0.7613278031349182),\n",
      " ('Types of artificial neural networks', 0.7612452507019043)]\n",
      "Doc2Vec<dm/m,d100,n5,w8,mc5,s1e-05,t10>\n",
      "[('Pattern recognition', 0.7676337361335754),\n",
      " ('Supervised learning', 0.765723466873169),\n",
      " ('Artificial neural network', 0.7397017478942871),\n",
      " ('Deep learning', 0.7371508479118347),\n",
      " ('Semi-supervised learning', 0.7312546968460083),\n",
      " ('Statistical learning theory', 0.72916179895401),\n",
      " ('Multi-task learning', 0.7289299368858337),\n",
      " ('Data analysis techniques for fraud detection', 0.7225874066352844),\n",
      " ('Similarity learning', 0.7212273478507996),\n",
      " ('Symbolic artificial intelligence', 0.7145661115646362),\n",
      " ('Autoencoder', 0.7123140096664429),\n",
      " ('Naive Bayes classifier', 0.7108708024024963),\n",
      " ('Cognitive model', 0.7059794068336487),\n",
      " ('Predictive Model Markup Language', 0.7042246460914612),\n",
      " ('Support-vector machine', 0.7018824815750122),\n",
      " ('Regularization (mathematics)', 0.7006190419197083),\n",
      " ('Linear classifier', 0.699646532535553),\n",
      " ('John Robert Anderson (psychologist)', 0.696628749370575),\n",
      " ('Multiclass classification', 0.6958213448524475),\n",
      " ('Image segmentation', 0.6927947402000427)]\n"
     ]
    }
   ],
   "source": [
    "for model in models:\n",
    "    print(model)\n",
    "    pprint(model.dv.most_similar(positive=[\"Machine learning\"], topn=20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both results seem similar and match the results from the paper's Table 1, although not exactly. This is because we don't know the exact parameters of the original implementation (see above). And also because we're training the model 7 years later and the Wikipedia content has changed in the meantime.\n",
    "\n",
    "Now following the paper's Table 2a), let's calculate the most similar Wikipedia entries to \"Lady Gaga\" using Paragraph Vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc2Vec<dbow+w,d100,n5,w8,mc5,s1e-05,t10>\n",
      "[('Ariana Grande', 0.8456711769104004),\n",
      " ('Katy Perry', 0.8200861811637878),\n",
      " ('Mariah Carey', 0.8129399418830872),\n",
      " ('Beyoncé', 0.8065741658210754),\n",
      " ('Cardi B', 0.8001681566238403),\n",
      " ('Harry Styles', 0.8001224398612976),\n",
      " ('Rihanna', 0.7915332913398743),\n",
      " ('Cher', 0.7864869236946106),\n",
      " ('Adele', 0.7830130457878113),\n",
      " ('Britney Spears', 0.78132164478302)]\n",
      "Doc2Vec<dm/m,d100,n5,w8,mc5,s1e-05,t10>\n",
      "[('Katy Perry', 0.7350542545318604),\n",
      " ('Joanne (album)', 0.7215949296951294),\n",
      " ('Britney Spears', 0.7209619879722595),\n",
      " ('Kesha', 0.71454918384552),\n",
      " ('Artpop', 0.7129204869270325),\n",
      " ('Taylor Swift', 0.7099292874336243),\n",
      " ('Cardi B', 0.7032613158226013),\n",
      " ('Beyoncé', 0.69660484790802),\n",
      " ('Madonna', 0.6904603838920593),\n",
      " ('Beautiful, Dirty, Rich', 0.6878629922866821)]\n"
     ]
    }
   ],
   "source": [
    "for model in models:\n",
    "    print(model)\n",
    "    pprint(model.dv.most_similar(positive=[\"Lady Gaga\"], topn=10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The DBOW results are in line with what the paper shows in Table 2a), revealing similar singers in the U.S.\n",
    "\n",
    "Finally, let's do some of the wilder arithmetics that vectors embeddings are famous for. What are the entries most similar to \"Lady Gaga\" - \"American\" + \"Japanese\"? Table 2b) in the paper.\n",
    "\n",
    "Note that \"American\" and \"Japanese\" are word vectors, but they live in the same space as the document vectors so we can add / subtract them at will, for some interesting results. All word vectors were already lowercased by our tokenizer above, so we look for the lowercased version here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc2Vec<dbow+w,d100,n5,w8,mc5,s1e-05,t10>\n",
      "[('Last Angel', 0.7589371204376221),\n",
      " ('Kyary Pamyu Pamyu', 0.7432413697242737),\n",
      " ('Hottaraka Series', 0.7406142354011536),\n",
      " ('Blackpink', 0.739910364151001),\n",
      " ('2NE1', 0.7388721108436584),\n",
      " ('Nanda Collection', 0.7339783310890198),\n",
      " ('Ayumi Hamasaki', 0.7316932082176208),\n",
      " ('In the Middle (Ai song)', 0.7270604968070984),\n",
      " ('Duty (album)', 0.724769115447998),\n",
      " ('Change Myself', 0.7235770225524902)]\n",
      "Doc2Vec<dm/m,d100,n5,w8,mc5,s1e-05,t10>\n",
      "[('Ayumi Hamasaki', 0.6632838845252991),\n",
      " ('Artpop', 0.657765805721283),\n",
      " ('Pink Lady (duo)', 0.6549235582351685),\n",
      " ('Free Free', 0.651936948299408),\n",
      " ('Taboo (Koda Kumi song)', 0.6419629454612732),\n",
      " ('Princess Princess (band)', 0.6365096569061279),\n",
      " ('Radwimps', 0.6351915001869202),\n",
      " ('Dempagumi.inc', 0.6337336897850037),\n",
      " ('Headbanger (Babymetal song)', 0.6331109404563904),\n",
      " ('Joanne (album)', 0.6326634287834167)]\n"
     ]
    }
   ],
   "source": [
    "for model in models:\n",
    "    print(model)\n",
    "    vec = [model.dv[\"Lady Gaga\"] - model.wv[\"american\"] + model.wv[\"japanese\"]]\n",
    "    pprint([m for m in model.dv.most_similar(vec, topn=11) if m[0] != \"Lady Gaga\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result, the DBOW model surfaced artists similar to Lady Gaga in Japan, such as **Ayumi Hamasaki** whose Wiki bio says:\n",
    "\n",
    "> Ayumi Hamasaki is a Japanese singer, songwriter, record producer, actress, model, spokesperson, and entrepreneur.\n",
    "\n",
    "So that sounds like a success. It's also the nr. 1 hit in the paper we're replicating.\n",
    "\n",
    "Similarly, the DM model thought **Kaela Kimura** is the closest hit:\n",
    "\n",
    "> Kaela Kimura is a Japanese pop rock singer, lyricist, fashion model and television presenter.\n",
    "\n",
    "Also pretty good.\n",
    "\n",
    "These results demonstrate that both training modes employed in the original paper are outstanding for calculating similarity between document vectors, word vectors, or a combination of both. The DM mode has the added advantage of being 4x faster to train."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you wanted to continue working with these trained models, you could save them to disk, to avoid having to re-train the models from scratch every time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-20 20:10:55,289 : INFO : Doc2Vec lifecycle event {'fname_or_handle': 'doc2vec_dbow.model', 'separately': 'None', 'sep_limit': 10485760, 'ignore': frozenset(), 'datetime': '2022-03-20T20:10:55.289405', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'saving'}\n",
      "2022-03-20 20:10:55,292 : INFO : storing np array 'vectors' to doc2vec_dbow.model.dv.vectors.npy\n",
      "2022-03-20 20:10:58,822 : INFO : storing np array 'vectors' to doc2vec_dbow.model.wv.vectors.npy\n",
      "2022-03-20 20:10:59,349 : INFO : storing np array 'syn1neg' to doc2vec_dbow.model.syn1neg.npy\n",
      "2022-03-20 20:10:59,842 : INFO : not storing attribute cum_table\n",
      "2022-03-20 20:11:03,053 : INFO : saved doc2vec_dbow.model\n",
      "2022-03-20 20:11:03,054 : INFO : Doc2Vec lifecycle event {'fname_or_handle': 'doc2vec_dm.model', 'separately': 'None', 'sep_limit': 10485760, 'ignore': frozenset(), 'datetime': '2022-03-20T20:11:03.054368', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'saving'}\n",
      "2022-03-20 20:11:03,054 : INFO : storing np array 'vectors' to doc2vec_dm.model.dv.vectors.npy\n",
      "2022-03-20 20:11:05,496 : INFO : storing np array 'vectors' to doc2vec_dm.model.wv.vectors.npy\n",
      "2022-03-20 20:11:06,366 : INFO : storing np array 'syn1neg' to doc2vec_dm.model.syn1neg.npy\n",
      "2022-03-20 20:11:06,792 : INFO : not storing attribute cum_table\n",
      "2022-03-20 20:11:09,296 : INFO : saved doc2vec_dm.model\n"
     ]
    }
   ],
   "source": [
    "models[0].save('doc2vec_dbow.model')\n",
    "models[1].save('doc2vec_dm.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To continue your doc2vec explorations, refer to the official API documentation in Gensim: https://radimrehurek.com/gensim/models/doc2vec.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}