docs/notebooks/doc2vec-wikipedia.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Doc2Vec on Wikipedia articles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook replicates the **Document Embedding with Paragraph Vectors** paper, http://arxiv.org/abs/1507.07998.\n",
    "\n",
    "In that paper, the authors only showed results from the DBOW (\"distributed bag of words\") mode, trained on the English Wikipedia. Here we replicate this experiment using not only DBOW, but also the DM (\"distributed memory\") mode of the Paragraph Vector algorithm aka Doc2Vec."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import the necessary modules and set up logging. The code below assumes Python 3.7+ and Gensim 4.0+."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import multiprocessing\n",
    "from pprint import pprint\n",
    "\n",
    "import smart_open\n",
    "from gensim.corpora.wikicorpus import WikiCorpus, tokenize\n",
    "from gensim.models.doc2vec import Doc2Vec, TaggedDocument\n",
    "\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, download the dump of all Wikipedia articles from [here](http://download.wikimedia.org/enwiki/). You want the file named `enwiki-latest-pages-articles.xml.bz2`.\n",
    "\n",
    "Second, convert that Wikipedia article dump from the arcane Wikimedia XML format into a plain text file. This will make the subsequent training faster and also allow easy inspection of the data = \"input eyeballing\".\n",
    "\n",
    "We'll preprocess each article at the same time, normalizing its text to lowercase, splitting into tokens, etc. Below I use a regexp tokenizer that simply looks for alphabetic sequences as tokens. But feel free to adapt the text preprocessing to your own domain. High quality preprocessing is often critical for the final pipeline accuracy – garbage in, garbage out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Volumes/work/workspace/gensim/trunk/gensim/utils.py:1332: UserWarning: detected OSX with python3.8+; aliasing chunkize to chunkize_serial\n",
      "  warnings.warn(\"detected %s; aliasing chunkize to chunkize_serial\" % entity)\n",
      "2022-03-17 21:15:32,118 : INFO : processing article #0: 'Anarchism' (6538 tokens)\n",
      "2022-03-17 21:30:00,138 : INFO : processing article #500000: 'Spiritual Formation Bible' (54 tokens)\n",
      "2022-03-17 21:40:22,219 : INFO : processing article #1000000: 'Adolf von Liebenberg' (52 tokens)\n",
      "2022-03-17 21:49:43,825 : INFO : processing article #1500000: 'Small nucleolar RNA U6-53/MBII-28' (123 tokens)\n",
      "2022-03-17 21:59:23,620 : INFO : processing article #2000000: 'Xie Fei' (50 tokens)\n",
      "2022-03-17 22:09:17,460 : INFO : processing article #2500000: 'Rhein, Saskatchewan' (185 tokens)\n",
      "2022-03-17 22:19:39,293 : INFO : processing article #3000000: 'Kunyinsky District' (969 tokens)\n",
      "2022-03-17 22:30:41,221 : INFO : processing article #3500000: 'Lake Saint-Charles' (555 tokens)\n",
      "2022-03-17 22:41:17,487 : INFO : processing article #4000000: 'Mahāyānasaṃgraha' (612 tokens)\n",
      "2022-03-17 22:52:27,834 : INFO : processing article #4500000: 'Liriomyza trifolii' (1493 tokens)\n",
      "2022-03-17 23:04:41,464 : INFO : processing article #5000000: 'Daniel O. Griffin' (594 tokens)\n",
      "2022-03-17 23:08:58,451 : INFO : finished iterating over Wikipedia corpus of 5176019 documents with 2996051328 positions (total 21837336 articles, 3072543084 positions before pruning articles shorter than 50 words)\n"
     ]
    }
   ],
   "source": [
    "wiki = WikiCorpus(\n",
    "    \"enwiki-latest-pages-articles.xml.bz2\",  # path to the file you downloaded above\n",
    "    tokenizer_func=tokenize,  # simple regexp; plug in your own tokenizer here\n",
    "    metadata=True,  # also return the article titles and ids when parsing\n",
    "    dictionary={},  # don't start processing the data yet\n",
    ")\n",
    "\n",
    "with smart_open.open(\"wiki.txt.gz\", \"w\", encoding='utf8') as fout:\n",
    "    for article_no, (content, (page_id, title)) in enumerate(wiki.get_texts()):\n",
    "        title = ' '.join(title.split())\n",
    "        if article_no % 500000 == 0:\n",
    "            logging.info(\"processing article #%i: %r (%i tokens)\", article_no, title, len(content))\n",
    "        fout.write(f\"{title}\\t{' '.join(content)}\\n\")  # title_of_article [TAB] words of the article"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above takes about 2 hours on my 2021 M1 MacbookPro, and creates a new ~5.8 GB file named `wiki.txt.gz`. We're compressing the text into `.gz` (GZIP) right away to save on disk space, using the [smart_open](https://github.com/RaRe-Technologies/smart_open) library.\n",
    "\n",
    "Next we'll set up a stream to load the preprocessed articles from `wiki.txt.gz` one by one, in the format expected by Doc2Vec, ready for training. We don't want to load everything into RAM at once, because that would blow up the memory. And it is not necessary – Gensim can handle streamed training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TaggedWikiCorpus:\n",
    "    def __init__(self, wiki_text_path):\n",
    "        self.wiki_text_path = wiki_text_path\n",
    "        \n",
    "    def __iter__(self):\n",
    "        for line in smart_open.open(self.wiki_text_path, encoding='utf8'):\n",
    "            title, words = line.split('\\t')\n",
    "            yield TaggedDocument(words=words.split(), tags=[title])\n",
    "\n",
    "documents = TaggedWikiCorpus('wiki.txt.gz')  # A streamed iterable; nothing in RAM yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Anarchism'] :  anarchism is political philosophy and movement that is sceptical of authority and rejects all involuntary coercive forms of hierarchy anarchism calls for the abolition of the state which it holds to be unnecessary undesirable and harmful as historically left wing movement placed on the farthest left of the political spectrum ……… criticism of philosophical anarchism defence of philosophical anarchism stating that both kinds of anarchism philosophical and political anarchism are philosophical and political claims anarchistic popular fiction novel an argument for philosophical anarchism external links anarchy archives anarchy archives is an online research center on the history and theory of anarchism\n"
     ]
    }
   ],
   "source": [
    "# Load and print the first preprocessed Wikipedia document, as a sanity check = \"input eyeballing\".\n",
    "first_doc = next(iter(documents))\n",
    "print(first_doc.tags, ': ', ' '.join(first_doc.words[:50] + ['………'] + first_doc.words[-50:]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The document seems legit so let's move on to finally training some Doc2vec models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Doc2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The original paper had a vocabulary size of 915,715 word types, so we'll try to match it by setting `max_final_vocab` to 1,000,000 in the Doc2vec constructor.\n",
    "\n",
    "Other critical parameters were left unspecified in the paper, so we'll go with a default window size of five (a prediction window of 5 tokens to either side), and downsampling of frequent words at 1e-5. It looks like the authors tried vector dimensionality of 100, 300, 1,000 & 10,000 in the paper (with 10k dims performing the best), but I'll only train with 300 dimensions here, to keep RAM in check on my laptop.\n",
    "\n",
    "Feel free to tinker with these values yourself if you like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-19 19:45:13,743 : INFO : Doc2Vec lifecycle event {'params': 'Doc2Vec<dbow+w,d300,n5,w5,mc5,s1e-05,t9>', 'datetime': '2022-03-19T19:45:13.743356', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'created'}\n",
      "2022-03-19 19:45:13,745 : INFO : Doc2Vec lifecycle event {'params': 'Doc2Vec<dm/m,d300,n5,w5,mc5,s1e-05,t9>', 'datetime': '2022-03-19T19:45:13.745470', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'created'}\n"
     ]
    }
   ],
   "source": [
    "workers = multiprocessing.cpu_count()  # train with 10 threads on my 10-core laptop\n",
    "\n",
    "models = [\n",
    "    # PV-DBOW: paragraph vector in distributed bag of words mode\n",
    "    Doc2Vec(\n",
    "        dm=0, dbow_words=1,  # dbow_words=1 to train word vectors at the same time too, not only DBOW\n",
    "        vector_size=300, window=5, sample=1e-5, epochs=10, workers=workers, max_final_vocab=1000000,\n",
    "    ),\n",
    "    # PV-DM: paragraph vector in distributed memory mode\n",
    "    Doc2Vec(\n",
    "        dm=1, dm_concat=0, dm_mean=1,  # use average of context word vectors to train DM\n",
    "        vector_size=300, window=5, sample=1e-5, epochs=10, workers=workers, max_final_vocab=1000000,\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-19 19:45:22,230 : INFO : collecting all words and their counts\n",
      "2022-03-19 19:45:22,237 : INFO : PROGRESS: at example #0, processed 0 words (0 words/s), 0 word types, 0 tags\n",
      "2022-03-19 19:47:16,714 : INFO : PROGRESS: at example #500000, processed 654950164 words (5721331 words/s), 3222179 word types, 500000 tags\n",
      "2022-03-19 19:48:22,015 : INFO : PROGRESS: at example #1000000, processed 1018611068 words (5569047 words/s), 4480366 word types, 1000000 tags\n",
      "2022-03-19 19:49:14,916 : INFO : PROGRESS: at example #1500000, processed 1305140647 words (5416499 words/s), 5420104 word types, 1500000 tags\n",
      "2022-03-19 19:50:00,348 : INFO : PROGRESS: at example #2000000, processed 1550245240 words (5395000 words/s), 6188355 word types, 2000000 tags\n",
      "2022-03-19 19:50:45,274 : INFO : PROGRESS: at example #2500000, processed 1790661139 words (5351430 words/s), 6941128 word types, 2500000 tags\n",
      "2022-03-19 19:51:30,706 : INFO : PROGRESS: at example #3000000, processed 2028261627 words (5229901 words/s), 7664997 word types, 3000000 tags\n",
      "2022-03-19 19:52:15,667 : INFO : PROGRESS: at example #3500000, processed 2264063867 words (5244690 words/s), 8347719 word types, 3500000 tags\n",
      "2022-03-19 19:52:58,112 : INFO : PROGRESS: at example #4000000, processed 2488354257 words (5284392 words/s), 8971529 word types, 4000000 tags\n",
      "2022-03-19 19:53:40,217 : INFO : PROGRESS: at example #4500000, processed 2703313059 words (5105821 words/s), 9605666 word types, 4500000 tags\n",
      "2022-03-19 19:54:22,431 : INFO : PROGRESS: at example #5000000, processed 2925111571 words (5254227 words/s), 10217554 word types, 5000000 tags\n",
      "2022-03-19 19:54:51,211 : INFO : collected 10427023 word types and 5176019 unique tags from a corpus of 5176019 examples and 2996051328 words\n",
      "2022-03-19 19:54:54,641 : INFO : Doc2Vec lifecycle event {'msg': 'max_final_vocab=1000000 and min_count=5 resulted in calc_min_count=23, effective_min_count=23', 'datetime': '2022-03-19T19:54:54.641858', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'prepare_vocab'}\n",
      "2022-03-19 19:54:54,642 : INFO : Creating a fresh vocabulary\n",
      "2022-03-19 19:54:58,144 : INFO : Doc2Vec lifecycle event {'msg': 'effective_min_count=23 retains 991887 unique words (9.51% of original 10427023, drops 9435136)', 'datetime': '2022-03-19T19:54:58.144360', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'prepare_vocab'}\n",
      "2022-03-19 19:54:58,144 : INFO : Doc2Vec lifecycle event {'msg': 'effective_min_count=23 leaves 2968296495 word corpus (99.07% of original 2996051328, drops 27754833)', 'datetime': '2022-03-19T19:54:58.144810', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'prepare_vocab'}\n",
      "2022-03-19 19:55:00,904 : INFO : deleting the raw counts dictionary of 10427023 items\n",
      "2022-03-19 19:55:00,968 : INFO : sample=1e-05 downsamples 4155 most-common words\n",
      "2022-03-19 19:55:00,969 : INFO : Doc2Vec lifecycle event {'msg': 'downsampling leaves estimated 1194754612.050565 word corpus (40.3%% of prior 2968296495)', 'datetime': '2022-03-19T19:55:00.969465', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'prepare_vocab'}\n",
      "2022-03-19 19:55:05,714 : INFO : estimated required memory for 991887 words and 300 dimensions: 10122898900 bytes\n",
      "2022-03-19 19:55:05,714 : INFO : resetting layer weights\n",
      "2022-03-19 19:55:14,899 : INFO : resetting layer weights\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc2Vec<dbow+w,d300,n5,w5,mc5,s1e-05,t9>\n",
      "Doc2Vec<dm/m,d300,n5,w5,mc5,s1e-05,t9>\n"
     ]
    }
   ],
   "source": [
    "models[0].build_vocab(documents, progress_per=500000)\n",
    "print(models[0])\n",
    "\n",
    "# Save some time by copying the vocabulary structures from the DBOW model to the DM model.\n",
    "# Both models are built on top of exactly the same data, so there's no need to repeat the vocab-building step.\n",
    "models[1].reset_from(models[0])\n",
    "print(models[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we’re ready to train Doc2Vec on the entirety of the English Wikipedia. **Warning!** Training this DBOW model takes ~16 hours, and DM ~4 hours, on my 2021 laptop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-19 19:55:38,660 : INFO : Doc2Vec lifecycle event {'msg': 'training model with 9 workers on 991887 vocabulary and 300 features, using sg=1 hs=0 sample=1e-05 negative=5 window=5 shrink_windows=True', 'datetime': '2022-03-19T19:55:38.660218', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'train'}\n",
      "2022-03-19 19:55:39,851 : INFO : EPOCH 1 - PROGRESS: at 0.00% examples, 5854 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-19 20:25:39,805 : INFO : EPOCH 1 - PROGRESS: at 48.25% examples, 393044 words/s, in_qsize 18, out_qsize 0\n",
      "2022-03-19 20:48:10,761 : INFO : EPOCH - 1 : training on 2996051328 raw words (1198675030 effective words) took 3152.1s, 380279 effective words/s\n",
      "2022-03-19 20:48:11,793 : INFO : EPOCH 2 - PROGRESS: at 0.00% examples, 367086 words/s, in_qsize 16, out_qsize 0\n",
      "2022-03-19 21:18:11,834 : INFO : EPOCH 2 - PROGRESS: at 46.01% examples, 380787 words/s, in_qsize 18, out_qsize 0\n",
      "2022-03-19 21:41:41,740 : INFO : EPOCH - 2 : training on 2996051328 raw words (1198715463 effective words) took 3210.9s, 373328 effective words/s\n",
      "2022-03-19 21:41:42,801 : INFO : EPOCH 3 - PROGRESS: at 0.00% examples, 371377 words/s, in_qsize 17, out_qsize 0\n",
      "2022-03-19 22:11:42,858 : INFO : EPOCH 3 - PROGRESS: at 47.08% examples, 386416 words/s, in_qsize 18, out_qsize 1\n",
      "2022-03-19 22:34:07,773 : INFO : EPOCH - 3 : training on 2996051328 raw words (1198709324 effective words) took 3145.9s, 381038 effective words/s\n",
      "2022-03-19 22:34:08,813 : INFO : EPOCH 4 - PROGRESS: at 0.00% examples, 391762 words/s, in_qsize 18, out_qsize 0\n",
      "2022-03-19 23:04:08,869 : INFO : EPOCH 4 - PROGRESS: at 47.18% examples, 387058 words/s, in_qsize 18, out_qsize 0\n",
      "2022-03-19 23:27:17,510 : INFO : EPOCH - 4 : training on 2996051328 raw words (1198691470 effective words) took 3189.6s, 375807 effective words/s\n",
      "2022-03-19 23:27:18,566 : INFO : EPOCH 5 - PROGRESS: at 0.00% examples, 341762 words/s, in_qsize 18, out_qsize 0\n",
      "2022-03-19 23:57:18,560 : INFO : EPOCH 5 - PROGRESS: at 49.49% examples, 400066 words/s, in_qsize 17, out_qsize 0\n",
      "2022-03-20 00:18:39,692 : INFO : EPOCH - 5 : training on 2996051328 raw words (1198651978 effective words) took 3082.1s, 388902 effective words/s\n",
      "2022-03-20 00:18:40,744 : INFO : EPOCH 6 - PROGRESS: at 0.00% examples, 389656 words/s, in_qsize 17, out_qsize 1\n",
      "2022-03-20 00:48:40,746 : INFO : EPOCH 6 - PROGRESS: at 48.94% examples, 396931 words/s, in_qsize 18, out_qsize 0\n",
      "2022-03-20 01:10:12,092 : INFO : EPOCH - 6 : training on 2996051328 raw words (1198683303 effective words) took 3092.4s, 387624 effective words/s\n",
      "2022-03-20 01:10:13,131 : INFO : EPOCH 7 - PROGRESS: at 0.00% examples, 380610 words/s, in_qsize 17, out_qsize 0\n",
      "2022-03-20 01:40:13,225 : INFO : EPOCH 7 - PROGRESS: at 48.99% examples, 397202 words/s, in_qsize 18, out_qsize 0\n",
      "2022-03-20 02:01:49,460 : INFO : EPOCH - 7 : training on 2996051328 raw words (1198700562 effective words) took 3097.3s, 387016 effective words/s\n",
      "2022-03-20 02:01:50,477 : INFO : EPOCH 8 - PROGRESS: at 0.00% examples, 379974 words/s, in_qsize 17, out_qsize 0\n",
      "2022-03-20 02:31:50,541 : INFO : EPOCH 8 - PROGRESS: at 49.01% examples, 397386 words/s, in_qsize 17, out_qsize 0\n",
      "2022-03-20 02:53:22,480 : INFO : EPOCH - 8 : training on 2996051328 raw words (1198739172 effective words) took 3093.0s, 387567 effective words/s\n",
      "2022-03-20 02:53:23,526 : INFO : EPOCH 9 - PROGRESS: at 0.00% examples, 403121 words/s, in_qsize 18, out_qsize 0\n",
      "2022-03-20 03:23:23,535 : INFO : EPOCH 9 - PROGRESS: at 49.86% examples, 402052 words/s, in_qsize 17, out_qsize 0\n",
      "2022-03-20 03:44:21,076 : INFO : EPOCH - 9 : training on 2996051328 raw words (1198699759 effective words) took 3058.5s, 391921 effective words/s\n",
      "2022-03-20 03:44:22,093 : INFO : EPOCH 10 - PROGRESS: at 0.00% examples, 363235 words/s, in_qsize 17, out_qsize 0\n",
      "2022-03-20 04:14:22,127 : INFO : EPOCH 10 - PROGRESS: at 47.77% examples, 390356 words/s, in_qsize 18, out_qsize 0\n",
      "2022-03-20 04:37:01,731 : INFO : EPOCH - 10 : training on 2996051328 raw words (1198674743 effective words) took 3160.6s, 379258 effective words/s\n",
      "2022-03-20 04:37:01,758 : INFO : Doc2Vec lifecycle event {'msg': 'training on 29960513280 raw words (11986940804 effective words) took 31282.8s, 383179 effective words/s', 'datetime': '2022-03-20T04:37:01.757530', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'train'}\n",
      "2022-03-20 04:37:01,760 : INFO : Doc2Vec lifecycle event {'msg': 'training model with 9 workers on 991887 vocabulary and 300 features, using sg=0 hs=0 sample=1e-05 negative=5 window=5 shrink_windows=True', 'datetime': '2022-03-20T04:37:01.760198', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'train'}\n",
      "2022-03-20 04:37:02,806 : INFO : EPOCH 1 - PROGRESS: at 0.00% examples, 51780 words/s, in_qsize 17, out_qsize 0\n",
      "2022-03-20 05:01:24,379 : INFO : EPOCH - 1 : training on 2996051328 raw words (1198699065 effective words) took 1462.6s, 819587 effective words/s\n",
      "2022-03-20 05:01:25,419 : INFO : EPOCH 2 - PROGRESS: at 0.01% examples, 1026623 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 05:25:51,408 : INFO : EPOCH - 2 : training on 2996051328 raw words (1198713530 effective words) took 1467.0s, 817102 effective words/s\n",
      "2022-03-20 05:25:52,466 : INFO : EPOCH 3 - PROGRESS: at 0.01% examples, 1053617 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 05:50:03,250 : INFO : EPOCH - 3 : training on 2996051328 raw words (1198725674 effective words) took 1451.7s, 825713 effective words/s\n",
      "2022-03-20 05:50:04,303 : INFO : EPOCH 4 - PROGRESS: at 0.01% examples, 994295 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 06:14:21,091 : INFO : EPOCH - 4 : training on 2996051328 raw words (1198700285 effective words) took 1457.8s, 822282 effective words/s\n",
      "2022-03-20 06:14:22,184 : INFO : EPOCH 5 - PROGRESS: at 0.01% examples, 1051640 words/s, in_qsize 0, out_qsize 1\n",
      "2022-03-20 06:38:35,360 : INFO : EPOCH - 5 : training on 2996051328 raw words (1198682356 effective words) took 1454.2s, 824283 effective words/s\n",
      "2022-03-20 06:38:36,422 : INFO : EPOCH 6 - PROGRESS: at 0.01% examples, 1055533 words/s, in_qsize 0, out_qsize 1\n",
      "2022-03-20 07:02:50,449 : INFO : EPOCH - 6 : training on 2996051328 raw words (1198672468 effective words) took 1455.0s, 823803 effective words/s\n",
      "2022-03-20 07:02:51,496 : INFO : EPOCH 7 - PROGRESS: at 0.01% examples, 1052539 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 07:27:17,109 : INFO : EPOCH - 7 : training on 2996051328 raw words (1198696667 effective words) took 1466.6s, 817342 effective words/s\n",
      "2022-03-20 07:27:18,148 : INFO : EPOCH 8 - PROGRESS: at 0.01% examples, 1047255 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 07:51:29,674 : INFO : EPOCH - 8 : training on 2996051328 raw words (1198691941 effective words) took 1452.5s, 825260 effective words/s\n",
      "2022-03-20 07:51:30,808 : INFO : EPOCH 9 - PROGRESS: at 0.01% examples, 915921 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 08:15:44,007 : INFO : EPOCH - 9 : training on 2996051328 raw words (1198707402 effective words) took 1454.2s, 824316 effective words/s\n",
      "2022-03-20 08:15:45,041 : INFO : EPOCH 10 - PROGRESS: at 0.01% examples, 1038693 words/s, in_qsize 0, out_qsize 0\n",
      "2022-03-20 08:39:48,080 : INFO : EPOCH - 10 : training on 2996051328 raw words (1198683355 effective words) took 1444.0s, 830092 effective words/s\n",
      "2022-03-20 08:39:48,155 : INFO : Doc2Vec lifecycle event {'msg': 'training on 29960513280 raw words (11986972743 effective words) took 14566.3s, 822924 effective words/s', 'datetime': '2022-03-20T08:39:48.155821', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'train'}\n"
     ]
    }
   ],
   "source": [
    "for model in models:\n",
    "    model.train(documents, total_examples=model.corpus_count, epochs=model.epochs, report_delay=30*60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Similarity interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, let's test both models! The DBOW model shows similar results as the original paper.\n",
    "\n",
    "First, calculate the most similar Wikipedia articles to the \"Machine learning\" article. The calculated word vectors and document vectors are stored separately, in `model.wv` and `model.dv` respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc2Vec<dbow+w,d300,n5,w5,mc5,s1e-05,t9>\n",
      "[('Deep learning', 0.6759864687919617),\n",
      " ('Pattern recognition', 0.6742060780525208),\n",
      " ('Supervised learning', 0.6707901358604431),\n",
      " ('Artificial neural network', 0.6665838956832886),\n",
      " ('Semi-supervised learning', 0.654625654220581),\n",
      " ('Outline of machine learning', 0.6472466588020325),\n",
      " ('Multi-task learning', 0.646246075630188),\n",
      " ('Boosting (machine learning)', 0.641443133354187),\n",
      " ('Neural network', 0.637937605381012),\n",
      " ('Types of artificial neural networks', 0.6377928853034973),\n",
      " ('Incremental learning', 0.637475848197937),\n",
      " ('Perceptron', 0.6358065605163574),\n",
      " ('Rule induction', 0.6326183676719666),\n",
      " ('Early stopping', 0.6307427883148193),\n",
      " ('Multilayer perceptron', 0.6273738741874695),\n",
      " ('Connectionist expert system', 0.6203325390815735),\n",
      " ('Neural Designer', 0.6193098425865173),\n",
      " ('Meta learning (computer science)', 0.6182209253311157),\n",
      " ('Feature (machine learning)', 0.6175855398178101),\n",
      " ('Domain adaptation', 0.6154839396476746)]\n",
      "Doc2Vec<dm/m,d300,n5,w5,mc5,s1e-05,t9>\n",
      "[('Pattern recognition', 0.7132866382598877),\n",
      " ('Deep learning', 0.66518634557724),\n",
      " ('Supervised learning', 0.6553921699523926),\n",
      " ('Artificial neural network', 0.6497268080711365),\n",
      " ('Semi-supervised learning', 0.6446605324745178),\n",
      " ('Statistical learning theory', 0.6335287094116211),\n",
      " ('Ensemble learning', 0.6282770037651062),\n",
      " ('Prior knowledge for pattern recognition', 0.6238192319869995),\n",
      " ('Boosting (machine learning)', 0.6212880611419678),\n",
      " ('Early stopping', 0.6177847981452942),\n",
      " ('Statistical classification', 0.6156905293464661),\n",
      " ('Autoencoder', 0.6140605211257935),\n",
      " ('Predictive Model Markup Language', 0.6070867776870728),\n",
      " ('Multi-task learning', 0.606650710105896),\n",
      " ('Similarity learning', 0.6065689921379089),\n",
      " ('Feature learning', 0.5982990860939026),\n",
      " ('Automatic image annotation', 0.5949181914329529),\n",
      " ('Linear classifier', 0.5930992960929871),\n",
      " ('Support-vector machine', 0.5929517149925232),\n",
      " ('Active learning (machine learning)', 0.5914105176925659)]\n"
     ]
    }
   ],
   "source": [
    "for model in models:\n",
    "    print(model)\n",
    "    pprint(model.dv.most_similar(positive=[\"Machine learning\"], topn=20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both results seem similar and match the results from the paper's Table 1, although not exactly. This is because we don't kno wthe exact parameters and also Wikipedia. FIXME\n",
    "\n",
    "\n",
    ", but note the DM model took 4x less time train (training 4x faster).\n",
    "\n",
    "Second, let's calculate the most similar Wikipedia entries to \"Lady Gaga\" using Paragraph Vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc2Vec<dbow+w,d300,n5,w5,mc5,s1e-05,t9>\n",
      "[('Katy Perry', 0.6756664514541626),\n",
      " ('Ariana Grande', 0.6528252363204956),\n",
      " ('Miley Cyrus', 0.6260033845901489),\n",
      " ('Taylor Swift', 0.6144734621047974),\n",
      " ('Britney Spears', 0.6105279922485352),\n",
      " ('List of awards and nominations received by Lady Gaga', 0.6099144220352173),\n",
      " ('Madonna', 0.5910527110099792),\n",
      " ('Christina Aguilera', 0.5906776785850525),\n",
      " ('Beyoncé', 0.5887077450752258),\n",
      " ('Demi Lovato', 0.5824941992759705)]\n",
      "Doc2Vec<dm/m,d300,n5,w5,mc5,s1e-05,t9>\n",
      "[('Taylor Swift', 0.5584290623664856),\n",
      " ('Joanne (album)', 0.5531442761421204),\n",
      " ('List of Lady Gaga live performances', 0.546563982963562),\n",
      " ('Katy Perry', 0.5364790558815002),\n",
      " ('Cynthia Germanotta', 0.5363910794258118),\n",
      " ('Artpop', 0.5359800457954407),\n",
      " ('Natali Germanotta', 0.5320610404014587),\n",
      " ('Beautiful, Dirty, Rich', 0.5298762917518616),\n",
      " ('Joanne Trattoria Cookbook', 0.5219755172729492),\n",
      " ('The Fame', 0.5171768069267273)]\n"
     ]
    }
   ],
   "source": [
    "for model in models:\n",
    "    print(model)\n",
    "    pprint(model.dv.most_similar(positive=[\"Lady Gaga\"], topn=10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The DBOW results are in line with what the paper shows in Table 2a), revealing similar singers in the U.S.\n",
    "\n",
    "Finally, let's do some of the wilder arithmetics that vectors embeddings are famous for. What are the entries most similar to \"Lady Gaga\" - \"American\" + \"Japanese\"? Table 2b) in the paper.\n",
    "\n",
    "Note that \"American\" and \"Japanese\" are word vectors, but they live in the same space as the document vectors so we can add / subtract them at will, for some interesting results. All word vectors were already lowercased by our tokenizer above, so we look for the lowercased version here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc2Vec<dbow+w,d300,n5,w5,mc5,s1e-05,t9>\n",
      "[('Katy Perry', 0.5954936146736145),\n",
      " ('Ariana Grande', 0.5549463033676147),\n",
      " ('Thank You, Love (Kana Nishino album)', 0.5462373495101929),\n",
      " ('Kōsui (Eito song)', 0.5363353490829468),\n",
      " ('Crazy Crazy / Harajuku Iyahoi', 0.5308663249015808),\n",
      " ('Big Boys Cry/Beautiful', 0.5303178429603577),\n",
      " ('23rd Monster', 0.5298969745635986),\n",
      " ('Koi (song)', 0.5275850296020508),\n",
      " ('X -Cross-', 0.5255367755889893),\n",
      " ('Suzume (song)', 0.525151252746582)]\n",
      "Doc2Vec<dm/m,d300,n5,w5,mc5,s1e-05,t9>\n",
      "[('Joanne (album)', 0.5087171196937561),\n",
      " ('Artpop', 0.49931594729423523),\n",
      " ('The Cure (song)', 0.4835745394229889),\n",
      " ('Natali Germanotta', 0.47394222021102905),\n",
      " ('Chisato Moritaka', 0.47318926453590393),\n",
      " ('Ayumi Hamasaki', 0.47156259417533875),\n",
      " ('List of Lady Gaga live performances', 0.4629444479942322),\n",
      " ('Joanne Trattoria Cookbook', 0.4615497887134552),\n",
      " ('Katy Perry', 0.46151337027549744),\n",
      " ('Blackpink', 0.45603010058403015)]\n"
     ]
    }
   ],
   "source": [
    "for model in models:\n",
    "    print(model)\n",
    "    vec = [model.dv[\"Lady Gaga\"] - model.wv[\"american\"] + model.wv[\"japanese\"]]\n",
    "    pprint([m for m in model.dv.most_similar(vec, topn=11) if m[0] != \"Lady Gaga\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result, the DBOW model surfaced artists similar to Lady Gaga in Japan, such as **Ayumi Hamasaki** whose Wiki bio says:\n",
    "\n",
    "> Ayumi Hamasaki is a Japanese singer, songwriter, record producer, actress, model, spokesperson, and entrepreneur.\n",
    "\n",
    "So that sounds like a success. It's also the nr. 1 hit in the paper we're replicating.\n",
    "\n",
    "Similarly, the DM model thought **Kaela Kimura** is the closest hit:\n",
    "\n",
    "> Kaela Kimura is a Japanese pop rock singer, lyricist, fashion model and television presenter.\n",
    "\n",
    "Also pretty good.\n",
    "\n",
    "These results demonstrate that both training modes employed in the original paper are outstanding for calculating similarity between document vectors, word vectors, or a combination of both. The DM mode has the added advantage of being 4x faster to train."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you wanted to continue working with these trained models, you could save them to disk, to avoid having to re-train the models from scratch every time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-18 19:08:34,623 : INFO : Doc2Vec lifecycle event {'fname_or_handle': 'doc2vec_dbow.model', 'separately': 'None', 'sep_limit': 10485760, 'ignore': frozenset(), 'datetime': '2022-03-18T19:08:34.622990', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'saving'}\n",
      "2022-03-18 19:08:34,641 : INFO : storing np array 'vectors' to doc2vec_dbow.model.dv.vectors.npy\n",
      "2022-03-18 19:08:40,244 : INFO : storing np array 'vectors' to doc2vec_dbow.model.wv.vectors.npy\n",
      "2022-03-18 19:08:46,811 : INFO : storing np array 'syn1neg' to doc2vec_dbow.model.syn1neg.npy\n",
      "2022-03-18 19:08:48,564 : INFO : not storing attribute cum_table\n",
      "2022-03-18 19:08:56,097 : INFO : saved doc2vec_dbow.model\n",
      "2022-03-18 19:08:56,098 : INFO : Doc2Vec lifecycle event {'fname_or_handle': 'doc2vec_dm.model', 'separately': 'None', 'sep_limit': 10485760, 'ignore': frozenset(), 'datetime': '2022-03-18T19:08:56.098765', 'gensim': '4.1.3.dev0', 'python': '3.8.9 (default, Oct 26 2021, 07:25:53) \\n[Clang 13.0.0 (clang-1300.0.29.30)]', 'platform': 'macOS-12.2.1-arm64-arm-64bit', 'event': 'saving'}\n",
      "2022-03-18 19:08:56,099 : INFO : storing np array 'vectors' to doc2vec_dm.model.dv.vectors.npy\n",
      "2022-03-18 19:09:09,087 : INFO : storing np array 'vectors' to doc2vec_dm.model.wv.vectors.npy\n",
      "2022-03-18 19:09:13,804 : INFO : storing np array 'syn1neg' to doc2vec_dm.model.syn1neg.npy\n",
      "2022-03-18 19:09:16,101 : INFO : not storing attribute cum_table\n",
      "2022-03-18 19:09:20,432 : INFO : saved doc2vec_dm.model\n"
     ]
    }
   ],
   "source": [
    "models[0].save('doc2vec_dbow.model')\n",
    "models[1].save('doc2vec_dm.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To continue your doc2vec explorations, refer to the official API documentation in Gensim: https://radimrehurek.com/gensim/models/doc2vec.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}