cwpk-63-staging-data-sci-preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#63: Staging Data Sci Resources and Preprocessing\n",
    "=======================================\n",
    "\n",
    "Clean Corpora and Datasets are a Major Part of the Effort\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "With our discussions of [network analysis](https://en.wikipedia.org/wiki/Network_theory) and knowledge extractions from our [knowledge graph](https://en.wikipedia.org/wiki/Knowledge_graph) now behind us, we are ready to tackle the questions of analytic applications and [machine learning](https://en.wikipedia.org/wiki/Machine_learning) in earnest for our [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) series. We will be devoting our next nine installments to this area. We devote two installments to data sources and input preparations, largely based on [NLP](https://en.wikipedia.org/wiki/Natural_language_processing) (natural language processing) applications. Then we devote two installments to 'standard' machine learning (largely) using the [scikit-learn](https://github.com/scikit-learn/scikit-learn) packages. We next devote four installments to [deep learning](https://en.wikipedia.org/wiki/Deep_learning), split equally between the Deep Learning Graph ([DGL](https://www.dgl.ai/)) and PyTorch Geometric ([PyG](https://github.com/rusty1s/pytorch_geometric)) frameworks. We then conclude this **Part VI** with a summary and comparison of results across these installments based on the task of node classification.\n",
    "\n",
    "In this particular installment we flesh out the plan for completing these installments and discuss data sources and completing data prep needed for the plan. We provide particular attention to the architecture and data flows within the [PyTorch](https://en.wikipedia.org/wiki/PyTorch) framework. We describe the additional Python packages we need for this work, and install and configure the first ones. We discuss general sources of data and corpora useful for machine learning purposes. Our coding efforts in this installment will obtain and clean the Wikipedia pages that supplement the two structural and annotation sources based on KBpedia that were covered in the prior installment. These three sources of **structure**, **annotations** and **pages** are the input basis to creating our own embeddings to be used in many of the machine learning tests."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plan for Completion of Part VI\n",
    "The broad ecosystem of Python packages I was considering looked, generally, to be good choices to work together, as first outlined in [**CWPK #61**](https://www.mkbergman.com/2411/cwpk-61-nlp-machine-learning-and-analysis/). I had done an adequate initial diligence. But, how all of this was to unfold, what my plan of attack should be, became driving factors I had to solve to shorten my development and coding efforts. So, with an understanding of how we could extract general information from KBpedia useful to analysis and machine learning, I needed to project out over the entire anticipated scope to see if, indeed, these initial sources looked to be the right ones for our purposes. And, if so, how shall the efforts be sequenced and what is the flow of data?\n",
    "\n",
    "Much reading and research went into this effort. It is true, for example, that we had already prepared a pretty robust series of analytic and machine learning case studies in [Clojure](https://en.wikipedia.org/wiki/Clojure), available from the [KBpedia Web site](https://kbpedia.org/use-cases/). I revisited each of these use cases and got some ideas of what made sense for us to attempt with Python. But I needed to understand the capabilities now available to us with Python, so I also studied each of the candidate keystone packages in some detail.\n",
    "\n",
    "I will weave the results of this research as the next installments unfold, providing background discussion in context and as appropriate. But, in total, I formulated about 30 tasks going forward that appeared necessary to cover the defined scope. The listing below summarizes these steps, and keys the transition point (as indicated by **CWPK** installment number) for proceeding to each next new installment:\n",
    "\n",
    "1. Formulate Part VI plan\n",
    "1. Extract two source files from KBpedia\n",
    "  - structure\n",
    "  - annotations\n",
    "3. Set environment up (not doing virtual)\n",
    "1. Obtain Wikipedia articles for matching RCs\n",
    "1. Set up gensim\n",
    "1. Clean Wikipedia articles, all KB annotations\n",
    "1. Set up spaCy\n",
    "1. ID, extract phrases\n",
    "1. Finish embeddings prep **#64**\n",
    "  - remove stoplist\n",
    "  - create numeric??\n",
    "1.  Create embedding models:\n",
    "  - word2vec and doc2vec\n",
    "1. Text summarization for short articles (gensim)   \n",
    "1. Named entity recognition\n",
    "1. Set up scikit-learn **#65**\n",
    "1. Create master pandas file\n",
    "1. Do event/action extraction\n",
    "1. Do scikit-learn classifier **#66**\n",
    "  - SVM\n",
    "  - k-nearest neighbors\n",
    "  - random forests\n",
    "1. Introduce the sklearn.metrics module and confusion matrix, etc. The standard for reporting\n",
    "1. Discuss basic test parameters/'gold standars'\n",
    "1. Knowledge graph embeddings **#67**\n",
    "1. Create embedding models -2\n",
    "  - KB-struct\n",
    "  - KB-annot\n",
    "  - KB-annot-full: what is above + below\n",
    "  - KB-annot-page\n",
    "1. Set up PyTorch/DLG-KE  **#68**\n",
    "1. Set up PyTorch/PyG \n",
    "1. Formulate learning pathway/code\n",
    "1. Do standard DL classifiers: **#69**\n",
    "  - TransE\n",
    "  - TransR\n",
    "  - RESCAL\n",
    "  - DistMult\n",
    "  - ComplEx\n",
    "  - RotatE\n",
    "1. Do research DL classifiers: **#70**\n",
    "  - VAE\n",
    "  - GGSNN\n",
    "  - MPNN\n",
    "  - ChebyNet\n",
    "  - GCN\n",
    "  - SAGE\n",
    "  - GAT\n",
    "1. Choose a model evaluator: **#71**\n",
    "  - scikit-learn\n",
    "  - pyTorch\n",
    "  - other?\n",
    "1. Collate prior results\n",
    "1. Evaluate prior results\n",
    "1. Present comparative results\n",
    "\n",
    "Some of these steps also needed some preliminary research before proceeding. For example, knowing I wanted to compare results across algorithms meant I needed to have a good understanding of testing and analysis requirements before starting any of the tests. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch Architecture\n",
    "\n",
    "A critical question in contemplating this plan was how exactly data needed to be produced, staged, and then fed into the analysis portions. From the earlier investigations I had identified the three categories of knowledge grounded in KBpedia that could act as bases or features to machine learning; namely, **structure**, **annotations** and **pages**. I also had identified PyTorch as a shared abstraction layer for deep and machine learning. \n",
    "\n",
    "I was particularly focused on the question of data formats and representations such that information could be readily passed from one step to the next in the analysis pipeline. *Figure 1* is the resulting data flow chart and architecture that arose from these investigations.\n",
    "\n",
    "First, the green block labeled 'owlready2' represents that Python package, but also the location where the intact knowledge graph of KBpedia is stored and accessed. As early installments covered, we can use either [owlready2](https://owlready2.readthedocs.io/en/latest/) or [Protégé](https://en.wikipedia.org/wiki/Prot%C3%A9g%C3%A9_(software)) to manage this knowledge graph, though owlready2 is the point at which the KBpedia information is exported or extracted for downstream uses, importantly machine learning. As our owlready2 discussions also indicated, there is a close relationship between it and [RDFLib](https://rdflib.readthedocs.io/en/stable/) (which is also the [SPARQL](https://en.wikipedia.org/wiki/SPARQL) access point). RDFLib can provide direct input into [NetworkX](https://en.wikipedia.org/wiki/NetworkX), but that is limited to **structure** only.\n",
    "\n",
    "The clearest common denominator format for entry into the machine learning pipeline is [pandas](https://en.wikipedia.org/wiki/Pandas_(software)) via [CSV](https://en.wikipedia.org/wiki/Comma-separated_values) files. This centrality is fortunate given that all of our prior KBpedia extract-and-build routines have been designed around this format. This format is also one of the direct feeds possible into the PyTorch datasets format, as the figure shows:\n",
    "\n",
    "<div style=\"margin: 10px auto; display: table;\">\n",
    "\n",
    "<img src=\"files/ml-data-flow.png\" title=\"Data Flows in Machine Learning and KG Analysis\" width=\"800\" alt=\"Data Flows in Machine Learning and KG Analysis\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "<div style=\"margin: 10px auto; display: table; font-style: italic;\">\n",
    "\n",
    "Figure 1: Data Flows in Machine Learning and Knowledge Graph Analysis\n",
    "\n",
    "</div>\n",
    "\n",
    "An important block on the figure is for 'embeddings'. If you recall, all text needs to first be encoded to a numeric form to be understood by the computer. This process can also undertake dimensionality reduction, important for a sparse matrix data form like language. This same ability can be applied to graph structure and interactions. Thus, the 'embedding' block is a pivotal point at which we can represent words, sentences, paragraphs, documents, nodes, or entire graphs. We will focus much on embeddings throughout this **Part VI**.\n",
    "\n",
    "For training purposes we can also feed pre-trained corpora or embeddings into the system. We address this topic in the next main section.\n",
    "\n",
    "*Figure 1* is not meant to be a comprehensive view of PyTorch, but it is one useful to understand data flows with respect to our use of the KBpedia knowledge graph. Over the course of this research, I also encountered many PyTorch-related extensions that, when warranted, I include in the discussion.\n",
    "\n",
    "#### Possible Extensions\n",
    "There are some extensions to the PyTorch ecosystem that we will not be using or testing in this **CWPK** series. Here are some of the ones that seem closest in capabilities to what we are doing with KBpedia:\n",
    "\n",
    "- [PyCaret](https://pycaret.org/) is essentially a Python wrapper around several machine learning libraries and frameworks such as scikit-learn, XGBoost, Microsoft LightGBM, spaCy, and many more\n",
    "- [PiePline](https://github.com/PiePline/piepline) is a neural networks training pipeline based on PyTorch. Designed to standardize training process and accelerate experiments\n",
    "- [Catalyst](https://github.com/catalyst-team/catalyst) helps to write full-featured deep learning pipelines in a few lines of code\n",
    "- [Poutyne](https://github.com/GRAAL-Research/poutyne) is a Keras-like framework for PyTorch and handles much of the boilerplating code needed to train neural networks  \n",
    "- [torchtext](https://pytorch.org/text/) has some capabilities in language modeling, sentiment analysis, text classification, question classification, entailment, machine translation, sequence tagging, question answering, and unsupervised learning\n",
    "- [Spotlight](https://github.com/maciejkula/spotlight) uses PyTorch to build both deep and shallow recommender models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Corpora and Datasets\n",
    "\n",
    "There are many off-the-shelf resources that can be of use when doing machine learning involving text and language. (There are as well for images, but that is out of scope to our current interests.) These resources fall into three main areas:\n",
    "\n",
    "- [corpora](https://en.wikipedia.org/wiki/Text_corpus) - are language resources of either a general or domain nature, with vetted relationships or annotations between terms and concepts or other pre-processing useful to [computational linguistics](https://en.wikipedia.org/wiki/Corpus_linguistics)\n",
    "- [pre-trained models](https://en.wikipedia.org/wiki/Language_model) - are pre-calculated language models, often expressing probability distributions over words or text. Some embeddings can act in this manner. [Transformers](transformers) use deep learning to train their representations, with [BERT](https://en.wikipedia.org/wiki/BERT_(language_model)) being a notable example \n",
    "- embeddings - are vector representations of chunks of text, ranging from individual words up to entire documents or languages. The numeric representation either represents a pooled statistical representation across all tokens (the so-called [CBOW](https://en.wikipedia.org/wiki/Bag-of-words_model#CBOW) approach) or context and adjacency using the [skip-gram](https://en.wikipedia.org/wiki/N-gram#Skip-gram) or similar method. [GloVe](https://nlp.stanford.edu/projects/glove/), [word2vec](https://en.wikipedia.org/wiki/Word2vec) and [fastText](https://en.wikipedia.org/wiki/FastText) are example methodologies for producing word embeddings.\n",
    "\n",
    "Example corpora include Wikipedia (in multiple languages), news articles, Web crawls, and many others. Such corpora can be used as the language input basis for training various models, or may be a reference vocabulary for scoring and ranking input text. Various pre-trained language models are available, and embedding methods are available in a number of Python packages, including scikit-learn, [gensim](https://radimrehurek.com/gensim/) and [spaCy](https://spacy.io/) used in *cowpoke*.\n",
    "  \n",
    "#### Pre-trained Resources\n",
    "There are a number of free or open-source resources for these corpora or datasets. Some include:\n",
    "\n",
    "- [Transformers](https://github.com/huggingface/transformers) provides thousands of pretrained models to perform tasks on texts such as classification, information extraction, question answering, summarization, translation, text generation, etc in 100+ languages\n",
    "- [HuggingFace datasets](https://github.com/huggingface/datasets)\n",
    "- [English Wikipedia dump](http://vectors.nlpl.eu/explore/embeddings/en/models/)\n",
    "- [Wikipedia2Vec pre-trained embeddings](https://wikipedia2vec.github.io/wikipedia2vec/pretrained/)\n",
    "- [gensim datasets](https://rare-technologies.com/new-download-api-for-pretrained-nlp-models-and-datasets-in-gensim) contain links to 8 options\n",
    "- [word2vec pre-trained models](https://github.com/3Top/word2vec-api#where-to-get-a-pretrained-models) lists 16 or so datasets\n",
    "- A comprehensive list of available [gensim datasets and models](moz-extension://679501e4-f7af-4a03-be07-bd99ebaf3542/page_panel.html?url=https%3A%2F%2Fraw.githubusercontent.com%2FRaRe-Technologies%2Fgensim-data%2Fmaster%2Flist.json&type=json&ext=json)\n",
    "- [11 pre-trained word embedding models](https://stackoverflow.com/questions/45310409/using-a-word2vec-model-pre-trained-on-wikipedia) in various embedding formats\n",
    "- Some older [GloVe embeddings](https://nlp.stanford.edu/projects/glove/)\n",
    "- [Word vectors for 157 languages](https://fasttext.cc/docs/en/crawl-vectors.html)\n",
    "- [DBpedia entity typing + word embeddings](https://github.com/ISE-FIZKarlsruhe/Entity-Typing-with-Word-Embeddings)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting Up the Environment\n",
    "In doing this research, I also assembled the list of Python packages needed to add these capabilities to *cowpoke*. Had I not just updated the conda packages, I would do so now:\n",
    "\n",
    "<code>conda update --all</code>\n",
    "\n",
    "Next, the general recommendation when installing multiple new packages in Python is to do them in one batch, which allows the package manager (<code>conda</code> in our circumstance) to check on version conflicts and compatibility during the install process. However, with some of the packages involved in the current expansion, there are other settings necessary that obviates this standard 'batch' install recommendation.\n",
    "\n",
    "Another note is important here. In an enterprise environment with many Python projects, it is also best to install these machine learning extensions into their own virtual environment. (I covered this topic a bit in [**CWPK #58**](https://www.mkbergman.com/2407/cwpk-58-setting-up-a-remote-instance-and-web-page-server/).) However, since we are keeping this entire series in its own environment, we will skip that step here. You may prefer the virtual option.\n",
    "\n",
    "So, we will begin with those Python packages and frameworks that pose their own unique set-up and install challenges. We begin with PyTorch. We need to first appreciate that the rationale for PyTorch was to abstract machine learning constructs while taking advantage of graphics processing units ([GPUs](https://en.wikipedia.org/wiki/Graphics_processing_unit)) (specifically, [Nvidia](https://en.wikipedia.org/wiki/Nvidia) via the [CUDA](https://en.wikipedia.org/wiki/CUDA) interface). The CUDA architecture provides one or two orders of magnitude speed up on a local machine. Unfortunately, my local Windows machine does not have the separate Nvidia GPU, so I want to install the no CUDA option. For the PyTorch install options, visit https://pytorch.org/get-started/locally/. This figure shows my selections prior to download (yours may vary): \n",
    "\n",
    "<div style=\"margin: 10px auto; display: table;\">\n",
    "\n",
    "<img src=\"files/pytorch-install.png\" title=\"PyTorch Download Screen\" width=\"800\" alt=\"PyTorch Download Screen\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "<div style=\"margin: 10px auto; display: table; font-style: italic;\">\n",
    "\n",
    "Figure 2: PyTorch Download Screen\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In my circumstance, my local machine does not have a separate graphics processor, so I set the CUDA requirement to 'None' **(1)**. I also removed the 'torchvision' command line specification **(2)** since that is an image-related package. (We may later need some libraries from this package, in which case we will then install it.) The PyTorch package is rather large, so install takes a few minutes. Here is the actual install command:\n",
    "\n",
    "<code>conda install pytorch cpuonly -c pytorch</code>\n",
    "\n",
    "Since we were not able to batch all new packages, I decide to continue with some of the other major additions in a sequential matter, with spaCy and its [installation](https://spacy.io/usage) next:\n",
    "\n",
    "<code>conda install -c conda-forge spacy</code>\n",
    "\n",
    "and then gensim and its [installation](https://radimrehurek.com/gensim/):\n",
    "\n",
    "<code>conda install -c conda-forge gensim</code>\n",
    "\n",
    "and then DLG, which has an [installation](https://www.dgl.ai/pages/start.html) screen similar to PyTorch in *Figure 2* with the same picked options:\n",
    "\n",
    "<code>conda install -c dglteam dgl</code>\n",
    "\n",
    "The DLG-KE extension needs to be built from source for Windows, so we will hold off on that now until we need it. We next [install](https://github.com/rusty1s/pytorch_geometric) PyTorch Geometric, which needs to be installed from a series of binaries, with CPU or GPU individually specified:\n",
    "\n",
    "<pre>\n",
    "pip install torch-scatter==latest+cpu -f https://pytorch-geometric.com/whl/torch-1.6.0.html\n",
    "pip install torch-sparse==latest+cpu -f https://pytorch-geometric.com/whl/torch-1.6.0.html\n",
    "pip install torch-cluster==latest+cpu -f https://pytorch-geometric.com/whl/torch-1.6.0.html\n",
    "pip install torch-spline-conv==latest+cpu -f https://pytorch-geometric.com/whl/torch-1.6.0.html\n",
    "pip install torch-geometric\n",
    "</pre>\n",
    "\n",
    "These new packages join these that are already a part of my local <code>conda</code> packages, and which will arise in the coming installments:\n",
    "\n",
    "<code>scikit-learn</code> and <code>tqdm</code>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting Wikipedia Pages\n",
    "\n",
    "With these preliminaries complete, we are now ready to resume our data preparation tasks for our embedding and machine learning experiments. In the prior installment, we discussed two of the three source files we had identified for these efforts, the KBpedia **structure** (<code>kbpedia/v300/extractions/data/graph_specs.csv</code>) and the KBpedia **annotations** (<code>kbpedia/v300/extractions/classes/Generals_annot_out.csv</code>) files. In this specific section we obtain the third source file of **pages** from Wikipedia.\n",
    "\n",
    "Of the 58,000 reference concepts presently contained in KBpedia, about 45,000 have a directly corresponding Wikipedia article or listing of category articles. These provide a potentially rich source of content for language models and embeddings. The challenge is how to obtain this content in a way that can be readily processed for our purposes.\n",
    "\n",
    "We have been working with Wikipedia since its inception, so we knew that there are data sources for downloads or dumps. For example, the periodic language dumps such as https://dumps.wikimedia.org/enwiki/20200920/ may be accessed to obtain full-text versions of articles. Such dumps have been used scores of times to produce Wikipedia corpora in many different languages and for many different purposes. But, our own mappings are a mere subset, about 1% of the nearly 6 million articles in the English Wikipedia alone. So, even if we grabbed the current dump or one of the corpora so derived, we would need to process much content to obtain the subset of interest.\n",
    "\n",
    "Unfortunately, Wikipedia does not have a direct query or SPARQL form as exists for Wikidata (which also does not have full-text articles). We could obtain the so-called 'long abstracts' of Wikipedia pages from DBpedia (see, for example, https://wiki.dbpedia.org/downloads-2016-10), but this source is dated and each abstract is limited to about 220 words; further, a full download of the specific file in English is about 15 GB!\n",
    "\n",
    "The basic approach, then, appeared that I would need to download the full Wikipedia article file, figure out how to split it into parts, and then match identifiers between KBpedia mappings and the full dataset to obtain the articles of interest. This approach is not technically difficult, but it is a real pain in the ass.\n",
    "\n",
    "So, shortly before I committed to this work effort, I challenged myself to find another way that was perhaps less onerous. Fortunately, I found the online Wikipedia service, https://en.wikipedia.org/wiki/Special:Export, that allows one to submit article names to a text box and then get the full page article back in XML format. I tested this online service with a few articles, then 100, and then ramped up to a listing of 5 K at a time. (Similar services often have governors that limit the frequency or amounts of individual requests.) This approach worked!, and within 30 min I had full articles in nine separate batches for all 45 K items in KBpedia."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean All Input Text\n",
    "This file is a single article from the Wikipedia English dump for <code>1-(2-Nitrophenoxy)octane</code>:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<page>\n",
    "    <title>1-(2-Nitrophenoxy)octane</title>\n",
    "    <ns>0</ns>\n",
    "    <id>11793192</id>\n",
    "    <revision>\n",
    "      <id>891140188</id>\n",
    "      <parentid>802024542</parentid>\n",
    "      <timestamp>2019-04-05T23:04:47Z</timestamp>\n",
    "      <contributor>\n",
    "        <username>Koavf</username>\n",
    "        <id>205121</id>\n",
    "      </contributor>\n",
    "      <minor/>\n",
    "      <comment>/* top */Replace HTML with MediaWiki markup or templates, replaced: &lt;sub&gt; → {{sub| (3), &lt;/sub&gt; → }} (3)</comment>\n",
    "      <model>wikitext</model>\n",
    "      <format>text/x-wiki</format>\n",
    "      <text bytes=\"2029\" xml:space=\"preserve\">{{chembox\n",
    "| Watchedfields = changed\n",
    "| verifiedrevid = 477206849\n",
    "| ImageFile =Nitrophenoxyoctane.png\n",
    "| ImageSize =240px\n",
    "| ImageFile1 = 1-(2-Nitrophenoxy)octane-3D-spacefill.png\n",
    "| ImageSize1 = 220\n",
    "| ImageAlt1 = NPOE molecule\n",
    "| PIN = 1-Nitro-2-(octyloxy)benzene\n",
    "| OtherNames = 1-(2-Nitrophenoxy)octane&lt;br /&gt;2-Nitrophenyl octyl ether&lt;br /&gt;1-Nitro-2-octoxy-benzene&lt;br /&gt;2-(Octyloxy)nitrobenzene&lt;br /&gt;Octyl o-nitrophenyl ether\n",
    "|Section1={{Chembox Identifiers\n",
    "| Abbreviations =NPOE\n",
    "| ChemSpiderID_Ref = {{chemspidercite|correct|chemspider}}\n",
    "| ChemSpiderID = 148623\n",
    "| InChI = 1/C14H21NO3/c1-2-3-4-5-6-9-12-18-14-11-8-7-10-13(14)15(16)17/h7-8,10-11H,2-6,9,12H2,1H3\n",
    "| InChIKey = CXVOIIMJZFREMM-UHFFFAOYAD\n",
    "| StdInChI_Ref = {{stdinchicite|correct|chemspider}}\n",
    "| StdInChI = 1S/C14H21NO3/c1-2-3-4-5-6-9-12-18-14-11-8-7-10-13(14)15(16)17/h7-8,10-11H,2-6,9,12H2,1H3\n",
    "| StdInChIKey_Ref = {{stdinchicite|correct|chemspider}}\n",
    "| StdInChIKey = CXVOIIMJZFREMM-UHFFFAOYSA-N\n",
    "| CASNo_Ref = {{cascite|correct|CAS}}\n",
    "| CASNo =37682-29-4\n",
    "| PubChem =169952\n",
    "| SMILES = [O-][N+](=O)c1ccccc1OCCCCCCCC\n",
    "}}\n",
    "|Section2={{Chembox Properties\n",
    "| Formula =C{{sub|14}}H{{sub|21}}NO{{sub|3}}\n",
    "| MolarMass =251.321\n",
    "| Appearance =\n",
    "| Density =1.04 g/mL\n",
    "| MeltingPt =\n",
    "| BoilingPtC = 197 to 198\n",
    "| BoilingPt_notes = (11 mm Hg)\n",
    "| Solubility =\n",
    "  }}\n",
    "|Section3={{Chembox Hazards\n",
    "| MainHazards =\n",
    "| FlashPt =\n",
    "| AutoignitionPt = \n",
    " }}\n",
    "}}\n",
    "\n",
    "'''1-(2-Nitrophenoxy)octane''', also known as '''nitrophenyl octyl ether''' and abbreviated '''NPOE''', is a \n",
    "[[chemical compound]] that is used as a matrix in [[fast atom bombardment]] [[mass spectrometry]], liquid \n",
    "[[secondary ion mass spectrometry]], and as a highly [[lipophilic]] [[plasticizer]] in [[polymer]] \n",
    "[[Polymeric membrane|membranes]] used in [[ion selective electrode]]s.\n",
    "\n",
    "== See also ==\n",
    "\n",
    "* [[Glycerol]]\n",
    "* [[3-Mercaptopropane-1,2-diol]]\n",
    "* [[3-Nitrobenzyl alcohol]]\n",
    "* [[18-Crown-6]]\n",
    "* [[Sulfolane]]\n",
    "* [[Diethanolamine]]\n",
    "* [[Triethanolamine]]\n",
    "\n",
    "{{DEFAULTSORT:Nitrophenoxy)octane, 1-(2-}}\n",
    "[[Category:Nitrobenzenes]]\n",
    "[[Category:Phenol ethers]]</text>\n",
    "      <sha1>0n15t2w0sp7a50fjptoytuyus0vsrww</sha1>\n",
    "    </revision>\n",
    "  </page>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to extract out the specific article text (denoted by the <code>&lt;text></code> field), perhaps capture some other specific fields, remove internal tags, and then create a clean text representation that we can further process. This additional processing includes removing stoplist words, finding and identifying phrases (multiple token chunks), and then tokenizing the text suitable for processing as computer input. \n",
    "\n",
    "There are multiple methods available for this kind of processing. One approach, for example, uses [XML parsing and specific code](https://www.heatonresearch.com/2017/03/03/python-basic-wikipedia-parsing.html) geared to the Wikipedia dump. Another approach uses a [dedicated Wikipedia extractor](https://www.heatonresearch.com/2017/03/03/python-basic-wikipedia-parsing.html). There are actually a few variants of dedicated extractors.\n",
    "\n",
    "However, one particular Python package, gensim, provides multiple utilities and [Wikipedia services](https://radimrehurek.com/gensim/corpora/wikicorpus.html). Since I had already identified gensim to provide services like sentiment analysis and some other NLP capabilities, I chose to focus on using this package for the needed Wikipedia cleaning tasks.\n",
    "\n",
    "Gensim has a <code>gensim.corpora.wikicorpus.WikiCorpus</code> class designed specifically for processing the Wikipedia article dump file. Fortunately, I was able to find some example code on KDnuggets that showed the way in how to process this file\n",
    "- https://stackoverflow.com/questions/56715394/how-do-i-use-the-wikipedia-dump-as-a-gensim-model (doc2vec example) and\n",
    "- https://www.kdnuggets.com/2017/11/building-wikipedia-text-corpus-nlp.html as another.\n",
    "\n",
    "However, prior to using gensim, I needed to combine the batch outputs from my Wikipedia page retrievals into a single <code>xml</code> file, which I could then bzip for direct ingest by gensim. (Most gensim models and capabilities can read either bzip or text files.)\n",
    "\n",
    "Each 5 K <code>xml</code> page retrieval from Wikipedia comes with its own header and closing tags. These need to be manually snipped out of the group retrieval files before combining. We prepared these into nine blocks that corresponded to each of the batch Wikipedia retrievals, and retained the header and closing tags in the first and last files respectively:\n",
    "\n",
    "<div style=\"background-color:#ffecec; border:1px dotted #f5aca6; vertical-align:middle; margin:15px 60px; padding:8px;\"><span style=\"font-weight: bold;\">NOTE:</span> Due to GitHub's file size limits (of 100 MB max), the nine text files listed in the next routine have been zipped and uploaded to <a href=\"kbpedia.org/cwpk-text/Wikipedia-pages-1.zip\">kbpedia.org/cwpk-text/Wikipedia-pages-1.zip</a>. To use these files, you will need to download to your local system and unzip. Increment the zip files as shown in the link through #9. Then, all following routines below must be repeated locally in order to progress through the various cleaning and preparation steps.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wikipedia-pages-full.xml'\n",
    "filenames = [r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\Wikipedia-pages-1.txt', \n",
    "             r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\Wikipedia-pages-2.txt', \n",
    "             r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\Wikipedia-pages-3.txt', \n",
    "             r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\Wikipedia-pages-4.txt', \n",
    "             r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\Wikipedia-pages-5.txt', \n",
    "             r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\Wikipedia-pages-6.txt', \n",
    "             r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\Wikipedia-pages-7.txt', \n",
    "             r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\Wikipedia-pages-8.txt', \n",
    "             r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\Wikipedia-pages-9.txt']\n",
    "with open(out_f, 'w', encoding='utf-8') as outfile:\n",
    "    for fname in filenames:\n",
    "        with open(fname, encoding='utf-8', errors='ignore') as infile:\n",
    "            i = 0\n",
    "            for line in infile:\n",
    "                i = i + 1\n",
    "                try:\n",
    "                    outfile.write(line)\n",
    "                except Exception as e:\n",
    "                    print('Error at line:' + i + str(e))\n",
    "            print('Now combined:' + fname)\n",
    "    outfile.close \n",
    "    print('Now all files combined!')            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of this routine is then bzipped offline, and then used as the submission to the gensim <code>WikiCorpus</code> function that processes the standard <code>xml</code> output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Creates a corpus from Wikipedia dump file.\n",
    "Inspired by:\n",
    "https://github.com/panyang/Wikipedia_Word2vec/blob/master/v1/process_wiki.py\n",
    "\"\"\"\n",
    "\n",
    "import sys\n",
    "from gensim.corpora import WikiCorpus\n",
    "\n",
    "in_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wikipedia-pages-full.xml.bz2'\n",
    "out_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wikipedia-output-full.txt'\n",
    "\n",
    "def make_corpus(in_f, out_f):\n",
    "\n",
    "    \"\"\"Convert Wikipedia xml dump file to text corpus\"\"\"\n",
    "\n",
    "    output = open(out_f, 'w', encoding='utf-8')            # made change\n",
    "    wiki = WikiCorpus(in_f)\n",
    "    i = 0\n",
    "    for text in wiki.get_texts():\n",
    "        try:\n",
    "            output.write(' '.join(map(lambda x:x.decode('utf-8'), text)) + '\\n')\n",
    "        except Exception as e:\n",
    "            print ('Exception error: ' + str(e))\n",
    "        i = i + 1\n",
    "        if (i % 10000 == 0):\n",
    "            print('Processed ' + str(i) + ' articles')\n",
    "    output.close()\n",
    "    print('Processed ' + str(i) + ' articles;')\n",
    "    print('Processing complete!')\n",
    "\n",
    "make_corpus(in_f, out_f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We further make a smaller input file, <code>enwiki-test-corpus.xml.bz2</code>, with only a few records from the Wikipedia XML dump in order to speed testing of the above code.\n",
    "\n",
    "### Initial Results\n",
    "\n",
    "Here is what the sample program produced for the entry for <code>1-(2-Nitrophenoxy)octane</code> listed above:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code>nitrophenoxy octane also known as nitrophenyl octyl ether and abbreviated npoe is chemical compound that is used as matrix in fast atom bombardment mass spectrometry liquid secondary ion mass spectrometry and as highly lipophilic plasticizer in polymer membranes used in ion selective electrodes see also glycerol mercaptopropane diol nitrobenzyl alcohol crown sulfolane diethanolamine triethanolamine</code>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see a couple of things that are perhaps not in keeping with the extracted information we desire:\n",
    "\n",
    "1. No title\n",
    "1. No sentence boundaries\n",
    "1. No internal category links\n",
    "1. No infobox specifications\n",
    "\n",
    "On the other hand, we do get the content from the 'See Also' section.\n",
    "\n",
    "We want sentence boundaries for cleaner training purposes for word embedding models like word2vec. We want the other items so as to improve the lexical richness and context for the given concept. Further, we want two versions: one with titles as a separate field and one for learning purposes that includes the title in the lexicon (titles, after all, are preferred labels and deserve an additional frequency boost).\n",
    "\n",
    "OK, so how does one make these modifications? My first hope was that arguments to these functions (<code>args</code>) might provide the specification latitude to deal with these changes. Unfortunately, none of the specified items fell into this category, though there is much latitude to modify underlying procedures. The second option was to find some third-party modification or override. Indeed, I did [find one](https://github.com/RaRe-Technologies/gensim/issues/552), that I found quite intriguing as a way to at least deal with sentence boundaries and possibly other areas. I spent nearly a full day trying to adapt this script, never succeeding. One fix would lead to another need for a fix, research on that problem, and then a fix and more problems. I'm sure most all of this is due to my amateur programming skills.\n",
    "\n",
    "Still, the effort was frustrating. The good thing, however, is that in trying to work out a third-party fix, I was learning the underlying module. Eventually, it became clear if I was to address all desired areas it was smartest to modify the source directly. The three key functions that emerged as needing attention were <code>tokenize</code>, <code>process_article</code> and the <code>class WikiCorpus(TextCorpus)</code> code. In fact, it was the text processing heart of the last class that was the focus for changes, but the other two functions got involved because of their supporting roles. As I attempted to sub-class this basis with my own parallel approach (<code>class KBWikiCorpus(WikiCorpus)</code>, I kept finding the need to bring into the picture more supporting functions. Some of this may have been due to nuances in how to specify imported functions and modules, which I am still learning about (see concluding installments). But it is also difficult to sub-set or modify any code. \n",
    "\n",
    "The real impact of these investigations was to help me understand the underlying module. What at first blush looked too intimidating, now was becoming understandable. I could also see other portions of the underlying module that addressed **ALL** aspects of my earlier desires. Third-party modifications choose their own scope; direct modification of the underlying module provides more aspects to tweak. So, I switched emphasis from modifying a third-party overlay to directly changing the core underlying module.\n",
    "\n",
    "### Modifying WikiCorpus\n",
    "We already knew the key functions needing focus. All changes to be made occur in the <code>wikicorpus.py</code> file that resides in your gensim package directory under Python packages. So, I make a copy of the original and name it such, then proceed to modify the base file. Though we will substitute this modified <code>wikicorpus_kb.py</code> file, I will also keep a backup of it as well such that we have copies of the original and modified file.\n",
    "\n",
    "Here is the resulting modified code, with notes about key changes following the listing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#!/usr/bin/env python\n",
      "# -*- coding: utf-8 -*-\n",
      "#\n",
      "# Copyright (C) 2010 Radim Rehurek <radimrehurek@seznam.cz>\n",
      "# Copyright (C) 2012 Lars Buitinck <larsmans@gmail.com>\n",
      "# Copyright (C) 2018 Emmanouil Stergiadis <em.stergiadis@gmail.com>\n",
      "# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html\n",
      "\n",
      "\"\"\"Construct a corpus from a Wikipedia (or other MediaWiki-based) database dump.\n",
      "\n",
      "Uses multiprocessing internally to parallelize the work and process the dump more quickly.\n",
      "\n",
      "Notes\n",
      "-----\n",
      "If you have the `pattern <https://github.com/clips/pattern>`_ package installed,\n",
      "this module will use a fancy lemmatization to get a lemma of each token (instead of plain alphabetic tokenizer).\n",
      "\n",
      "See :mod:`gensim.scripts.make_wiki` for a canned (example) command-line script based on this module.\n",
      "\n",
      "\"\"\"\n",
      "\n",
      "import bz2\n",
      "import logging\n",
      "import multiprocessing\n",
      "import re\n",
      "import signal\n",
      "from pickle import PicklingError\n",
      "# LXML isn't faster, so let's go with the built-in solution\n",
      "try:\n",
      "    from xml.etree.cElementTree import iterparse\n",
      "except ImportError:\n",
      "    from xml.etree.ElementTree import iterparse\n",
      "\n",
      "\n",
      "from gensim import utils\n",
      "# cannot import whole gensim.corpora, because that imports wikicorpus...\n",
      "from gensim.corpora.dictionary import Dictionary\n",
      "from gensim.corpora.textcorpus import TextCorpus\n",
      "\n",
      "from six import raise_from\n",
      "\n",
      "\n",
      "logger = logging.getLogger(__name__)\n",
      "\n",
      "ARTICLE_MIN_WORDS = 50\n",
      "\"\"\"Ignore shorter articles (after full preprocessing).\"\"\"\n",
      "\n",
      "# default thresholds for lengths of individual tokens\n",
      "TOKEN_MIN_LEN = 2\n",
      "TOKEN_MAX_LEN = 15\n",
      "\n",
      "RE_P0 = re.compile(r'<!--.*?-->', re.DOTALL | re.UNICODE)\n",
      "\"\"\"Comments.\"\"\"\n",
      "RE_P1 = re.compile(r'<ref([> ].*?)(</ref>|/>)', re.DOTALL | re.UNICODE)\n",
      "\"\"\"Footnotes.\"\"\"\n",
      "RE_P2 = re.compile(r'(\\n\\[\\[[a-z][a-z][\\w-]*:[^:\\]]+\\]\\])+$', re.UNICODE)\n",
      "\"\"\"Links to languages.\"\"\"\n",
      "RE_P3 = re.compile(r'{{([^}{]*)}}', re.DOTALL | re.UNICODE)\n",
      "\"\"\"Template.\"\"\"\n",
      "RE_P4 = re.compile(r'{{([^}]*)}}', re.DOTALL | re.UNICODE)\n",
      "\"\"\"Template.\"\"\"\n",
      "RE_P5 = re.compile(r'\\[(\\w+):\\/\\/(.*?)(( (.*?))|())\\]', re.UNICODE)\n",
      "\"\"\"Remove URL, keep description.\"\"\"\n",
      "RE_P6 = re.compile(r'\\[([^][]*)\\|([^][]*)\\]', re.DOTALL | re.UNICODE)\n",
      "\"\"\"Simplify links, keep description.\"\"\"\n",
      "RE_P7 = re.compile(r'\\n\\[\\[[iI]mage(.*?)(\\|.*?)*\\|(.*?)\\]\\]', re.UNICODE)\n",
      "\"\"\"Keep description of images.\"\"\"\n",
      "RE_P8 = re.compile(r'\\n\\[\\[[fF]ile(.*?)(\\|.*?)*\\|(.*?)\\]\\]', re.UNICODE)\n",
      "\"\"\"Keep description of files.\"\"\"\n",
      "RE_P9 = re.compile(r'<nowiki([> ].*?)(</nowiki>|/>)', re.DOTALL | re.UNICODE)\n",
      "\"\"\"External links.\"\"\"\n",
      "RE_P10 = re.compile(r'<math([> ].*?)(</math>|/>)', re.DOTALL | re.UNICODE)\n",
      "\"\"\"Math content.\"\"\"\n",
      "RE_P11 = re.compile(r'<(.*?)>', re.DOTALL | re.UNICODE)\n",
      "\"\"\"All other tags.\"\"\"\n",
      "RE_P12 = re.compile(r'(({\\|)|(\\|-(?!\\d))|(\\|}))(.*?)(?=\\n)', re.UNICODE)\n",
      "\"\"\"Table formatting.\"\"\"\n",
      "RE_P13 = re.compile(r'(?<=(\\n[ ])|(\\n\\n)|([ ]{2})|(.\\n)|(.\\t))(\\||\\!)([^[\\]\\n]*?\\|)*', re.UNICODE)\n",
      "\"\"\"Table cell formatting.\"\"\"\n",
      "RE_P14 = re.compile(r'\\[\\[Category:[^][]*\\]\\]', re.UNICODE)\n",
      "\"\"\"Categories.\"\"\"\n",
      "RE_P15 = re.compile(r'\\[\\[([fF]ile:|[iI]mage)[^]]*(\\]\\])', re.UNICODE)\n",
      "\"\"\"Remove File and Image templates.\"\"\"\n",
      "RE_P16 = re.compile(r'\\[{2}(.*?)\\]{2}', re.UNICODE)\n",
      "\"\"\"Capture interlinks text and article linked\"\"\"\n",
      "RE_P17 = re.compile(\n",
      "    r'(\\n.{0,4}((bgcolor)|(\\d{0,1}[ ]?colspan)|(rowspan)|(style=)|(class=)|(align=)|(scope=))(.*))|'\n",
      "    r'(^.{0,2}((bgcolor)|(\\d{0,1}[ ]?colspan)|(rowspan)|(style=)|(class=)|(align=))(.*))',\n",
      "    re.UNICODE\n",
      ")\n",
      "\"\"\"Table markup\"\"\"\n",
      "IGNORED_NAMESPACES = [\n",
      "    'Wikipedia', 'Category', 'File', 'Portal', 'Template',\n",
      "    'MediaWiki', 'User', 'Help', 'Book', 'Draft', 'WikiProject',\n",
      "    'Special', 'Talk'\n",
      "]\n",
      "\"\"\"MediaWiki namespaces that ought to be ignored.\"\"\"\n",
      "\n",
      "\n",
      "def filter_example(elem, text, *args, **kwargs):\n",
      "    \"\"\"Example function for filtering arbitrary documents from wikipedia dump.\n",
      "\n",
      "\n",
      "    The custom filter function is called _before_ tokenisation and should work on\n",
      "    the raw text and/or XML element information.\n",
      "\n",
      "    The filter function gets the entire context of the XML element passed into it,\n",
      "    but you can of course choose not the use some or all parts of the context. Please\n",
      "    refer to :func:`gensim.corpora.wikicorpus.extract_pages` for the exact details\n",
      "    of the page context.\n",
      "\n",
      "    Parameters\n",
      "    ----------\n",
      "    elem : etree.Element\n",
      "        XML etree element\n",
      "    text : str\n",
      "        The text of the XML node\n",
      "    namespace : str\n",
      "        XML namespace of the XML element\n",
      "    title : str\n",
      "       Page title\n",
      "    page_tag : str\n",
      "        XPath expression for page.\n",
      "    text_path : str\n",
      "        XPath expression for text.\n",
      "    title_path : str\n",
      "        XPath expression for title.\n",
      "    ns_path : str\n",
      "        XPath expression for namespace.\n",
      "    pageid_path : str\n",
      "        XPath expression for page id.\n",
      "\n",
      "    Example\n",
      "    -------\n",
      "    .. sourcecode:: pycon\n",
      "\n",
      "        >>> import gensim.corpora\n",
      "        >>> filter_func = gensim.corpora.wikicorpus.filter_example\n",
      "        >>> dewiki = gensim.corpora.WikiCorpus(\n",
      "        ...     './dewiki-20180520-pages-articles-multistream.xml.bz2',\n",
      "        ...     filter_articles=filter_func)\n",
      "\n",
      "    \"\"\"\n",
      "    # Filter German wikipedia dump for articles that are marked either as\n",
      "    # Lesenswert (featured) or Exzellent (excellent) by wikipedia editors.\n",
      "    # *********************\n",
      "    # regex is in the function call so that we do not pollute the wikicorpus\n",
      "    # namespace do not do this in production as this function is called for\n",
      "    # every element in the wiki dump\n",
      "    _regex_de_excellent = re.compile(r'.*\\{\\{(Exzellent.*?)\\}\\}[\\s]*', flags=re.DOTALL)\n",
      "    _regex_de_featured = re.compile(r'.*\\{\\{(Lesenswert.*?)\\}\\}[\\s]*', flags=re.DOTALL)\n",
      "\n",
      "    if text is None:\n",
      "        return False\n",
      "    if _regex_de_excellent.match(text) or _regex_de_featured.match(text):\n",
      "        return True\n",
      "    else:\n",
      "        return False\n",
      "\n",
      "\n",
      "def find_interlinks(raw):\n",
      "    \"\"\"Find all interlinks to other articles in the dump.\n",
      "\n",
      "    Parameters\n",
      "    ----------\n",
      "    raw : str\n",
      "        Unicode or utf-8 encoded string.\n",
      "\n",
      "    Returns\n",
      "    -------\n",
      "    list\n",
      "        List of tuples in format [(linked article, the actual text found), ...].\n",
      "\n",
      "    \"\"\"\n",
      "    filtered = filter_wiki(raw, promote_remaining=False, simplify_links=False)\n",
      "    interlinks_raw = re.findall(RE_P16, filtered)\n",
      "\n",
      "    interlinks = []\n",
      "    for parts in [i.split('|') for i in interlinks_raw]:\n",
      "        actual_title = parts[0]\n",
      "        try:\n",
      "            interlink_text = parts[1]\n",
      "        except IndexError:\n",
      "            interlink_text = actual_title\n",
      "        interlink_tuple = (actual_title, interlink_text)\n",
      "        interlinks.append(interlink_tuple)\n",
      "\n",
      "    legit_interlinks = [(i, j) for i, j in interlinks if '[' not in i and ']' not in i]\n",
      "    return legit_interlinks\n",
      "\n",
      "\n",
      "def filter_wiki(raw, promote_remaining=True, simplify_links=True):\n",
      "    \"\"\"Filter out wiki markup from `raw`, leaving only text.\n",
      "\n",
      "    Parameters\n",
      "    ----------\n",
      "    raw : str\n",
      "        Unicode or utf-8 encoded string.\n",
      "    promote_remaining : bool\n",
      "        Whether uncaught markup should be promoted to plain text.\n",
      "    simplify_links : bool\n",
      "        Whether links should be simplified keeping only their description text.\n",
      "\n",
      "    Returns\n",
      "    -------\n",
      "    str\n",
      "        `raw` without markup.\n",
      "\n",
      "    \"\"\"\n",
      "    # parsing of the wiki markup is not perfect, but sufficient for our purposes\n",
      "    # contributions to improving this code are welcome :)\n",
      "    text = utils.to_unicode(raw, 'utf8', errors='ignore')\n",
      "    text = utils.decode_htmlentities(text)  # '&amp;nbsp;' --> '\\xa0'\n",
      "    return remove_markup(text, promote_remaining, simplify_links)\n",
      "\n",
      "\n",
      "def remove_markup(text, promote_remaining=True, simplify_links=True):\n",
      "    \"\"\"Filter out wiki markup from `text`, leaving only text.\n",
      "\n",
      "    Parameters\n",
      "    ----------\n",
      "    text : str\n",
      "        String containing markup.\n",
      "    promote_remaining : bool\n",
      "        Whether uncaught markup should be promoted to plain text.\n",
      "    simplify_links : bool\n",
      "        Whether links should be simplified keeping only their description text.\n",
      "\n",
      "    Returns\n",
      "    -------\n",
      "    str\n",
      "        `text` without markup.\n",
      "\n",
      "    \"\"\"\n",
      "    text = re.sub(RE_P2, '', text)  # remove the last list (=languages)\n",
      "    # the wiki markup is recursive (markup inside markup etc)\n",
      "    # instead of writing a recursive grammar, here we deal with that by removing\n",
      "    # markup in a loop, starting with inner-most expressions and working outwards,\n",
      "    # for as long as something changes.\n",
      "#    text = remove_template(text)                                # Note\n",
      "    text = remove_file(text)\n",
      "    iters = 0\n",
      "    while True:\n",
      "        old, iters = text, iters + 1\n",
      "        text = re.sub(RE_P0, '', text)  # remove comments\n",
      "        text = re.sub(RE_P1, '', text)  # remove footnotes\n",
      "        text = re.sub(RE_P9, '', text)  # remove outside links\n",
      "        text = re.sub(RE_P10, '', text)  # remove math content\n",
      "        text = re.sub(RE_P11, '', text)  # remove all remaining tags\n",
      "#        text = re.sub(RE_P14, '', text)  # remove categories                           # Note\n",
      "        text = re.sub(RE_P5, '\\\\3', text)  # remove urls, keep description\n",
      "\n",
      "        if simplify_links:\n",
      "            text = re.sub(RE_P6, '\\\\2', text)  # simplify links, keep description only\n",
      "        # remove table markup\n",
      "        text = text.replace(\"!!\", \"\\n|\")  # each table head cell on a separate line\n",
      "        text = text.replace(\"|-||\", \"\\n|\")  # for cases where a cell is filled with '-'\n",
      "        text = re.sub(RE_P12, '\\n', text)  # remove formatting lines\n",
      "        text = text.replace('|||', '|\\n|')  # each table cell on a separate line(where |{{a|b}}||cell-content)\n",
      "        text = text.replace('||', '\\n|')  # each table cell on a separate line\n",
      "        text = re.sub(RE_P13, '\\n', text)  # leave only cell content\n",
      "        text = re.sub(RE_P17, '\\n', text)  # remove formatting lines\n",
      "\n",
      "        # remove empty mark-up\n",
      "        text = text.replace('[]', '')\n",
      "        # stop if nothing changed between two iterations or after a fixed number of iterations\n",
      "        if old == text or iters > 2:\n",
      "            break\n",
      "\n",
      "    if promote_remaining:\n",
      "        text = text.replace('[', '').replace(']', '')  # promote all remaining markup to plain text\n",
      "\n",
      "    return text\n",
      "\n",
      "\n",
      "def remove_template(s):\n",
      "    \"\"\"Remove template wikimedia markup.\n",
      "\n",
      "    Parameters\n",
      "    ----------\n",
      "    s : str\n",
      "        String containing markup template.\n",
      "\n",
      "    Returns\n",
      "    -------\n",
      "    str\n",
      "        Ð¡opy of `s` with all the `wikimedia markup template <http://meta.wikimedia.org/wiki/Help:Template>`_ removed.\n",
      "\n",
      "    Notes\n",
      "    -----\n",
      "    Since template can be nested, it is difficult remove them using regular expressions.\n",
      "\n",
      "    \"\"\"\n",
      "    # Find the start and end position of each template by finding the opening\n",
      "    # '{{' and closing '}}'\n",
      "    n_open, n_close = 0, 0\n",
      "    starts, ends = [], [-1]\n",
      "    in_template = False\n",
      "    prev_c = None\n",
      "    for i, c in enumerate(s):\n",
      "        if not in_template:\n",
      "            if c == '{' and c == prev_c:\n",
      "                starts.append(i - 1)\n",
      "                in_template = True\n",
      "                n_open = 1\n",
      "        if in_template:\n",
      "            if c == '{':\n",
      "                n_open += 1\n",
      "            elif c == '}':\n",
      "                n_close += 1\n",
      "            if n_open == n_close:\n",
      "                ends.append(i)\n",
      "                in_template = False\n",
      "                n_open, n_close = 0, 0\n",
      "        prev_c = c\n",
      "\n",
      "    # Remove all the templates\n",
      "    starts.append(None)\n",
      "    return ''.join(s[end + 1:start] for end, start in zip(ends, starts))\n",
      "\n",
      "\n",
      "def remove_file(s):\n",
      "    \"\"\"Remove the 'File:' and 'Image:' markup, keeping the file caption.\n",
      "\n",
      "    Parameters\n",
      "    ----------\n",
      "    s : str\n",
      "        String containing 'File:' and 'Image:' markup.\n",
      "\n",
      "    Returns\n",
      "    -------\n",
      "    str\n",
      "        Ð¡opy of `s` with all the 'File:' and 'Image:' markup replaced by their `corresponding captions\n",
      "        <http://www.mediawiki.org/wiki/Help:Images>`_.\n",
      "\n",
      "    \"\"\"\n",
      "    # The regex RE_P15 match a File: or Image: markup\n",
      "    for match in re.finditer(RE_P15, s):\n",
      "        m = match.group(0)\n",
      "        caption = m[:-2].split('|')[-1]\n",
      "        s = s.replace(m, caption, 1)\n",
      "    return s\n",
      "\n",
      "def tokenize(content):\n",
      "# ORIGINAL VERSION\n",
      "#def tokenize(content, token_min_len=TOKEN_MIN_LEN, token_max_len=TOKEN_MAX_LEN, lower=True):\n",
      "    \"\"\"Tokenize a piece of text from Wikipedia.\n",
      "\n",
      "    Set `token_min_len`, `token_max_len` as character length (not bytes!) thresholds for individual tokens.\n",
      "\n",
      "    Parameters\n",
      "    ----------\n",
      "    content : str\n",
      "        String without markup (see :func:`~gensim.corpora.wikicorpus.filter_wiki`).\n",
      "    token_min_len : int\n",
      "        Minimal token length.\n",
      "    token_max_len : int\n",
      "        Maximal token length.\n",
      "    lower : bool\n",
      "         Convert `content` to lower case?\n",
      "\n",
      "    Returns\n",
      "    -------\n",
      "    list of str\n",
      "        List of tokens from `content`.\n",
      "\n",
      "    \"\"\"\n",
      "    # ORIGINAL VERSION\n",
      "    # TODO maybe ignore tokens with non-latin characters? (no chinese, arabic, russian etc.)\n",
      "#    return [\n",
      "#        utils.to_unicode(token) for token in utils.tokenize(content, lower=lower, errors='ignore')\n",
      "#        if token_min_len <= len(token) <= token_max_len and not token.startswith('_')\n",
      "#    ]\n",
      "# NEW VERSION\n",
      "    return [token.encode('utf8') for token in utils.tokenize(content, lower=True, errors='ignore')\n",
      "            if len(token) <= 15 and not token.startswith('_')]\n",
      "# TO RESTORE MOST PUNCTUATION\n",
      "#    return [token.encode('utf8') for token in content.split() \n",
      "#           if len(token) <= 15 and not token.startswith('_')]\n",
      "\n",
      "def get_namespace(tag):\n",
      "    \"\"\"Get the namespace of tag.\n",
      "\n",
      "    Parameters\n",
      "    ----------\n",
      "    tag : str\n",
      "        Namespace or tag.\n",
      "\n",
      "    Returns\n",
      "    -------\n",
      "    str\n",
      "        Matched namespace or tag.\n",
      "\n",
      "    \"\"\"\n",
      "    m = re.match(\"^{(.*?)}\", tag)\n",
      "    namespace = m.group(1) if m else \"\"\n",
      "    if not namespace.startswith(\"http://www.mediawiki.org/xml/export-\"):\n",
      "        raise ValueError(\"%s not recognized as MediaWiki dump namespace\" % namespace)\n",
      "    return namespace\n",
      "\n",
      "\n",
      "_get_namespace = get_namespace\n",
      "\n",
      "\n",
      "def extract_pages(f, filter_namespaces=False, filter_articles=None):\n",
      "    \"\"\"Extract pages from a MediaWiki database dump.\n",
      "\n",
      "    Parameters\n",
      "    ----------\n",
      "    f : file\n",
      "        File-like object.\n",
      "    filter_namespaces : list of str or bool\n",
      "         Namespaces that will be extracted.\n",
      "\n",
      "    Yields\n",
      "    ------\n",
      "    tuple of (str or None, str, str)\n",
      "        Title, text and page id.\n",
      "\n",
      "    \"\"\"\n",
      "    elems = (elem for _, elem in iterparse(f, events=(\"end\",)))\n",
      "\n",
      "    # We can't rely on the namespace for database dumps, since it's changed\n",
      "    # it every time a small modification to the format is made. So, determine\n",
      "    # those from the first element we find, which will be part of the metadata,\n",
      "    # and construct element paths.\n",
      "    elem = next(elems)\n",
      "    namespace = get_namespace(elem.tag)\n",
      "    ns_mapping = {\"ns\": namespace}\n",
      "    page_tag = \"{%(ns)s}page\" % ns_mapping\n",
      "    text_path = \"./{%(ns)s}revision/{%(ns)s}text\" % ns_mapping\n",
      "    title_path = \"./{%(ns)s}title\" % ns_mapping\n",
      "    ns_path = \"./{%(ns)s}ns\" % ns_mapping\n",
      "    pageid_path = \"./{%(ns)s}id\" % ns_mapping\n",
      "\n",
      "    for elem in elems:\n",
      "        if elem.tag == page_tag:\n",
      "            title = elem.find(title_path).text\n",
      "            text = elem.find(text_path).text\n",
      "\n",
      "            if filter_namespaces:\n",
      "                ns = elem.find(ns_path).text\n",
      "                if ns not in filter_namespaces:\n",
      "                    text = None\n",
      "\n",
      "            if filter_articles is not None:\n",
      "                if not filter_articles(\n",
      "                        elem, namespace=namespace, title=title,\n",
      "                        text=text, page_tag=page_tag,\n",
      "                        text_path=text_path, title_path=title_path,\n",
      "                        ns_path=ns_path, pageid_path=pageid_path):\n",
      "                    text = None\n",
      "\n",
      "            pageid = elem.find(pageid_path).text\n",
      "            yield title, text or \"\", pageid  # empty page will yield None\n",
      "\n",
      "            # Prune the element tree, as per\n",
      "            # http://www.ibm.com/developerworks/xml/library/x-hiperfparse/\n",
      "            # except that we don't need to prune backlinks from the parent\n",
      "            # because we don't use LXML.\n",
      "            # We do this only for <page>s, since we need to inspect the\n",
      "            # ./revision/text element. The pages comprise the bulk of the\n",
      "            # file, so in practice we prune away enough.\n",
      "            elem.clear()\n",
      "\n",
      "_extract_pages = extract_pages  # for backward compatibility\n",
      "\n",
      "\n",
      "def process_article(args):\n",
      "# ORIGINAL VERSION\n",
      "#def process_article(args, tokenizer_func=tokenize, token_min_len=TOKEN_MIN_LEN,\n",
      "#                    token_max_len=TOKEN_MAX_LEN, lower=True):\n",
      "    \"\"\"Parse a Wikipedia article, extract all tokens.\n",
      "\n",
      "    Notes\n",
      "    -----\n",
      "    Set `tokenizer_func` (defaults is :func:`~gensim.corpora.wikicorpus.tokenize`) parameter for languages\n",
      "    like Japanese or Thai to perform better tokenization.\n",
      "    The `tokenizer_func` needs to take 4 parameters: (text: str, token_min_len: int, token_max_len: int, lower: bool).\n",
      "\n",
      "    Parameters\n",
      "    ----------\n",
      "    args : (str, bool, str, int)\n",
      "        Article text, lemmatize flag (if True, :func:`~gensim.utils.lemmatize` will be used), article title,\n",
      "        page identificator.\n",
      "    tokenizer_func : function\n",
      "        Function for tokenization (defaults is :func:`~gensim.corpora.wikicorpus.tokenize`).\n",
      "        Needs to have interface:\n",
      "        tokenizer_func(text: str, token_min_len: int, token_max_len: int, lower: bool) -> list of str.\n",
      "    token_min_len : int\n",
      "        Minimal token length.\n",
      "    token_max_len : int\n",
      "        Maximal token length.\n",
      "    lower : bool\n",
      "         Convert article text to lower case?\n",
      "\n",
      "    Returns\n",
      "    -------\n",
      "    (list of str, str, int)\n",
      "        List of tokens from article, title and page id.\n",
      "\n",
      "    \"\"\"\n",
      "\n",
      "    text, lemmatize, title, pageid = args\n",
      "    text = filter_wiki(text)\n",
      "    if lemmatize:\n",
      "        result = utils.lemmatize(text)\n",
      "    else:\n",
      "# ORIGINAL VERSION\n",
      "#        result = tokenizer_func(text, token_min_len, token_max_len, lower)\n",
      "# NEW VERSION\n",
      "        result = tokenize(text)\n",
      "#        result = title + text\n",
      "    return result, title, pageid\n",
      "\n",
      "\n",
      "def init_to_ignore_interrupt():\n",
      "    \"\"\"Enables interruption ignoring.\n",
      "\n",
      "    Warnings\n",
      "    --------\n",
      "    Should only be used when master is prepared to handle termination of\n",
      "    child processes.\n",
      "\n",
      "    \"\"\"\n",
      "    signal.signal(signal.SIGINT, signal.SIG_IGN)\n",
      "\n",
      "\n",
      "def _process_article(args):\n",
      "    \"\"\"Same as :func:`~gensim.corpora.wikicorpus.process_article`, but with args in list format.\n",
      "\n",
      "    Parameters\n",
      "    ----------\n",
      "    args : [(str, bool, str, int), (function, int, int, bool)]\n",
      "        First element - same as `args` from :func:`~gensim.corpora.wikicorpus.process_article`,\n",
      "        second element is tokenizer function, token minimal length, token maximal length, lowercase flag.\n",
      "\n",
      "    Returns\n",
      "    -------\n",
      "    (list of str, str, int)\n",
      "        List of tokens from article, title and page id.\n",
      "\n",
      "    Warnings\n",
      "    --------\n",
      "    Should not be called explicitly. Use :func:`~gensim.corpora.wikicorpus.process_article` instead.\n",
      "\n",
      "    \"\"\"\n",
      "    tokenizer_func, token_min_len, token_max_len, lower = args[-1]\n",
      "    args = args[:-1]\n",
      "\n",
      "    return process_article(\n",
      "        args, tokenizer_func=tokenizer_func, token_min_len=token_min_len,\n",
      "        token_max_len=token_max_len, lower=lower\n",
      "    )\n",
      "\n",
      "\n",
      "class WikiCorpus(TextCorpus):\n",
      "    \"\"\"Treat a Wikipedia articles dump as a read-only, streamed, memory-efficient corpus.\n",
      "\n",
      "    Supported dump formats:\n",
      "\n",
      "    * <LANG>wiki-<YYYYMMDD>-pages-articles.xml.bz2\n",
      "    * <LANG>wiki-latest-pages-articles.xml.bz2\n",
      "\n",
      "    The documents are extracted on-the-fly, so that the whole (massive) dump can stay compressed on disk.\n",
      "\n",
      "    Notes\n",
      "    -----\n",
      "    Dumps for the English Wikipedia can be founded at https://dumps.wikimedia.org/enwiki/.\n",
      "\n",
      "    Attributes\n",
      "    ----------\n",
      "    metadata : bool\n",
      "        Whether to write articles titles to serialized corpus.\n",
      "\n",
      "    Warnings\n",
      "    --------\n",
      "    \"Multistream\" archives are *not* supported in Python 2 due to `limitations in the core bz2 library\n",
      "    <https://docs.python.org/2/library/bz2.html#de-compression-of-files>`_.\n",
      "\n",
      "    Examples\n",
      "    --------\n",
      "    .. sourcecode:: pycon\n",
      "\n",
      "        >>> from gensim.test.utils import datapath, get_tmpfile\n",
      "        >>> from gensim.corpora import WikiCorpus, MmCorpus\n",
      "        >>>\n",
      "        >>> path_to_wiki_dump = datapath(\"enwiki-latest-pages-articles1.xml-p000000010p000030302-shortened.bz2\")\n",
      "        >>> corpus_path = get_tmpfile(\"wiki-corpus.mm\")\n",
      "        >>>\n",
      "        >>> wiki = WikiCorpus(path_to_wiki_dump)  # create word->word_id mapping, ~8h on full wiki\n",
      "        >>> MmCorpus.serialize(corpus_path, wiki)  # another 8h, creates a file in MatrixMarket format and mapping\n",
      "\n",
      "    \"\"\"\n",
      "    def __init__(self, fname, processes=None, lemmatize=utils.has_pattern(), dictionary=None,\n",
      "                 filter_namespaces=('0',)):\n",
      "# ORIGINAL VERSION\n",
      "#                 filter_namespaces=('0',), tokenizer_func=tokenize, article_min_tokens=ARTICLE_MIN_WORDS,\n",
      "#                 token_min_len=TOKEN_MIN_LEN, token_max_len=TOKEN_MAX_LEN, lower=True, filter_articles=None):\n",
      "        \"\"\"Initialize the corpus.\n",
      "\n",
      "        Unless a dictionary is provided, this scans the corpus once,\n",
      "        to determine its vocabulary.\n",
      "\n",
      "        Parameters\n",
      "        ----------\n",
      "        fname : str\n",
      "            Path to the Wikipedia dump file.\n",
      "        processes : int, optional\n",
      "            Number of processes to run, defaults to `max(1, number of cpu - 1)`.\n",
      "        lemmatize : bool\n",
      "            Use lemmatization instead of simple regexp tokenization.\n",
      "            Defaults to `True` if you have the `pattern <https://github.com/clips/pattern>`_ package installed.\n",
      "        dictionary : :class:`~gensim.corpora.dictionary.Dictionary`, optional\n",
      "            Dictionary, if not provided,  this scans the corpus once, to determine its vocabulary\n",
      "            **IMPORTANT: this needs a really long time**.\n",
      "        filter_namespaces : tuple of str, optional\n",
      "            Namespaces to consider.\n",
      "        tokenizer_func : function, optional\n",
      "            Function that will be used for tokenization. By default, use :func:`~gensim.corpora.wikicorpus.tokenize`.\n",
      "            If you inject your own tokenizer, it must conform to this interface:\n",
      "            `tokenizer_func(text: str, token_min_len: int, token_max_len: int, lower: bool) -> list of str`\n",
      "        article_min_tokens : int, optional\n",
      "            Minimum tokens in article. Article will be ignored if number of tokens is less.\n",
      "        token_min_len : int, optional\n",
      "            Minimal token length.\n",
      "        token_max_len : int, optional\n",
      "            Maximal token length.\n",
      "        lower : bool, optional\n",
      "             If True - convert all text to lower case.\n",
      "        filter_articles: callable or None, optional\n",
      "            If set, each XML article element will be passed to this callable before being processed. Only articles\n",
      "            where the callable returns an XML element are processed, returning None allows filtering out\n",
      "            some articles based on customised rules.\n",
      "\n",
      "        Warnings\n",
      "        --------\n",
      "        Unless a dictionary is provided, this scans the corpus once, to determine its vocabulary.\n",
      "\n",
      "        \"\"\"\n",
      "        self.fname = fname\n",
      "        self.filter_namespaces = filter_namespaces\n",
      "#        self.filter_articles = filter_articles\n",
      "        self.metadata = True\n",
      "        if processes is None:\n",
      "            processes = max(1, multiprocessing.cpu_count() - 1)\n",
      "        self.processes = processes\n",
      "        self.lemmatize = lemmatize\n",
      "#        self.tokenizer_func = tokenizer_func\n",
      "#        self.article_min_tokens = article_min_tokens\n",
      "#        self.token_min_len = token_min_len\n",
      "#        self.token_max_len = token_max_len\n",
      "#        self.lower = lower\n",
      "#        get_title = cur_title\n",
      "\n",
      "        if dictionary is None:\n",
      "            self.dictionary = Dictionary(self.get_texts())\n",
      "        else:\n",
      "            self.dictionary = dictionary\n",
      "\n",
      "    def get_texts(self):\n",
      "        \"\"\"Iterate over the dump, yielding a list of tokens for each article that passed\n",
      "        the length and namespace filtering.\n",
      "\n",
      "        Uses multiprocessing internally to parallelize the work and process the dump more quickly.\n",
      "\n",
      "        Notes\n",
      "        -----\n",
      "        This iterates over the **texts**. If you want vectors, just use the standard corpus interface\n",
      "        instead of this method:\n",
      "\n",
      "        Examples\n",
      "        --------\n",
      "        .. sourcecode:: pycon\n",
      "\n",
      "            >>> from gensim.test.utils import datapath\n",
      "            >>> from gensim.corpora import WikiCorpus\n",
      "            >>>\n",
      "            >>> path_to_wiki_dump = datapath(\"enwiki-latest-pages-articles1.xml-p000000010p000030302-shortened.bz2\")\n",
      "            >>>\n",
      "            >>> for vec in WikiCorpus(path_to_wiki_dump):\n",
      "            ...     pass\n",
      "\n",
      "        Yields\n",
      "        ------\n",
      "        list of str\n",
      "            If `metadata` is False, yield only list of token extracted from the article.\n",
      "        (list of str, (int, str))\n",
      "            List of tokens (extracted from the article), page id and article title otherwise.\n",
      "\n",
      "        \"\"\"\n",
      "        articles, articles_all = 0, 0\n",
      "        positions, positions_all = 0, 0\n",
      "# ORIGINAL VERSION\n",
      "#        tokenization_params = (self.tokenizer_func, self.token_min_len, self.token_max_len, self.lower)\n",
      "#        texts = \\\n",
      "#            ((text, self.lemmatize, title, pageid, tokenization_params)\n",
      "#             for title, text, pageid\n",
      "#             in extract_pages(bz2.BZ2File(self.fname), self.filter_namespaces, self.filter_articles))\n",
      "#        pool = multiprocessing.Pool(self.processes, init_to_ignore_interrupt)\n",
      "# NEW VERSION\n",
      "        texts = ((text, self.lemmatize, title, pageid) for title, text, pageid \n",
      "                in extract_pages(bz2.BZ2File(self.fname), self.filter_namespaces))\n",
      "        pool = multiprocessing.Pool(self.processes)\n",
      "        try:\n",
      "            # process the corpus in smaller chunks of docs, because multiprocessing.Pool\n",
      "            # is dumb and would load the entire input into RAM at once...\n",
      "\n",
      "# ORIGINAL VERSION            \n",
      "#            for group in utils.chunkize(texts, chunksize=10 * self.processes, maxsize=1):\n",
      "# NEW VERSION\n",
      "            for group in utils.chunkize_serial(texts, chunksize=10 * self.processes):\n",
      "# ORIGINAL VERSION\n",
      "#                for tokens, title, pageid in pool.imap(_process_article, group):\n",
      "# NEW VERSION\n",
      "                for tokens, title, pageid in pool.imap(process_article, group):  # chunksize=10):\n",
      "                    articles_all += 1\n",
      "                    positions_all += len(tokens)\n",
      "                    # article redirects and short stubs are pruned here\n",
      "# ORIGINAL VERSION\n",
      "#                    if len(tokens) < self.article_min_tokens or \\\n",
      "#                            any(title.startswith(ignore + ':') for ignore in IGNORED_NAMESPACES):\n",
      "# NEW VERSION FOR ENTIRE BLOCK\n",
      "                    if len(tokens) < ARTICLE_MIN_WORDS or \\\n",
      "                          any(title.startswith(ignore + ':') for ignore in IGNORED_NAMESPACES):\n",
      "                        continue\n",
      "                    articles += 1\n",
      "                    positions += len(tokens)\n",
      "                    try:\n",
      "                        if self.metadata:\n",
      "                            title = title.replace(' ', '_')\n",
      "                            title = (title + ',')\n",
      "                            title = bytes(title, 'utf-8')\n",
      "                            tokens.insert(0,title)\n",
      "                            yield tokens\n",
      "                        else:\n",
      "                            yield tokens\n",
      "                    except Exception as e:\n",
      "                        print('Wikicorpus exception error: ' + str(e))\n",
      "        except KeyboardInterrupt:\n",
      "            logger.warn(\n",
      "                \"user terminated iteration over Wikipedia corpus after %i documents with %i positions \"\n",
      "                \"(total %i articles, %i positions before pruning articles shorter than %i words)\",\n",
      "# ORIGINAL VERSION\n",
      "#                articles, positions, articles_all, positions_all, self.article_min_tokens\n",
      "# NEW VERSION\n",
      "                articles, positions, articles_all, positions_all\n",
      "            )\n",
      "        except PicklingError as exc:\n",
      "            raise_from(PicklingError('Can not send filtering function {} to multiprocessing, '\n",
      "                'make sure the function can be pickled.'.format(self.filter_articles)), exc)\n",
      "        else:\n",
      "            logger.info(\n",
      "                \"finished iterating over Wikipedia corpus of %i documents with %i positions \"\n",
      "                \"(total %i articles, %i positions before pruning articles shorter than %i words)\",\n",
      "# ORIGINAL VERSION\n",
      "#                articles, positions, articles_all, positions_all, self.article_min_tokens\n",
      "# NEW VERSION\n",
      "                articles, positions, articles_all, positions_all\n",
      "            )\n",
      "            self.length = articles  # cache corpus length\n",
      "        finally:\n",
      "            pool.terminate()\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('files/wikicorpus_kb.py', 'r') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gensim provides well documented code that is written in an understandable way.\n",
    "\n",
    "Most of the modifications I made occurred at the bottom of the code listing. However, the text routine at the top of the file allows us to tailor what page 'sections' are kept or not in each Wikipedia article. Because of their substantive lexical content, I add the page templates and category names to be retained with the text body.\n",
    "\n",
    "Assuming I will want to retain these modifications and understand them at a later date, I block off all modified sections with <code>ORIGINAL VERSION</code> and <code>NEW VERSION</code> tags. One change was to remove punctuation. Another was to grab and capture the article title. \n",
    "\n",
    "This file, then, becomes a replacement to the original <code>wikicorpus.py</code> code. I am cognizant that changing underlying source code for local purposes is generally considered to be a **BAD** idea. It very well may be so in this case. However, with the backups, and being attentive to version updates and keeping working code in sync, I guess I do not see where keeping track of a modification is any less sustainable than needing to update existing code to a modification. Both require inspection and effort. If I diff on the changed underlying module, I suspect it is of equivalent effort or lesser effort to change a third-party interface modification.\n",
    "\n",
    "The net result is that I am now capturing the substantive content of these articles in a form I want to process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove Stoplist\n",
    "\n",
    "In my initial workflow, I had the step of stoplist removal later in the process because I thought it might be helpful to have all text prior to phrase identification. A stoplist (also known as '[stop words](https://en.wikipedia.org/wiki/Stop_word)'), by the way, is a listing of very common words (mostly conjuctions, common verb tenses, articles and propositions) that can be removed from a block of text without adversely affecting its meaning or readability.\n",
    "\n",
    "Since it proved superior to not retain these stop words when forming [n-grams](https://en.wikipedia.org/wiki/N-gram) (see next section), I moved the routine up to be next processing of the Wikipedia pages. Here is the relevant code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stopwords applied to 10000 articles\n",
      "Stopwords applied to 20000 articles\n",
      "Stopwords applied to 30000 articles\n",
      "Stopwords applied to 31157 articles;\n",
      "Processing complete!\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "from gensim.parsing.preprocessing import remove_stopwords  # Key line for stoplist\n",
    "from smart_open import smart_open\n",
    "\n",
    "in_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wikipedia-output-full.txt'\n",
    "out_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wikipedia-output-full-stopped.txt'\n",
    "\n",
    "more_stops = ['b', 'c', 'category', 'com', 'd', 'f', 'formatnum', 'g', 'gave', 'gov', 'h', \n",
    "              'htm', 'html', 'http', 'https', 'id', 'isbn', 'j', 'k', 'l', 'loc', 'm', 'n', \n",
    "              'need', 'needed', 'org', 'p', 'properties', 'q', 'r', 's', 'took', 'url', 'use', \n",
    "              'v', 'w', 'www', 'y', 'z']  \n",
    "documents = smart_open(in_f, 'r', encoding='utf-8')\n",
    "content = [doc.split(' ') for doc in documents]\n",
    "with open(out_f, 'w', encoding='utf-8') as output:\n",
    "    i = 0\n",
    "    for line in content:\n",
    "        try:\n",
    "            line = ', '.join(line)\n",
    "            line = line.replace(', ', ' ')\n",
    "            line = remove_stopwords(line)  \n",
    "            querywords = line.split()\n",
    "            resultwords = [word for word in querywords if word.lower() not in more_stops]\n",
    "            line = ' '.join(resultwords)\n",
    "            line = line + '\\n'\n",
    "            output.write(line)\n",
    "        except Exception as e:\n",
    "            print ('Exception error: ' + str(e))\n",
    "        i = i + 1\n",
    "        if (i % 10000 == 0):\n",
    "            print('Stopwords applied to ' + str(i) + ' articles')\n",
    "    output.close()\n",
    "    print('Stopwords applied to ' + str(i) + ' articles;')\n",
    "    print('Processing complete!')  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gensim comes with its own stoplist, to which I added a few of my own, including removal of the <code>category</code> keyword that arose from adding that grouping. The output of this routine is the next file in the pipeline, <code>wikipedia-output-full-stopped.txt</code>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phrase Identification and Extraction\n",
    "Phrases are n-grams, generally composed of two or three paired words, which are known as 'bigrams' and 'trigrams', respectively. Phrases are one of the most powerful ways to capture domain or technical language, since these compounded terms arise through the use and consensus of their users. Some phrases help disambiguate specific entities or places, as when for example 'river', 'state', 'university' or 'buckeyes' does when combined with the term 'ohio'. \n",
    "\n",
    "Generally, most embeddings or corpora do not include n-grams in their initial preparation. But, for the reasons above, and experience of the usefulness of n-grams to text retrieval, we decided to includ phrase identification and extraction as part of our preprocessing.\n",
    "\n",
    "Again, gensim comes with a pre-trained phrase identifier (like all gensim models, you can re-train and tune these models as you gain experience and want them to perform differently). The main work of this routine is the <code>ngram</code> call, wherein term adjacency is used to construct paired term indentifications. Here is the code and settings for our first pass with this function to create our initial bigrams from the stopped input text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ngrams calculated for 10000 articles\n",
      "ngrams calculated for 20000 articles\n",
      "ngrams calculated for 30000 articles\n",
      "Calculated ngrams for 31157 articles;\n",
      "Processing complete!\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "from gensim.models.phrases import Phraser, Phrases\n",
    "from gensim.parsing.preprocessing import remove_stopwords  # Key line for stoplist\n",
    "from smart_open import smart_open\n",
    "\n",
    "in_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wikipedia-output-full-stopped.txt'\n",
    "out_f = r'C:\\1-PythonProjects\\kbpedia\\v300\\models\\inputs\\wikipedia-bigram.txt'\n",
    "\n",
    "documents = smart_open(in_f, 'r', encoding='utf-8')\n",
    "sentence_stream = [doc.split(' ') for doc in documents]\n",
    "common_terms = ['aka']\n",
    "ngram = Phrases(sentence_stream, min_count=3,threshold=10, max_vocab_size=80000000, \n",
    "                delimiter=b'_', common_terms=common_terms)\n",
    "ngram = Phraser(ngram)\n",
    "content = list(ngram[sentence_stream])\n",
    "with open(out_f, 'w', encoding='utf-8') as output:\n",
    "    i = 0\n",
    "    for line in content:\n",
    "        try:\n",
    "            line = ', '.join(line)\n",
    "            line = line.replace(', ', ' ')\n",
    "            line = line.replace(' s ', '')\n",
    "            output.write(line)\n",
    "        except Exception as e:\n",
    "            print ('Exception error: ' + str(e))\n",
    "        i = i + 1\n",
    "        if (i % 10000 == 0):\n",
    "            print('ngrams calculated for ' + str(i) + ' articles')\n",
    "    output.close()\n",
    "    print('Calculated ngrams for ' + str(i) + ' articles;')\n",
    "    print('Processing complete!')   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This routine takes about 14 minutes to run on my laptop, with the settings as shown. Note in the routine where we set the delimiter to be the underscore character; this is how we know the bigram.\n",
    "\n",
    "Once this routine finishes, we can take its output and re-use it as input to a subsequent run. Now, we will be producing trigrams where we can match to existing bigrams. Generally, we set our thresholds and minimum counts higher. In our case, the new settings are <code>min_count=8, threshold=50</code> The trigram analysis takes 19 min to run.\n",
    "\n",
    "We have now completed our preprocessing steps the for embedding models we introduce in the next installment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Documentation\n",
    "Here are many supplementary resources useful to the environment and natural language processing capabilities introduced in this installment. \n",
    "\n",
    "- [Transformers: State-of-the-art Natural Language Processing](http://arxiv.org/abs/1910.03771)\n",
    "- [Corpus from a Wikipedia Dump](https://radimrehurek.com/gensim/corpora/wikicorpus.html)\n",
    "- [Building a Wikipedia Text Corpus for Natural Language Processing](https://www.kdnuggets.com/2017/11/building-wikipedia-text-corpus-nlp.html).\n",
    "\n",
    "#### PyTorch and pandas\n",
    "- [Convert Pandas Dataframe to PyTorch Tensor](https://stackoverflow.com/questions/50307707/convert-pandas-dataframe-to-pytorch-tensor) - pandas &rarr; numpy &rarr; torch\n",
    "- [PyTorch Dataset: Reading Data Using Pandas vs. NumPy](https://jamesmccaffrey.wordpress.com/2020/08/31/pytorch-dataset-reading-data-using-pandas-vs-numpy/)\n",
    "- See [**CWPK #56**](https://www.mkbergman.com/2404/cwpk-56-graph-visualization-and-extraction/) for pandas/networkx reads and imports.\n",
    "\n",
    "#### PyTorch Resources and Tutorials\n",
    "- [The Most Complete Guide to PyTorch for Data Scientists](https://www.kdnuggets.com/2020/09/most-complete-guide-pytorch-data-scientists.html) provides the basics of tensors and tensor operations and then provides a high-level overview of the PyTorch capabilities\n",
    "- [Awesome-Pytorch-list](https://awesomeopensource.com/project/bharathgs/Awesome-pytorch-list)  provides multiple resource categories, with 230 in related packages alone\n",
    "- The official [PyTorch documentation](https://pytorch.org/docs/stable/index.html)\n",
    "- [Building Efficient Custom Datasets in PyTorch](https://towardsdatascience.com/building-efficient-custom-datasets-in-pytorch-2563b946fd9f)\n",
    "- [Getting Started with PyTorch: A Deep Learning Tutorial](https://adatis.co.uk/getting-started-with-pytorch-a-deep-learning-tutorial/)\n",
    "- [Incredible PyTorch](https://www.ritchieng.com/the-incredible-pytorch/) is a curated list of tutorials, papers, projects, communities and more relating to PyTorch\n",
    "- [Learning PyTorch with Examples](https://pytorch.org/tutorials/beginner/pytorch_with_examples.html).\n",
    "\n",
    "#### spaCy and gensim\n",
    "- [Natural Language in Python using spaCy: An Introduction](https://blog.dominodatalab.com/natural-language-in-python-using-spacy/)\n",
    "- [Gensim Tutorial – A Complete Beginners Guide](https://www.machinelearningplus.com/nlp/gensim-tutorial/) is excellent and comprehensive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <div style=\"background-color:#ffecec; border:1px dotted #f5aca6; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site. The <em>cowpoke</em> Python <a href=\"https://github.com/Cognonto/cowpoke\">code listing covering the series</a> is also available from GitHub.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}