cwpk-61-nlp-machine-learning-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#61: NLP, Machine Learning and Analysis\n",
    "=======================================\n",
    "\n",
    "A Wealth of Applications Sets the Stage for Pay Offs from KBpedia\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "With this installment of the [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) series we move into **Part VI** of seven parts, a part with the bulk of the analytical and machine learning (that is, \"data science\") discussion, and the last part where significant code is developed and documented. Because of the complexity of these installments, we will also be reducing the number released per week for the next month or so. We also will not be able to post fully operational electronic notebooks to [MyBinder](https://mybinder.org/v2/gh/Cognonto/CWPK/master) since the supporting libraries strain the limits of that service. At the conclusion of this part, which itself has 11 installments, we have four installments to wrap up the series and provide a consistent roadmap to the entire project.\n",
    "\n",
    "[Knowledge graphs](https://en.wikipedia.org/wiki/Knowledge_graph) are unique information artifacts, and [KBpedia](https://kbpedia.org/) is further unique in terms of its consistent and logical construction as well as its incorporation of significant text content via Wikipedia pages. These characteristics provide unique value for KBpedia, but it is also a combination not duplicated anywhere else in the data science ecosystem. One of the objectives, therefore, of this part of our **CWPK** series is the creation of some baseline knowledge representations useful to data science aims that capture these unique characteristics.\n",
    "\n",
    "KBpedia's (or any knowledge graph constructed in a similar manner) combination of characteristics make it a powerful resource in three areas of data science and machine learning. First, the nearly universal scope and degree of topic coverage with about 56,000 concepts, logically organized into typologies with a high degree of disjointedness, means that accurate 'slices' or training sets may be extracted from KBpedia nearly instantaneously. Labeled training sets are one of the most time consuming and expensive activities in doing [supervised machine learning](https://en.wikipedia.org/wiki/Supervised_learning). We can extract these nearly for free from KBpedia. Further, with its links to tens of millions of entities in its mapped knowledge bases such as Wikidata, literally tens of thousands of conceptual entities in KBpedia can be the retrieval points to nucleate training sets for fine-grained [entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition). \n",
    "\n",
    "Second, 80% of KBpedia's concepts are mapped to Wikipedia articles. While many Wikipedia-based word embedding models exist, the ones in KBpedia are logically categorized and have rough equivalence in terms of scope and prominence, hopefully providing cleaner topic 'signals'. To probe these assertions, we will create a unique KBpedia-based word embedding corpus that also leverages labels for items of structural importance, such as typology membership. We will use this corpus in many of our tests and as a general focus in our training sets.\n",
    "\n",
    "And, third, perhaps the most important area, knowledge graphs offer unique structures and challenges for machine learning, especially innovations in geometric, heterogeneous methods for [deep learning](https://en.wikipedia.org/wiki/Deep_learning). The first generation of deep machine learning was designed for grid-patterned data and matrices through approaches such as deep [neural networks](https://en.wikipedia.org/wiki/Artificial_neural_network), convolutional neural networks ([CNN](https://en.wikipedia.org/wiki/Convolutional_neural_network)\n",
    "), or recurrent neural networks ([RNN](https://en.wikipedia.org/wiki/Recurrent_neural_network)). The 'deep' appelation comes from having multiple calculated, intermediate layers of transformations between the grid inputs and outputs for the model. Graphs, on the other hand, are heterogeneous between nodes and edges. They may be directed (subsumptive) in nature. And, for knowledge graphs, they have much labeling and annotation, including varying degrees of attribute completeness. Language embedding, itself often a product of deep learning, enables the efficient incorporation of text. It is only in the past five years that concerted attention has been devoted to better capturing this feature richness for knowledge graphs.\n",
    "\n",
    "The eleven installments in this part will look in more depth at networks and graphs, focus on how to create training sets and embeddings for the learners, discuss some natural language packages and uses, and then look in depth at 'standard' machine learners and deep learners. We will install the first generation of deep graph learners and then explore some on the cutting edge. We will test many use cases, but will also try to invoke classifiers across this spectrum so that we can draw some general conclusions.\n",
    "\n",
    "The material below introduces and tees up these topics. We describe leading Python packages for data science, and how we have architected our own approach, We have picked a particular Python machine learning framework, [PyTorch](https://en.wikipedia.org/wiki/PyTorch), to which we will then tie four different NLP and deep learning libraries. We devote two installments each to these four libraries. The use cases we document across these installments are in addition to the existing ones we have in [Clojure posted online](https://kbpedia.org/use-cases/).\n",
    "\n",
    "So, we think we have an interesting suite of benefits to cover in this part, some arising from being based on KBpedia and some arising from the nature of knowledge graphs. On the other hand, due to the relative immaturity of the field, we are still actively learning and innovating around the juncture of AI and knowledge graphs. Thus, one of the reasons we emphasize Python 'ecosystems' and 'frameworks' in this part is to be better prepared to incorporate those innovations and learnings to come."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Background\n",
    "\n",
    "One of the first prototypes of machine learning comes from the statistician Ronald Fisher in the 1930s regarding how to [classify Iris species](https://en.wikipedia.org/wiki/Iris_flower_data_set) based on the attributes of their flowers. It was a multivariate data example using the method we today call [linear discriminant analysis](https://en.wikipedia.org/wiki/Linear_discriminant_analysis). This classic example is still taught. But many dozens of new algorithms and combined approaches have joined the machine learning field since then.\n",
    "\n",
    "*Figure 1* below is one way to characterize the field, with ML standing for machine learning and DL for deep learning, with this one oriented to sub-fields in which some Python package already exists:\n",
    "\n",
    "<div style=\"margin: 10px auto; display: table;\">\n",
    "\n",
    "<img src=\"files/machine-learning-landscape.png\" title=\"Machine Learning Landscape\" alt=\"Machine Learning Landscape\" width=\"800\"/>\n",
    "\n",
    "</div>\n",
    "\n",
    "<div style=\"margin: 10px auto; display: table; font-style: italic;\">\n",
    "\n",
    "Figure 1: Machine Learning Landscape (from S. Chen, <a href=\"https://towardsai.net/machine-learning-algorithms\">\"Machine Learning Algorithms For Beginners with Code Examples in Python\"</a>, June 2020) \n",
    "\n",
    "</div>\n",
    "\n",
    "There are many possible diagrams that one might prepare to show the machine learning landscape, including ones with a larger emphasis on text and knowledge graphs. Most all schematics of the field show a basic split between supervised learning and [unsupervised learning](https://en.wikipedia.org/wiki/Unsupervised_learning), (sometimes with [reinforcement learning](https://en.wikipedia.org/wiki/Reinforcement_learning) as another main branch), with the main difference being that supervised approaches iterate to achieve statistical fit with pre-determined labels, whereas unsupervised is unlabeled. Accurate labeling can be costly and time consuming. Note that the idea of 'classification' is a supervised one, 'clustering' a notion of unsupervised.\n",
    "\n",
    "We will include a 'standard' machine learning library in our proposed toolkit, the selection of which I discuss below. However, the most evaluation time I spent in researching these installments was directed to the idea of knowledge representation and embeddings applicable to graphs. Graphs pose a number of differences and challenges to standard machine learning. They have only been a recent (5 yr) focus in machine learning, which is also rapidly changing over time.\n",
    "\n",
    "All machine learners need to operate on their feature spaces in numerical representations. Text is a tricky form because language is difficult and complex, and how to represent the tokens within our language usable by a computer needs to consider, what? Parts-of-speech, the word itself, sentence construction, semantic meaning, context, adjacency, entity recognition or characterization? These may all figure into how one might represent text. Machine learning has brought us unsupervised methods for converting words to sentences to documents and, now, graphs, to a reduced, numeric representation known as \"embeddings.\" The embedding method may capture one or more of these textual or structural aspects. \n",
    "\n",
    "Much of the first interest in machine learning based on graphs was driven by these interests in embeddings for language text. Standard machine classifiers with deep learning using neural networks have given us [word2vec](https://en.wikipedia.org/wiki/Word2vec), and more recently [BERT](https://en.wikipedia.org/wiki/BERT_(language_model)) and its dozens of variants have reinforced the usefulness of deep learning to create pre-trained text representations.\n",
    "\n",
    "Indeed, embeddings **do** figure prominently in knowledge graph representation, but only as one among many useful features. Knowledge graphs with hierarchical (subsumption) relationships, as might be found in any taxonomy, become directed. Knowledge graphs are asymmetrical, and often multi-typed and sometimes multi-modal. There is heterogeneity among nodes and links or edges. Not all knowledge graphs are created equal and some of these aspects may not apply. Whether there is an accompanying richness of text description that accompanies the node or edges is another wrinkle. None of the early CNN or RNN or simple neural net approaches match well with these structures. \n",
    "\n",
    "The general category that appears to have emerged for this scope is [geometric deep learning](http://geometricdeeplearning.com/), which applies to all forms of graphs and manifolds. There are other nuances in this area, for example whether a static representation is the basis for analysis or one that is dynamic, essentially allowing learning parameters to be changed as the deep learning progresses through its layers. But GDL has the theoretical potential to address and incorporate all of the wrinkles associated with heterogeneous knowledge graphs.\n",
    "\n",
    "So, this discussion helps define our desired scope. We want to be able to embrace Python packages that range from simple statistics to simple machine learning, throwing in natural language processing and creating embedding representations, that can then range all the way through deep learning to the cutting-edge aspects of geometric or graph deep learning. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Leading Python Data Science Packages\n",
    "\n",
    "This background provides the necessary context for our investigations of Python packages, frameworks, or libraries that may fulfill the data science objectives of this part. Our new components often build upon and need to play nice with some of the other requisite packages introduced in earlier installments, including [pandas](https://en.wikipedia.org/wiki/Pandas_(software)) ([**CWPK #55**](https://www.mkbergman.com/2402/cwpk-55-charting/)), [NetworkX](https://en.wikipedia.org/wiki/NetworkX) ([**CWPK #56**](https://www.mkbergman.com/2402/cwpk-55-charting/)), and [PyViz](https://pyviz.org/) ([**CWPK #55**](https://www.mkbergman.com/2402/cwpk-55-charting/)). [NumPy](https://en.wikipedia.org/wiki/NumPy) has been installed, but not discussed.\n",
    "\n",
    "We want to focus our evaluation of Python options in these areas:\n",
    "\n",
    "- Natural Language Processing, including embeddings\n",
    "- 'Standard' Machine Learning\n",
    "- Deep Learning and Abstraction Frameworks, and\n",
    "- Knowledge Graph Representation Learning.\n",
    "\n",
    "The latter area may help us tie these various components together.\n",
    "\n",
    "#### Natural Language Processing\n",
    "It is not fair to say that natural language processing has become a 'commodity' in the data science space, but it is also true there is a wealth of capable, complete packages within Python. There are standard NLP requirements like text cleaning, tokenization, parts-of-speech identification, parsing, lemmatization, phrase identification, and so forth. We want these general text processing capabilities since they are often building blocks and sometimes needed in their own right. We also would like to add to this baseline such considerations as interoperability, creating embeddings, or other special functions.\n",
    "\n",
    "The two leading NLP packages in Python appear to be:\n",
    "\n",
    "- [NLTK](http://www.nltk.org/index.html) - the natural language toolkit that is proven and has been a leader for twenty years\n",
    "- [spaCy](https://spacy.io/) - a newer, but very impressive package oriented more to tasks, not function calls.\n",
    "\n",
    "Other leading packages, with varying NLP scope, include:\n",
    "\n",
    "- [flair](https://github.com/flairNLP/flair) - a very simple framework for state-of-the-art NLP that is based on PyTorch and works based on context\n",
    "- [gensim](https://radimrehurek.com/gensim/) - a semantic and topic modeling library; not general purpose, but with valuable capabilities\n",
    "- [OpenNMT-py](https://opennmt.net/OpenNMT-py/) - an open source library for neural machine translation and neural sequence learning; provided for both the PyTorch and TensorFlow environments\n",
    "- [Polyglot](https://pypi.org/project/polyglot/) - a natural language pipeline that supports massive multilingual applications\n",
    "- [Stanza](https://stanfordnlp.github.io/stanza/) - a neural network pipeline for text analytics; beyond standard functions, has multi-word token (MWT) expansion, morphological features, and dependency parsing; uses the Java CoreNLP from Stanford\n",
    "- [TextBlob](http://textblob.readthedocs.io/en/dev/) - a simplified text processor, which is an extension to NLTK.\n",
    "\n",
    "Another key area is [language embedding](https://machinelearningmastery.com/what-are-word-embeddings/). Language embeddings are means to translate language into a numerical representation for use in downstream analysis, with great variety in what aspects of language are captured and how to craft them. The simplest and still widely-used representation is [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) (term frequency–inverse document frequency) statistical measure. A common variant after that was the [vector space model](https://en.wikipedia.org/wiki/Vector_space_model). We also have latent (unsupervised) models such as [LDA](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation). A more easily calculated option is [explicit semantic analysis](https://en.wikipedia.org/wiki/Explicit_semantic_analysis) (ESA). At the word level, two of the prominent options are [word2vec](https://en.wikipedia.org/wiki/Word2vec) and [gloVe](https://en.wikipedia.org/wiki/GloVe_(machine_learning)), which are used directly in spaCy. These have arisen from deep learning models. We also have similar approaches to represent [topics](https://github.com/askerlee/topicvec) (topicvec), [sentences](https://github.com/stanleyfok/sentence2vec) (sentence2vec), [categories and paragraphs](https://github.com/rakuten-nlp/category2vec) (Category2Vec), [documents](https://radimrehurek.com/gensim/models/doc2vec.html) (doc2vec), [node2vec](https://en.wikipedia.org/wiki/Node2vec) or entire languages ([BERT](https://en.wikipedia.org/wiki/BERT_(language_model)) and variants and [GPT-3](https://en.wikipedia.org/wiki/GPT-3) and related methods). In all of these cases, the embedding consists of reducing the dimensionality of the input text, which is then represented in numeric form.\n",
    "\n",
    "There are internal methods for creating embeddings in multiple machine learning libraries. Some packages are more dedicated, such as [fastText](https://en.wikipedia.org/wiki/FastText), which is a library for learning of word embeddings and text classification created by Facebook's AI Research (FAIR) lab. Another option is [TextBrewer](https://textbrewer.readthedocs.io/en/latest/), which is an open-source knowledge distillation toolkit based on PyTorch and which uses (among others) BERT to provide text classification, reading comprehension, NER or sequence labeling.\n",
    "\n",
    "Closely related to how we represent text are corpora and datasets that may be used either for reference or training purposes. These need to be assembled and tested as well as software packages. The availability of corpora to different packages is a useful evaluation criterion. But, the picking of specific corpora depends on the ultimate Python packages  used and the task at hand. We will return to this topic in **CWPK #63**.\n",
    "\n",
    "#### 'Standard' Machine Learning\n",
    "Of course, nearly all of the Python packages mentioned in this **Part VI** have some relation to machine learning in one form or another. I call out this category separately because, like for NLP, I think it makes sense to have a general machine learning library not devoted to deep learning but providing a repository of classic learning methods.\n",
    "\n",
    "There really is no general option that compares with [scikit-learn](https://en.wikipedia.org/wiki/Scikit-learn). It features various classification, regression, and clustering algorithms, including support vector machines, random forests, gradient boosting, k-means and DBSCAN data clustering, and is designed to interoperate with NumPy and [SciPy](https://en.wikipedia.org/wiki/SciPy). The [project](https://github.com/scikit-learn/scikit-learn) is extremely active with good [documentation](https://scikit-learn.org/stable/) and examples.\n",
    "\n",
    "We'll return to scikit-learn below.\n",
    "\n",
    "#### Deep Learning and Abstraction Frameworks\n",
    "Deep learning is characterized by many options, methods and philosophies, all in a fast-changing area of knowledge. New methods need to be compared on numerous grounds from feature and training set selection to testing, parameter tuning, and performance comparisons. These realities have put a premium on libraries and frameworks that wrap methods in repeatable interfaces and provide abstract functions for setting up and managing various deep (and other) learning algorithms.\n",
    "\n",
    "The space of deep learning thus embraces many individual methods and forms, often expressed through a governing ecosystem of other tools and packages. These demands lead to a confusing and overlapping and non-intersecting space of Python options that are hard to describe and [comparatively evaluate](https://en.wikipedia.org/wiki/Comparison_of_deep-learning_software). Here are some of the libraries and packages that fit within the deep and machine learning space, including abstraction frameworks:\n",
    "\n",
    "- [Chainer](https://en.wikipedia.org/wiki/Chainer) is an open source deep learning framework written purely in Python on top of NumPy and CuPy Python libraries\n",
    "- Microsoft Cognitive Toolkit ([CNTK](https://docs.microsoft.com/en-us/cognitive-toolkit/)) is an open-source toolkit for commercial-grade distributed deep learning; however, it has seen its last [main release](https://github.com/Microsoft/CNTK) in favor of the interoperable approach, [ONNX](https://en.wikipedia.org/wiki/Open_Neural_Network_Exchange) (see below)\n",
    "- [Keras](https://github.com/keras-team/keras) is an open-source library that provides a Python interface for artificial neural networks. [Keras](https://keras.io/) now acts as an interface for the TensorFlow library and is built on top of Theano; it has a high-level library for working with datasets \n",
    "- [PlaidML](https://en.wikipedia.org/wiki/PlaidML) is a portable tensor compiler; it runs as a component under Keras\n",
    "- [PyTorch](https://en.wikipedia.org/wiki/PyTorch) is an open source machine learning library based on the Torch library with a very rich ecosystem of interfacing or contributing projects\n",
    "- [TensorFlow](https://en.wikipedia.org/wiki/TensorFlow) is a well-known open source machine learning library developed by Google \n",
    "- [Theano](https://en.wikipedia.org/wiki/Theano_(software)) is a Python library and optimizing compiler for manipulating and evaluating mathematical expressions, especially matrix-valued ones;  it is tightly integrated with NumPy, and uses it at the lowest level.\n",
    "\n",
    "Keras is increasingly aligning with TensorFlow and some, like Chainer and CNTK, are being deprecated in favor of the two leading gorillas, PyTorch and TensorFlow. One approach to improve interoperability is the Open Neural Network Exchange (ONNX) with the repository available on [GitHub](https://github.com/onnx/onnx). There are existing converters to ONNX for Keras, TensorFlow, PyTorch and scikit-learn. \n",
    "\n",
    "A key development from deep learning of the past three years has been the usefulness of [Transformers](https://en.wikipedia.org/wiki/Transformer_(machine_learning_model), a technique that marries encoders and decoders converging on the same representation. The technique is particularly helpful to sequential data and NLP, with state-of-the-art performance to date for:\n",
    "- next-sentence prediction\n",
    "- question answering\n",
    "- reading comprehension\n",
    "- sentiment analysis, and\n",
    "- paraphrasing.\n",
    "\n",
    "Both BERT and GPT are pre-trained products that utilize this method. Both TensorFlow and PyTorch contain Transformer capabilities. \n",
    "\n",
    "#### Knowledge Graph Representation Learning\n",
    "As noted, most of my research for this **Part VI** has resided in the area of a subset of deep graph learning applicable to knowledge graphs. The leading deep learning libraries do not, in general, provide support for this area of representational learning, sometimes called knowledge representation learning ([KRL](https://deepai.org/publication/knowledge-representation-learning-a-quantitative-review)) or knowledge graph embedding ([KGE](https://mnick.github.io/project/knowledge-graph-embeddings/)). Within this rather limited scope, most options also seem oriented to [link prediction](https://arxiv.org/pdf/2002.00819v2.pdf) and knowledge graph completion ([KGC](https://github.com/topics/knowledge-graph-completion?l=python)), rather than the heterogeneous aspects with text and OWL2 orientation characteristic of KBpedia.\n",
    "\n",
    "Various capabilities desired or tested for knowledge graph representational learning include:\n",
    "- low-dimensional vectors (embeddings) with semantic meaning\n",
    "- knowledge graph completion (KGC)\n",
    "- triple classification\n",
    "- entity recognition\n",
    "- entity disambiguation (linking)\n",
    "- relation extraction\n",
    "- recommendation systems\n",
    "- question answering, and\n",
    "- common sense reasoning.\n",
    "\n",
    "Unsupervised graph relational learning is used for:\n",
    "- link prediction\n",
    "- graph reconstruction\n",
    "- visualization, or\n",
    "- clustering.\n",
    "\n",
    "Supervised GRL is used for:\n",
    "  - node classification, and\n",
    "  - graph classification (predict node labels).\n",
    "  \n",
    "This kind of learning is a subset of areas called [geometric deep learning](https://blog.paperspace.com/geometric-deep-learning-framework-comparison/), [deep graphs](https://www.dgl.ai/), or [graph representation (or representation) learning](https://www.cambridge.org/core/services/aop-cambridge-core/content/view/23B9870F91F7E6DA14784959A9BC9E7A/S204877032000013Xa.pdf/div-class-title-graph-representation-learning-a-survey-div.pdf). We thus see this rough hierachy:\n",
    "\n",
    "<span style=\"margin-left: 40px;\">machine learning &rarr; deep learning &rarr; geometric deep learning &rarr; graph (R) learning &rarr; KG learning</span>\n",
    "\n",
    "In terms of specific packages or libraries, there is a wealth of options in this new field:\n",
    "\n",
    "- [AmpliGraph](https://github.com/Accenture/AmpliGraph) is a suite of neural machine learning models for relational learning using supervised learning on knowledge graphs\n",
    "- [DGL-KE](https://github.com/awslabs/dgl-ke) is a high performance, easy-to-use, and scalable package for learning large-scale knowledge graph embeddings based on the [Deep Graph Library](https://www.dgl.ai/), which is a library for GNN in PyTorch. Can run DGL-KE on CPU machine, GPU machine, as well as clusters with a set of popular models, including TransE, TransR, RESCAL, DistMult, ComplEx, and RotatE\n",
    "- [Graph Nets](https://github.com/deepmind/graph_nets) is DeepMind's library for building graph networks in TensorFlow\n",
    "- [KGCN](https://github.com/graknlabs/kglib) is the novel knowledge graph convolutional network model that is part of KGLIB; it requires [GRAKN](https://en.wikipedia.org/wiki/GRAKN.AI)\n",
    "- [OpenKE](https://github.com/thunlp/OpenKE) is an efficient implementation based on PyTorch for knowledge representation learning\n",
    "- [OWL2Vec*](https://arxiv.org/pdf/2009.14654.pdf) represents each OWL named entity (class, instance or property) by a vector, which then can feed downstream tasks; see [GitHub](https://github.com/KRR-Oxford/OWL2Vec-Star)\n",
    "- [PyKEEN](https://arxiv.org/abs/2007.14175v2) is a Python library for training and evaluating knowledge graph embeddings; see [GitHub](https://github.com/pykeen/pykeen)\n",
    "- [Pykg2vec: A Python Library for Knowledge Graph Embedding](https://www.researchgate.net/publication/333716909_Pykg2vec_A_Python_Library_for_Knowledge_Graph_Embedding) is a Python library for knowledge graph embedding and representation learning; see [GitHub](https://github.com/Sujit-O/pykg2vec)\n",
    "- [PyTorch Geometric](https://github.com/rusty1s/pytorch_geometric) (PyG) is a geometric deep learning extension library for PyTorch with excellent [documentation](https://pytorch-geometric.readthedocs.io/en/latest/) and an emphasis of providing wrappers to state-of-art models\n",
    "- [RDF2Vec](https://github.com/IBCNServices/pyRDF2Vec) is an unsupervised technique that builds further on Word2Vec; [RDF2Vec Light](https://arxiv.org/abs/2009.07659) is a lightweight approach to KG embeddings\n",
    "- [scikit-kge](https://github.com/mnick/scikit-kge) is a Python library to compute embeddings of knowledge graphs that ties directly into scikit-learn; an umbrella for the RESCAL, HolE, TransE, and ER-MLP algorithms; it has not been updated in five years \n",
    "- [StellarGraph](https://www.stellargraph.io/library) provides multiple variants of neural networks for both homogeneous and heterogeneous graphs, and relies on the TensorFlow, Keras, NetworkX, and scikit-learn libraries\n",
    "- [TorchKGE](https://arxiv.org/pdf/2009.02963v1.pdf) provides knowledge graph embedding in PyTorch; it [reportedly](https://github.com/torchkge-team/torchkge) is faster than AmpiGraph and OpenKE.\n",
    "\n",
    "One graph learning framework that caught my eye is [KarateClub](xxx), an unsupervised machine learning extension library for NetworkX. I like the approach they are taking, but their library can not yet handle directed graphs. I will be checking periodically on their announced intention to extend this framework to directed graphs in the near future.\n",
    "\n",
    "Lastly, more broadly, there is the recently announced [KGTK](https://arxiv.org/pdf/2006.00088.pdf), which is a generalized toolkit with broader capabilities for large scale knowledge graph manipulation and analysis. KGTK also puts forward a [standard KG file format](https://github.com/usc-isi-i2/kgtk/), among other [tools](https://kgtk.readthedocs.io/en/latest/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Generalized Python Data Science Architecture\n",
    "\n",
    "With what we already have in hand, plus the libraries and packages described above, we have a pretty good inventory of candidates to choose from in proceeding with our next installments. Like our investigations around graphics and visualization (see [**CWPK #55**](https://www.mkbergman.com/2402/cwpk-55-charting/)), the broad areas of data science, machine learning, and deep learning have been evolving to one of comprehensive ecosystems. *Figure 2* below presents a representation of the Python components that make sense for the machine learning and application environment. As noted, our local Windows machines lack separate GPUs (graphical processing units), so the hardware is based on a standard CPU (which has an integrated GPU that can not be separately targeted). We have already introduced and discusses some of the major Python packages and libraries, including [pandas](https://en.wikipedia.org/wiki/Pandas_(software)), [NetworkX](https://en.wikipedia.org/wiki/NetworkX), and [PyViz](https://pyviz.org/). Here is that representative data science architecture:\n",
    "\n",
    "<div style=\"margin: 10px auto; display: table;\">\n",
    "\n",
    "<img src=\"files/representative-python-components.png\" title=\"Representative Python Components\" alt=\"Representative Python Components\" width=\"800\"/>\n",
    "\n",
    "</div>\n",
    "\n",
    "<div style=\"margin: 10px auto; display: table; font-style: italic;\">\n",
    "\n",
    "Figure 2: Representative Python Components (from S. Raschka et al. <a href=\"https://arxiv.org/pdf/2002.04803.pdf\">\"Machine Learning in Python: Main Developments and technology Trends in Data Science, Machine Learning, and Artificial Intelligence\"</a>, March 31, 2020) \n",
    "\n",
    "</div>\n",
    "\n",
    "The defining architectural question for this **Part VI** is what general deep and machine learning framework we want (if any). I think using a framework makes sense over scripting together individual packages, though for some tests that still might be necessary. If I was to adopt a framework, I would also want one that has a broad set of tools in its ecosystem and common and simpler ways to define projects and manage the overall pipelines from data to results. As noted, the two major candidates appear to be [TensorFlow](https://en.wikipedia.org/wiki/TensorFlow) and [PyTorch](https://en.wikipedia.org/wiki/PyTorch).\n",
    "\n",
    "TensorFlow has been around the longest, has, today, the strongest ecosystem, and reportedly is better for commercial deployments. Google, besides being the sponsor, uses TensorFlow in most of its ML projects and has shown a commitment to compete with the upstart PyTorch by significantly re-designing and enhancing TensorFlow 2.0.\n",
    "\n",
    "On the other hand, I very much like the more 'application' orientation of PyTorch. Innovation has been fast and market share has been rising. The consensus from online reviews is that PyTorch, in comparison to TensorFlow:\n",
    "\n",
    "- runs dramatically faster on both CPU and GPU architectures\n",
    "- is easier to learn\n",
    "- produces faster graphs, and\n",
    "- Is more amenable to third-party tools.\n",
    "\n",
    "Though some of the intriguing packages for TensorFlow are not apparently available for PyTorch, including Graph Nets, Keras, Plaid ML, and StellarGraph, PyTorch does have these other packages not yet mentioned that look potentially valuable down the road:\n",
    "\n",
    "- [Captum](https://captum.ai/) - a unified and generic model interpretability library\n",
    "- [Catalyst](https://github.com/catalyst-team/catalyst) - a framework for deep learning R&D\n",
    "- [DGL](https://www.dgl.ai/) - the Deep Graph Libary needed for DGL-KE discussed below\n",
    "- [fastai](https://docs.fast.ai/) - simplifies training fast and accurate neural nets using modern best practices\n",
    "- [flair](https://github.com/flairNLP/flair) - a simple framework for state-of-the-art NLP that may complement or supplement spaCy\n",
    "- [PyTorch Geometric](https://github.com/rusty1s/pytorch_geometric) - a geometric deep learning extension library, also discussed below\n",
    "- [PyTorch-NLP](https://github.com/PetrochukM/PyTorch-NLP) - a library of basic utilities for PyTorch NLP that may supplement or replace spaCy or flair, and\n",
    "- [skorch](https://github.com/skorch-dev/skorch) - a scikit-learn compatible neural network library that wraps PyTorch.\n",
    "\n",
    "One disappointment is that neither of these two leading packages directly ingest [RDFLib](https://rdflib.readthedocs.io/en/stable/) graph files, though with PyTorch and DGL you can import or export a NetworkX graph directly. pandas is also a possible data exchange format.\n",
    "\n",
    "Consideration of all of these points has led us to select PyTorch as the initial data science framework. It is good to know, however, that a fairly comparable alternative also exists with TensorFlow and Keras.\n",
    "\n",
    "Finally, with respect to *Figure 2* above, we have no plans at present to use the [Dask](https://dask.org/) package for parallelizing analytic calculations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Four Additional Key Packages\n",
    "With the PyTorch decision made, at least for the present, we are now clear to deal with specific additional packages and libraries. I highlight four of these in this section. Each of these four is the focus of two separate installments as we work to complete this **Part VI**. One of these four is in natural language processing (spaCy), one in general machine learning (scikit-learn), and two in deep learning with an emphasis on graphs (DGL and DGL-KE, and PyG). These choices again tend to reinforce the idea of evaluating whole ecosystems, as opposed to single packages. Note, of course, that more specifics on these four packages will be presented in the forthcoming installments.\n",
    "\n",
    "#### spaCy\n",
    "I find [spaCy](https://spacy.io/) to be very impressive, with many potentially useful extensions or compatible packages including [sense2vec](https://spacy.io/universe/project/sense2vec), [spacy-stanza](https://spacy.io/universe/project/spacy-stanza), [spacy-wordnet](https://spacy.io/universe/project/spacy-wordnet), [torchtext](https://spacy.io/universe/project/torchtext), and [gensim](https://radimrehurek.com/gensim/). \n",
    "\n",
    "The major competitor is NLTK. The reputation of the NLTK package is stellar and it has proven itself for decades. It is a more disaggregate approach often favored by scholars and researchers to enable users to build complex NLP functionality. It is therefore harder to use and configure, and is also less performant. The real differentiator, however, is the more object or application orientation of spaCy. \n",
    "\n",
    "Though NLTK appears to have good NLP tools for processing data pipelines using text, most of these functions appear to be in spaCy and there are also the flair and PyTorch-NLP packages available in the PyTorch environment if needed. gensim looks to be a very useful enhancement to the environment because of the advanced text evaluation modes it offers, including sentiment. Not all of these will be tested during this **CWPK** series, but it will be good to have these general capabilities resident in *cowpoke*. \n",
    "\n",
    "#### scikit-learn\n",
    "We earlier signaled our intent to embrace [scikit-learn](https://github.com/scikit-learn/scikit-learn), principally to provide basic machine learning support. scikit-learn provides a unified API to these basic tasks, including crafting pipelines and meta-functions to integrate the data flow. scikit-learn works on any numeric data stored as NumPy arrays or SciPy sparse matrices. Other types that are convertible to numeric arrays such as pandas DataFrames are also acceptable.\n",
    "\n",
    "Some of the general ML methods, and there are about [40 supervised ones](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning) in the package, may be useful and applicable to specific circumstances include:\n",
    "\n",
    "- dimensionality reduction\n",
    "- model testing\n",
    "- preprocessing\n",
    "- scoring methods, and\n",
    "- principal component analysis ([PCA](https://en.wikipedia.org/wiki/Principal_component_analysis)).\n",
    "\n",
    "A real test of this package will be ease of creating (and then being able to easily modify) data processing and analysis pipelines. Another test will be ingesting, using, and exporting data formats useful to the KBpedia knowledge graph. We know that scikit-learn doesn't talk directly to NetworkX, though there may be [recipes for the transfer](https://stackoverflow.com/questions/55599825/how-to-use-networkx-graphs-as-input-for-sklearn); graphs are represented in scikit-learn as connectivity matrices. pandas can interface via common formats including CSV, Excel, JSON and SQL, or, with some processing, DataFrames. scikit-learn supports data formats from NumPy and SciPy, and it supports a <code>datasets.load_files</code> format that may be suitable for transferring many and longer text fields. One option that is intriguing is how to leverage the CSV flat-file orientation of our KG build and extract routines in *cowpoke* for data transfer and transformation.\n",
    "\n",
    "I also want to keep an eye on the possible use of [skorch](https://github.com/skorch-dev/skorch) to better integrate with the overall PyTorch environment, or to add perhaps needed and missing functionality or ease of development. There is much to explore with these various packages and environments.\n",
    "\n",
    "#### DGL-KE\n",
    "For our basic, 'vanilla', deep graph analysis package we have chosen the eponymous [Deep Graph Library](https://www.dgl.ai/) for basic graph neural network operations, which may run on CPU or GPU machines or clusters. The better interface relevant to KBpedia is through [DGL-KE](https://github.com/awslabs/dgl-ke), a high performance, reportedly easy-to-use, and scalable package for learning large-scale knowledge graph embeddings that extends DGL. DGL-KE also comes configured with the popular models of TransE, TransR, RESCAL, DistMult, ComplEx, and RotatE.\n",
    "\n",
    "#### PyTorch Geometric\n",
    "[PyTorch Geometric](https://github.com/rusty1s/pytorch_geometric) is closely tied to PyTorch, and most impressively has uniform wrappers to about 40 state-of-art graph neural net methods. The idea of \"message passing\" in the approach means that heterogeneous features such as structure and text may be combined and made dynamic in their interactions with one another. Many of these intrigued me on paper, and now it will be exciting to test and have the capability to inspect these new methods as they arise. [DeepSNAP](https://github.com/snap-stanford/deepsnap) may provide a direct bridge between NetworkX and PyTorch Geometric."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Possible Future Extensions\n",
    "During the research on this **Part VI** I encountered a few leads that are either not ready for prime time or are off scope to the present **CWPK** series. A potentially powerful, but experimental approach that makes sense is to use SPARQL as the request-and-retrieval mechanism against the graph to feed the machine learners. [RDFFrames](https://ui.adsabs.harvard.edu/link_gateway/2020arXiv200203614M/arxiv:2002.03614) provides an imperative Python API that gets internally translated to SPARQL, and it is integrated with the PyData machine learning software stack; see [GitHub](https://github.com/qcri/rdfframes). Some methods above also use SPARQL. One of the benefits of a SPARQL approach, besides its sheer query and inferencing power, is the ability to keep the knowledge graph intact without data transform pipelines. It is quite available to serve up results in very flexible formats. The relative immaturity of the approach and performance considerations may be difficult challenges to overcome.\n",
    "\n",
    "I earlier mentioned [KarateClub](https://github.com/benedekrozemberczki/karateclub), a Python framework combining about 40 state-of-the-art unsupervised graph mining algorithms in the areas of node embedding, whole-graph embedding, and community detection. It builds on the packages of NetworkX, [PyGSP](https://pygsp.readthedocs.io/en/stable/), Gensim, NumPy, and SciPy. Unfortunately, the package does not support directed graphs, though plans to do so have been stated. This project is worth monitoring.\n",
    "\n",
    "A third intriguing area involves the use of [quaternions](https://arxiv.org/pdf/1904.10281v3.pdf) based on [Clifford algebras](https://en.wikipedia.org/wiki/Clifford_algebra) in their machine learning codes. [Charles Peirce](https://en.wikipedia.org/wiki/Charles_Sanders_Peirce), the intellectual guide for the design of KBpedia, was a mathematician of some renown in his own time, and studied and applauded [William Kingdon Clifford](https://en.wikipedia.org/wiki/William_Kingdon_Clifford) and his emerging algebra as a contemporary in the 1870s, shortly before Clifford's untimely death. Peirce scholars have often pointed to this influence in the development of Peirce's own algebras. I am personally interested in probing this approach to learn a bit more of Peirce's manifest interests."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Organization of This Part's Installments\n",
    "These selections and the emphasis on our four areas lead to these anticipated **CWPK** installments over the coming weeks:\n",
    "\n",
    "- **CWPK #61 - NLP, Machine Learning and Analysis**\n",
    "- **CWPK #62 - Network and Graph Analysis**\n",
    "- **CWPK #63 - Staging Data Sci Resources and Preprocessing**\n",
    "- **CWPK #64 - Embedding, NLP Analysis, and Entity Recognition**\n",
    "- **CWPK #65 - scikit-learn Basics and Initial Analyses**\n",
    "- **CWPK #66 - scikit-learn Classifiers**\n",
    "- **CWPK #67 - Knowledge Graph Embedding Models**\n",
    "- **CWPK #68 - Setting Up and Configuring the Deep Graph Learners**\n",
    "- **CWPK #69 - DGL-KE Classifiers**\n",
    "- **CWPK #70 - State-of-Art PyG 2 Classifiers**\n",
    "- **CWPK #71 - A Comparison of Results**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Documentation\n",
    "Here are some general resources:\n",
    "\n",
    "- [Natural Language Processing Recipes: Best Practices and Examples](https://www.kdnuggets.com/2020/05/natural-language-processing-recipes-best-practices-examples.html) - nice set of NLP notebooks\n",
    "- Another set of notebooks https://aihub.cloud.google.com/s?category=notebook\n",
    "- [Machine Learning in Python: Main Developments and Technology Trends in Data Science, Machine Learning, and Artificial Intelligence](https://arxiv.org/abs/2002.04803) see Figure 1 for a possible Python architecture diagram; 48 pp and more of an academic overview. has many links to GitHub projects\n",
    "- [An Intuitive Understanding of Word Embeddings: From Count Vectors to Word2Vec](https://www.analyticsvidhya.com/blog/2017/06/word-embeddings-count-word2veec/) a pretty comprehensive overview from 2017\n",
    "- [Python Machine Learning Tutorials](https://realpython.com/tutorials/machine-learning/) - an overview of ML tutorials from a generally good source, [Real Python](https://realpython.com/)\n",
    "- [PyTorch vs. TensorFlow – A Detailed Comparison](https://www.tooploox.com/blog/pytorch-vs-tensorflow-a-detailed-comparison) - a balanced and fair comparison of the two frameworks.\n",
    "\n",
    "#### Network Representational Learning\n",
    "- [Machine Learning on Graphs: A Model and Comprehensive Taxonomy](https://arxiv.org/pdf/2005.03675.pdf) - the goal of this survey is to provide a unified view of representation learning methods for graph-structured data, to better understand the different ways to leverage graph structure in deep learning models; see [GitHub GCNN TensorFlow implementation](https://github.com/google/gcnn-survey-paper)\n",
    "- [Awesome Graph Classification](https://github.com/benedekrozemberczki/awesome-graph-classification) - a collection of graph classification methods, covering embedding, deep learning, graph kernel and factorization papers\n",
    "- [A Comprehensive Comparison of Unsupervised Network Representation Learning Methods](https://arxiv.org/pdf/1903.07902v2.pdf) - a comparison of unsupervised only and not attentive to heterogeneous networks.\n",
    "\n",
    "#### Knowledge Graph Representational Learning\n",
    "- [A Review of Relational Machine Learning for Knowledge Graphs](https://arxiv.org/pdf/1503.00759.pdf)(2015); one of the first to focus on the space\n",
    "- [Awesome Graph Representation Learning](https://github.com/ky-zhang/awesome-graph-representation-learning) - a curated list of awesome graph representation learning\n",
    "- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://arxiv.org/pdf/2002.00388v2.pdf) from August 2020\n",
    "- [Heterogeneous Network Representation Learning: Survey and Benchmark](https://arxiv.org/pdf/2004.00216.pdf) best paper for understanding the challenges of heterogeneous network embeddings\n",
    "- [Knowledge Graph Embedding: A Survey of Approaches and Applications](https://mnick.github.io/project/knowledge-graph-embeddings/) is an overview of embedding models of entities and relationships for knowledge base completion\n",
    "- [Introduction to Geometric Deep Learning](https://blog.paperspace.com/introduction-to-geometric-deep-learning/) \"GDL also shines in applications where the use of graphs is more common, like knowledge graphs.\"\n",
    "- [Geometric Deep Learning Library Comparison](https://blog.paperspace.com/geometric-deep-learning-framework-comparison/) follow-on to the above paper. Compares PyTorch Geometric, Deep Graph Library, Graph Nets\n",
    "- [RDF2Vec Light -- A Lightweight Approach for Knowledge Graph Embeddings](https://arxiv.org/abs/2009.07659) allows to train partial, task-specific models withonly a fraction of the computation requirements compared to other embedding ap-proaches, while retaining a high performance on multiple tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:#ffecec; border:1px dotted #f5aca6; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site. The <em>cowpoke</em> Python <a href=\"https://github.com/Cognonto/cowpoke\">code listing covering the series</a> is also available from GitHub.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}