rel_ext.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Relation extraction using distant supervision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Bill MacCartney (wcmac@cs.stanford.edu)\"\n",
    "__version__ = \"CS224U, Stanford, Spring 2018\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "0. [Overview](#Overview)\n",
    "    0. [The task of relation extraction](#The-task-of-relation-extraction)\n",
    "    0. [Hand-built patterns](#Hand-built-patterns)\n",
    "    0. [Supervised learning](#Supervised-learning)\n",
    "    0. [Distant supervision](#Distant-supervision)\n",
    "0. [Set-up](#Set-up)\n",
    "0. [The corpus](#The-corpus)\n",
    "0. [The knowledge base](#The-knowledge-base)\n",
    "0. [Problem formulation](#Problem-formulation)\n",
    "    0. [Joining the corpus and the KB](#Joining-the-corpus-and-the-KB)\n",
    "    0. [Negative examples]([Negative-examples])\n",
    "    0. [Multi-label classification]([Multi-label-classification])\n",
    "    0. [Building datasets](#Building-datasets)\n",
    "0. [Evaluation](#Evaluation)\n",
    "    0. [Splitting the data](#Splitting-the-data)\n",
    "    0. [Choosing evaluation metrics](#Choosing-evaluation-metrics)\n",
    "    0. [Running evaluations](#Running-evaluations)\n",
    "    0. [Evaluating a random-guessing strategy](#Evaluating-a-random-guessing-strategy)\n",
    "0. [A simple baseline model](#A-simple-baseline-model)\n",
    "0. [Building a classifier](#Building-a-classifier)\n",
    "    0. [Featurizers](#Featurizers)\n",
    "    0. [Experiments](#Experiments)\n",
    "    0. [Examining the trained models](#Examining-the-trained-models)\n",
    "    0. [Discovering new relation instances](#Discovering-new-relation-instances)\n",
    "0. [Next steps](#Next-steps)\n",
    "0. [Homework 3](#Homework-3)\n",
    "0. [Bake-off](#Bake-off)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "This codebook illustrates an approach to [relation extraction](http://deepdive.stanford.edu/relation_extraction) using [distant supervision](http://deepdive.stanford.edu/distant_supervision). It uses a simplified version of the approach taken by Mintz et al. in their 2009 paper, [Distant supervision for relation extraction without labeled data](https://www.aclweb.org/anthology/P09-1113). If you haven't yet read that paper, read it now! The rest of the codebook will make a lot more sense after you're familiar with it.\n",
    "\n",
    "### The task of relation extraction\n",
    "\n",
    "Relation extraction is the task of extracting from natural language text relational triples such as:\n",
    "\n",
    "```\n",
    "(founders, SpaceX, Elon_Musk)\n",
    "(has_spouse, Elon_Musk, Talulah_Riley)\n",
    "(worked_at, Elon_Musk, Tesla_Motors)\n",
    "```\n",
    "\n",
    "If we can accumulate a large knowledge base (KB) of relational triples, we can use it to power question answering and other applications. Building a KB manually is slow and expensive, but much of the knowledge we'd like to capture is already expressed in abundant text on the web. The aim of relation extraction, therefore, is to accelerate the construction of new KBs — and facilitate the ongoing curation of existing KBs — by extracting relational triples from natural language text.\n",
    "\n",
    "### Hand-built patterns\n",
    "\n",
    "An obvious way to start is to write down a few patterns which express each relation. For example, we can use the pattern \"X is the founder of Y\" to find new instances of the `founders` relation. If we search a large corpus, we may find the phrase \"Elon Musk is the founder of SpaceX\", which we can use as evidence for the relational triple `(founders, SpaceX, Elon_Musk)`.\n",
    "\n",
    "Unfortunately, this approach doesn't get us very far. The central challenge of relation extraction is the fantastic diversity of language, the multitude of possible ways to express a given relation. For example, each of the following sentences expressed the relational triple `(founders, SpaceX, Elon_Musk)`:\n",
    "\n",
    "- \"You may also be thinking of _Elon Musk_ (founder of _SpaceX_), who started PayPal.\"\n",
    "- \"Interesting Fact: _Elon Musk_, co-founder of PayPal, went on to establish _SpaceX_, one of the most promising space travel startups in the world.\"\n",
    "- \"If Space Exploration (_SpaceX_), founded by Paypal pioneer _Elon Musk_ succeeds, commercial advocates will gain credibility and more support in Congress.\"\n",
    "\n",
    "The patterns which connect \"Elon Musk\" with \"SpaceX\" in these examples are not ones we could have easily anticipated. To do relation extraction effectively, we need to go beyond hand-built patterns.\n",
    "\n",
    "### Supervised learning\n",
    "\n",
    "Effective relation extraction will require applying machine learning methods. The natural place to start is with supervised learning. This means training an extraction model from a dataset of examples which have been labeled with the target output. Sentences like the three examples above would be annotated with the `founders` relation, but we'd also have sentences which include \"Elon Musk\" and \"SpaceX\" but do not express the `founders` relation, such as:\n",
    "\n",
    "- \"Billionaire entrepreneur _Elon Musk_ announced the latest addition to the _SpaceX_ arsenal: the 'Big F---ing Rocket' (BFR)\".\n",
    "\n",
    "Such \"negative examples\" would be labeled as such, and the fully-supervised model would then be able to learn from both positive and negative examples the linguistic patterns that indicate each relation.\n",
    "\n",
    "The difficulty with the fully-supervised approach is the cost of generating training data. Because of the great diversity of linguistic expression, our model will need lots and lots of training data: at least tens of thousands of examples, although hundreds of thousands or millions would be much better. But labeling the examples is just a slow and expensive as building the KB by hand would be.\n",
    "\n",
    "### Distant supervision\n",
    "\n",
    "The goal of distant supervision is to capture the benefits of supervised learning without paying the cost of labeling training data. Instead of labeling extraction examples by hand, we use existing relational triples to automatically identify extraction examples in a large corpus. For example, if we already have in our KB the relational triple `(founders, SpaceX, Elon_Musk)`, we can search a large corpus for sentences in which \"SpaceX\" and \"Elon Musk\" co-occur, make the (unreliable!) assumption that all the sentences express the `founder` relation, and then use them as training data for a learned model to identify new instances of the `founder` relation — all without doing any manual labeling.\n",
    "\n",
    "This is a powerful idea, but it has two limitations. The first is that, inevitably, some of the sentences in which \"SpaceX\" and \"Elon Musk\" co-occur will not express the `founder` relation — like the BFR example above. By making the blind assumption that all such sentences do express the `founder` relation, we are essentially injecting noise into our training data, and making it harder for our learning algorithms to learn good models. Distant supervision is effective in spite of this problem because it makes it possible to leverage vastly greater quantities of training data, and the benefit of more data outweighs the harm of noisier data.\n",
    "\n",
    "The second limitation is that we need an existing KB to start from. We can only train a model to extract new instances of the `founders` relation if we already have many instances of the `founders` relation. Thus, while distant supervision is a great way to extend an existing KB, it's not useful for creating a KB containing new relations from scratch.\n",
    "\n",
    "\\[ [top](#Relation-extraction-using-distant-supervision) \\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up\n",
    "\n",
    "- Make sure your environment includes all the requirements for the cs224u repository.\n",
    "- Download the [data distribution for this unit](https://web.stanford.edu/class/cs224u/data/rel_ext_data.zip), unpack it, and place it in the directory containing the course repository. (If you want to put it somewhere else, change `rel_ext_data_home` below.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "rel_ext_data_home = 'rel_ext_data'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gzip\n",
    "import numpy as np\n",
    "import random\n",
    "import os\n",
    "\n",
    "from collections import Counter, defaultdict, namedtuple\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import precision_recall_fscore_support\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\[ [top](#Relation-extraction-using-distant-supervision) \\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual when we're doing NLP, we need to start with a _corpus_ — a large sample of natural language text. And because our goal is to do relation extraction with distant supervision, we need to be able to identify entities in the text and connect them to a knowledge base of relations between entities. So, we need a corpus in which entity mentions are annotated with _entity resolutions_ which map them to a unique, unambiguous identifiers. Entity resolution serves two purposes:\n",
    "\n",
    "0. It ensures that if an entity mention could refer to two different entities, it is properly disambiguated. For example, \"New York\" could refer to the city or the state.\n",
    "0. It ensures that if two different entity mentions refer to the same entity, they are properly identified. For example, both \"New York City\" and \"The Big Apple\" refer to New York City.\n",
    "\n",
    "The corpus we'll use for this project is derived from the [Wikilinks dataset](https://code.google.com/archive/p/wiki-links/) [announced by Google in 2013](https://research.googleblog.com/2013/03/learning-from-big-data-40-million.html). This dataset contains over 40M mentions of 3M distinct entities spanning 10M webpages. It provides entity resolutions by mapping each entity mention to a Wikipedia URL.\n",
    "\n",
    "Now, in order to do relation extraction, we actually need _pairs_ of entity mentions, and it's important to have the context around and between the two mentions. Fortunately, UMass has provided an [expanded version of Wikilinks](http://www.iesl.cs.umass.edu/data/data-wiki-links) which includes the context around each entity mention. We've written code to stitch together pairs of entity mentions along with their contexts, and we've filtered the examples extensively. The result is a compact corpus suitable for our purposes. Let's take a closer look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading examples from rel_ext_data/corpus.tsv.gz\n",
      "Read 414123 examples\n"
     ]
    }
   ],
   "source": [
    "Example = namedtuple('Example',\n",
    "    'entity_1, entity_2, left, mention_1, middle, mention_2, right, '\n",
    "    'left_POS, mention_1_POS, middle_POS, mention_2_POS, right_POS')\n",
    "\n",
    "def read_examples():\n",
    "    examples = []\n",
    "    path = os.path.join(rel_ext_data_home, 'corpus.tsv.gz')\n",
    "    print('Reading examples from {}'.format(path))\n",
    "    with gzip.open(path) as f:\n",
    "        for line in f:\n",
    "            fields = line[:-1].decode('utf-8').split('\\t')\n",
    "            examples.append(Example(*fields))\n",
    "    print('Read {} examples'.format(len(examples)))            \n",
    "    return examples\n",
    "\n",
    "examples = read_examples()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, that's a lot of examples! Let's take a closer look at one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example(entity_1='New_Mexico', entity_2='Arizona', left='to all Spanish-occupied lands . The horno has a beehive shape and uses wood as the only heat source . The procedure still used in parts of', mention_1='New Mexico', middle='and', mention_2='Arizona', right='is to build a fire inside the Horno and , when the proper amount of time has passed , remove the embers and ashes and insert the', left_POS='to/TO all/DT Spanish-occupied/JJ lands/NNS ./. The/DT horno/NN has/VBZ a/DT beehive/NN shape/NN and/CC uses/VBZ wood/NN as/IN the/DT only/JJ heat/NN source/NN ./. The/DT procedure/NN still/RB used/VBN in/IN parts/NNS of/IN', mention_1_POS='New/NNP Mexico/NNP', middle_POS='and/CC', mention_2_POS='Arizona/NNP', right_POS='is/VBZ to/TO build/VB a/DT fire/NN inside/IN the/DT Horno/NNP and/CC ,/, when/WRB the/DT proper/JJ amount/NN of/IN time/NN has/VBZ passed/VBN ,/, remove/VB the/DT embers/NNS and/CC ashes/NNS and/CC insert/VB the/DT')\n"
     ]
    }
   ],
   "source": [
    "print(examples[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every example represents a fragment of webpage text containing two entity mentions. The first two fields, `entity_1` and `entity_2`, contain unique identifiers for the two entities mentioned. We name entities using Wiki IDs, which you can think of as the last portion of a Wikipedia URL. Thus the Wiki ID `Barack_Obama` designates the entity described by [https://en.wikipedia.org/wiki/Barack_Obama](https://en.wikipedia.org/wiki/Barack_Obama).\n",
    "\n",
    "The next five fields represent the text surrounding the two mentions, divided into five chunks: `left` contains the text before the first mention, `mention_1` is the first mention itself, `middle` contains the text between the two mentions, `mention_2` is the second mention, and the field `right` contains the text after the second mention. Thus, we can reconstruct the context as a single string like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'to all Spanish-occupied lands . The horno has a beehive shape and uses wood as the only heat source . The procedure still used in parts of New Mexico and Arizona is to build a fire inside the Horno and , when the proper amount of time has passed , remove the embers and ashes and insert the'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex = examples[1]\n",
    "' '.join((ex.left, ex.mention_1, ex.middle, ex.mention_2, ex.right))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last five fields contain the same five chunks of text, but this time annotated with part-of-speech (POS) tags, which may turn out to be useful when we start building models for relation extraction.\n",
    "\n",
    "Let's look at the distribution of entities over the corpus. How many entities are there, and what are the most common ones?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The corpus contains 107820 entities\n",
      "The most common entities are:\n",
      "      9399 India\n",
      "      6214 England\n",
      "      4585 Germany\n",
      "      4486 France\n",
      "      4128 Australia\n",
      "      3939 China\n",
      "      3930 Canada\n",
      "      3897 Italy\n",
      "      3368 California\n",
      "      3125 Pakistan\n",
      "      3103 Europe\n",
      "      3097 New_York_City\n",
      "      3025 London\n",
      "      2470 Japan\n",
      "      2468 United_Kingdom\n",
      "      2279 New_Zealand\n",
      "      2275 New_York\n",
      "      2259 Spain\n",
      "      2132 Philippines\n",
      "      2120 Asia\n"
     ]
    }
   ],
   "source": [
    "counter = Counter()\n",
    "for example in examples:\n",
    "    counter[example.entity_1] += 1\n",
    "    counter[example.entity_2] += 1\n",
    "print('The corpus contains {} entities'.format(len(counter)))\n",
    "counts = sorted([(count, key) for key, count in counter.items()], reverse=True)\n",
    "print('The most common entities are:')\n",
    "for count, key in counts[:20]:\n",
    "    print('{:10d} {}'.format(count, key))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we're frequently going to want to retrieve corpus examples containing specific entities, it will be convenient to create a `Corpus` class which holds not only the examples themselves, but also a precomputed index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Corpus():\n",
    "\n",
    "    def __init__(self, examples):\n",
    "        self._examples = examples\n",
    "        self._examples_by_entities = {}\n",
    "        self._index_examples_by_entities()\n",
    "\n",
    "    def _index_examples_by_entities(self):\n",
    "        for ex in self._examples:\n",
    "            if ex.entity_1 not in self._examples_by_entities:\n",
    "                self._examples_by_entities[ex.entity_1] = {}\n",
    "            if ex.entity_2 not in self._examples_by_entities[ex.entity_1]:\n",
    "                self._examples_by_entities[ex.entity_1][ex.entity_2] = []\n",
    "            self._examples_by_entities[ex.entity_1][ex.entity_2].append(ex)\n",
    "    \n",
    "    def get_examples(self):\n",
    "        return iter(self._examples)\n",
    "        \n",
    "    def get_examples_for_entities(self, e1, e2):\n",
    "        try:\n",
    "            return self._examples_by_entities[e1][e2]\n",
    "        except KeyError:\n",
    "            return []\n",
    "        \n",
    "    def __repr__(self):\n",
    "        return 'Corpus with {} examples'.format(len(self._examples))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Corpus with 414123 examples"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus = Corpus(examples)\n",
    "corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main benefit we gain from the `Corpus` class is the ability to retrieve examples containing specific entities. Let's find examples containing `Steve_Jobs` and `Pixar`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The first of 9 examples for Steve_Jobs and Pixar is:\n",
      "Example(entity_1='Steve_Jobs', entity_2='Pixar', left='of visual effects on films like The Abyss ( 1989 ) , Terminator 2 ( 1991 ) and Jurassic Park ( 1993 ) The computer graphics division of ILM was bought by', mention_1='Steve Jobs', middle='and became', mention_2='Pixar', right=', who would go on to make several groundbreaking animated films starting with Toy Story ( 1995 ) – more information on the history of that here', left_POS='of/IN visual/JJ effects/NNS on/IN films/NNS like/IN The/DT Abyss/NN -LRB-/-LRB- 1989/CD -RRB-/-RRB- ,/, Terminator/NNP 2/CD -LRB-/-LRB- 1991/CD -RRB-/-RRB- and/CC Jurassic/JJ Park/NN -LRB-/-LRB- 1993/CD -RRB-/-RRB- The/DT computer/NN graphics/NNS division/NN of/IN ILM/NNP was/VBD bought/VBN by/IN', mention_1_POS='Steve/NNP Jobs/NNP', middle_POS='and/CC became/VBD', mention_2_POS='Pixar/NNP', right_POS=',/, who/WP would/MD go/VB on/IN to/TO make/VB several/JJ groundbreaking/VBG animated/JJ films/NNS starting/VBG with/IN Toy/NNP Story/NNP -LRB-/-LRB- 1995/CD -RRB-/-RRB- --/: more/JJR information/NN on/IN the/DT history/NN of/IN that/DT here/RB')\n"
     ]
    }
   ],
   "source": [
    "def show_examples_for_pair(e1, e2, corpus):\n",
    "    exs = corpus.get_examples_for_entities(e1, e2)\n",
    "    if exs:\n",
    "        print('The first of {} examples for {} and {} is:'.format(len(exs), e1, e2))\n",
    "        print(exs[0])\n",
    "    else:\n",
    "        print('No examples for {} and {} is:'.format(e1, e2))\n",
    "\n",
    "show_examples_for_pair('Steve_Jobs', 'Pixar', corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actually, this might not be all of the examples containing `Steve_Jobs` and `Pixar`. It's only the examples where `Steve_Jobs` was mentioned first and `Pixar` second. There may be additional examples that have them in the reverse order. Let's check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The first of 2 examples for Pixar and Steve_Jobs is:\n",
      "Example(entity_1='Pixar', entity_2='Steve_Jobs', left='in the visual accompaniment to his recordings of Bach ’ s Six Suites for Unaccompanied Cello . Ma has also been seen with Apple Inc. and former', mention_1='Pixar', middle='CEO', mention_2='Steve Jobs', right='. Ma is often invited to press events for Jobs ’ s companies , and has performed on stage during event keynote presentations , as well as appearing in', left_POS=\"in/IN the/DT visual/JJ accompaniment/NN to/TO his/PRP$ recordings/NNS of/IN Bach/NNP '/POS s/NNS Six/CD Suites/NNP for/IN Unaccompanied/NNP Cello/NNP ./. Ma/NNP has/VBZ also/RB been/VBN seen/VBN with/IN Apple/NNP Inc./NNP and/CC former/JJ\", mention_1_POS='Pixar/NNP', middle_POS='CEO/NNP', mention_2_POS='Steve/NNP Jobs/NNP', right_POS=\"./. Ma/NNP is/VBZ often/RB invited/VBN to/TO press/VB events/NNS for/IN Jobs/NNP '/POS s/NNS companies/NNS ,/, and/CC has/VBZ performed/VBN on/IN stage/NN during/IN event/NN keynote/NN presentations/NNS ,/, as/RB well/RB as/IN appearing/VBG in/IN\")\n"
     ]
    }
   ],
   "source": [
    "show_examples_for_pair('Pixar', 'Steve_Jobs', corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sure enough. Going forward, we'll have to remember to check both \"directions\" when we're looking for examples contains a specific pair of entities.\n",
    "\n",
    "This corpus is not without flaws. As you get more familiar with it, you will likely discover that it contains many examples that are nearly — but not exactly — duplicates. This seems to be a consequence of the web document sampling methodology that was used in the construction of the Wikilinks dataset. However, despite a few warts, it will serve our purposes.\n",
    "\n",
    "One thing this corpus does _not_ include is any annotation about relations. Thus, it could not be used for the fully-supervised approach to relation extraction, because the fully-supervised approach requires that each pair of entity mentions be annotated with the relation (if any) that holds between the two entities. In order to make any headway, we'll need to connect the corpus with an external source of knowledge about relations. We need a knowledge base.\n",
    "\n",
    "\\[ [top](#Relation-extraction-using-distant-supervision) \\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The knowledge base"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data distribution for this unit includes a _knowledge base_ (KB) ultimately derived from [Freebase](https://en.wikipedia.org/wiki/Freebase). Unfortunately, Freebase was shut down in 2016, but the Freebase data is still available from various sources and in various forms. The KB included here was extracted from the [Freebase Easy data dump](http://freebase-easy.cs.uni-freiburg.de/dump/).\n",
    "\n",
    "The KB is a collection of _relational triples_, each consisting of a _relation_, a _subject_, and an _object_. For example, here are three triples from the KB:\n",
    "\n",
    "```\n",
    "(place_of_birth, Barack_Obama, Honolulu)\n",
    "(has_spouse, Barack_Obama, Michelle_Obama)\n",
    "(author, The_Audacity_of_Hope, Barack_Obama)\n",
    "```\n",
    "\n",
    "As you might guess:\n",
    "\n",
    "- The relation is one of a handful of predefined constants, such as `place_of_birth` or `has_spouse`.\n",
    "- The subject and object are entities represented by Wiki IDs (that is, suffixes of Wikipedia URLs).\n",
    "\n",
    "Let's write some code to read the KB so that we can take a closer look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading KB triples from rel_ext_data/kb.tsv.gz ...\n",
      "Read 56575 KB triples\n"
     ]
    }
   ],
   "source": [
    "KBTriple = namedtuple('KBTriple', 'rel, sbj, obj')\n",
    "\n",
    "def read_kb_triples():\n",
    "    kb_triples = []\n",
    "    path = os.path.join(rel_ext_data_home, 'kb.tsv.gz')\n",
    "    print('Reading KB triples from {} ...'.format(path))\n",
    "    with gzip.open(path) as f:\n",
    "        for line in f:\n",
    "            rel, sbj, obj = line[:-1].decode('utf-8').split('\\t')\n",
    "            kb_triples.append(KBTriple(rel, sbj, obj))\n",
    "    print('Read {} KB triples'.format(len(kb_triples)))\n",
    "    return kb_triples\n",
    "\n",
    "kb_triples = read_kb_triples()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK great, we have a KB!\n",
    "\n",
    "Now, just as we did for the corpus, we'll create a `KB` class to store the KB triples and some associated indexes. We'll want to be able to look up KB triples both by relation and by entities, so we'll create indexes for both of those access patterns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class KB():\n",
    "\n",
    "    def __init__(self, kb_triples):\n",
    "        self._kb_triples = kb_triples\n",
    "        self._all_relations = []\n",
    "        self._all_entity_pairs = []\n",
    "        self._kb_triples_by_relation = {}\n",
    "        self._kb_triples_by_entities = {}\n",
    "        self._collect_all_entity_pairs()\n",
    "        self._index_kb_triples_by_relation()\n",
    "        self._index_kb_triples_by_entities()\n",
    "\n",
    "    def _collect_all_entity_pairs(self):\n",
    "        pairs = set()\n",
    "        for kbt in self._kb_triples:\n",
    "            pairs.add((kbt.sbj, kbt.obj))\n",
    "        self._all_entity_pairs = sorted(list(pairs))\n",
    "        \n",
    "    def _index_kb_triples_by_relation(self):\n",
    "        for kbt in self._kb_triples:\n",
    "            if kbt.rel not in self._kb_triples_by_relation:\n",
    "                self._kb_triples_by_relation[kbt.rel] = []\n",
    "            self._kb_triples_by_relation[kbt.rel].append(kbt)\n",
    "        self._all_relations = sorted(list(self._kb_triples_by_relation))\n",
    "    \n",
    "    def _index_kb_triples_by_entities(self):\n",
    "        for kbt in self._kb_triples:\n",
    "            if kbt.sbj not in self._kb_triples_by_entities:\n",
    "                self._kb_triples_by_entities[kbt.sbj] = {}\n",
    "            if kbt.obj not in self._kb_triples_by_entities[kbt.sbj]:\n",
    "                self._kb_triples_by_entities[kbt.sbj][kbt.obj] = []\n",
    "            self._kb_triples_by_entities[kbt.sbj][kbt.obj].append(kbt)\n",
    "\n",
    "    def get_triples(self):\n",
    "        return iter(self._kb_triples)\n",
    "        \n",
    "    def get_all_relations(self):\n",
    "        return self._all_relations\n",
    "            \n",
    "    def get_all_entity_pairs(self):\n",
    "        return self._all_entity_pairs\n",
    "            \n",
    "    def get_triples_for_relation(self, rel):\n",
    "        try:\n",
    "            return self._kb_triples_by_relation[rel]\n",
    "        except KeyError:\n",
    "            return []\n",
    "\n",
    "    def get_triples_for_entities(self, e1, e2):\n",
    "        try:\n",
    "            return self._kb_triples_by_entities[e1][e2]\n",
    "        except KeyError:\n",
    "            return []\n",
    "\n",
    "    def __repr__(self):\n",
    "        return 'KB with {} triples'.format(len(self._kb_triples))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KB with 56575 triples"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kb = KB(kb_triples)\n",
    "kb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get a sense of the high-level characteristics of this KB. Some questions we'd like to answer:\n",
    "\n",
    "- How many relations are there?\n",
    "- How big is each relation?\n",
    "- Examples of each relation.\n",
    "- How many unique entities does the KB include?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16\n"
     ]
    }
   ],
   "source": [
    "all_relations = kb.get_all_relations()\n",
    "print(len(all_relations))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How big is each relation? That is, how many triples does each relation contain?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        2140 adjoins\n",
      "        3316 author\n",
      "         637 capital\n",
      "       22489 contains\n",
      "        4958 film_performance\n",
      "        2404 founders\n",
      "        1012 genre\n",
      "        3280 has_sibling\n",
      "        3774 has_spouse\n",
      "        3153 is_a\n",
      "        1981 nationality\n",
      "        2013 parents\n",
      "        1388 place_of_birth\n",
      "        1031 place_of_death\n",
      "        1526 profession\n",
      "        1473 worked_at\n"
     ]
    }
   ],
   "source": [
    "for rel in all_relations:\n",
    "    print('{:12d} {}'.format(len(kb.get_triples_for_relation(rel)), rel))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at one example from each relation, so that we can get a sense of what they mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('adjoins', 'Siegburg', 'Bonn')\n",
      "('author', 'Uncle_Silas', 'Sheridan_Le_Fanu')\n",
      "('capital', 'Tunisia', 'Tunis')\n",
      "('contains', 'Brickfields', 'Kuala_Lumpur_Sentral_railway_station')\n",
      "('film_performance', 'Colin_Hanks', 'The_Great_Buck_Howard')\n",
      "('founders', 'Bomis', 'Jimmy_Wales')\n",
      "('genre', 'SPARQL', 'Semantic_Web')\n",
      "('has_sibling', 'Ari_Emanuel', 'Rahm_Emanuel')\n",
      "('has_spouse', 'Percy_Bysshe_Shelley', 'Mary_Shelley')\n",
      "('is_a', 'Bhanu_Athaiya', 'Costume_designer')\n",
      "('nationality', 'Ruben_Rausing', 'Sweden')\n",
      "('parents', 'Prince_Arthur_of_Connaught', 'Prince_Arthur,_Duke_of_Connaught_and_Strathearn')\n",
      "('place_of_birth', 'William_Penny_Brookes', 'Much_Wenlock')\n",
      "('place_of_death', 'Jean_Drapeau', 'Montreal')\n",
      "('profession', 'Rufus_Wainwright', 'Actor')\n",
      "('worked_at', 'Ray_Jackendoff', 'Tufts_University')\n"
     ]
    }
   ],
   "source": [
    "for rel in all_relations:\n",
    "    print(tuple(kb.get_triples_for_relation(rel)[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `get_triples_for_entities()` method allows us to look up triples by the entities they contain. Let's use it to see what relation(s) hold between `France` and `Germany`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[KBTriple(rel='adjoins', sbj='France', obj='Germany')]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kb.get_triples_for_entities('France', 'Germany')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Relations like `adjoins` and `has_sibling` are intuitively symmetric — if the relation holds between _X_ and _Y_, then we expect it to hold between _Y_ and _X_ as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[KBTriple(rel='adjoins', sbj='Germany', obj='France')]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kb.get_triples_for_entities('Germany', 'France')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, there's no guarantee that all such inverse triples actually appear in the KB. (You could write some code to check.)\n",
    "\n",
    "Most relations, however, are intuitively asymmetric. Let's see what relation holds between `Pixar` and `Steve_Jobs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[KBTriple(rel='founders', sbj='Pixar', obj='Steve_Jobs')]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kb.get_triples_for_entities('Pixar', 'Steve_Jobs')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a bit arbitrary that the KB includes a given asymmetric relation rather than its inverse. For example, instead of the `founders` relation with triple `(founders, Pixar, Steve_Jobs)`, we might have had a `founder_of` relation with triple `(founder_of, Steve_Jobs, Pixar)`. It doesn't really matter.\n",
    "\n",
    "Although we don't have a `founder_of` relation, there might still be a relation between `Steve_Jobs` and `Pixar`. Let's check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[KBTriple(rel='worked_at', sbj='Steve_Jobs', obj='Pixar')]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kb.get_triples_for_entities('Steve_Jobs', 'Pixar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aha, yes, that makes sense. So it can be the case that one relation holds between _X_ and _Y_, and a different relation holds between _Y_ and _X_.\n",
    "\n",
    "One more observation: there may be more than one relation that holds between a given pair of entities, even in one direction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[KBTriple(rel='has_sibling', sbj='Cleopatra', obj='Ptolemy_XIII_Theos_Philopator'),\n",
       " KBTriple(rel='has_spouse', sbj='Cleopatra', obj='Ptolemy_XIII_Theos_Philopator')]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kb.get_triples_for_entities('Cleopatra', 'Ptolemy_XIII_Theos_Philopator')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No! What? Yup, it's true — [Cleopatra](https://en.wikipedia.org/wiki/Cleopatra) married her younger brother, [Ptolemy XIII](https://en.wikipedia.org/wiki/Ptolemy_XIII_Theos_Philopator). Wait, it gets worse — she also married her _even younger_ brother, [Ptolemy XIV](https://en.wikipedia.org/wiki/Ptolemy_XIV_of_Egypt). Apparently this was normal behavior in ancient Egypt.\n",
    "\n",
    "Moving on ...\n",
    "\n",
    "Let's look at the distribution of entities in the KB. How many entities are there, and what are the most common ones?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The KB contains 46275 entities\n",
      "The most common entities are:\n",
      "       962 England\n",
      "       815 India\n",
      "       465 London\n",
      "       456 Italy\n",
      "       437 France\n",
      "       420 Germany\n",
      "       412 California\n",
      "       396 United_Kingdom\n",
      "       378 Canada\n",
      "       324 New_York_City\n",
      "       262 Actor\n",
      "       248 New_York\n",
      "       244 Australia\n",
      "       235 China\n",
      "       226 Philippines\n",
      "       224 Japan\n",
      "       223 Russia\n",
      "       214 Scotland\n",
      "       204 Europe\n",
      "       177 Pakistan\n"
     ]
    }
   ],
   "source": [
    "counter = Counter()\n",
    "for kbt in kb.get_triples():\n",
    "    counter[kbt.sbj] += 1\n",
    "    counter[kbt.obj] += 1\n",
    "print('The KB contains {} entities'.format(len(counter)))\n",
    "counts = sorted([(count, key) for key, count in counter.items()], reverse=True)\n",
    "print('The most common entities are:')\n",
    "for count, key in counts[:20]:\n",
    "    print('{:10d} {}'.format(count, key))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of entities in the KB is less than half the number of entities in the corpus! Evidently the corpus has much broader coverage than the KB.\n",
    "\n",
    "Note that there is no promise or expectation that this KB is _complete_. Not only does the KB contain no mention of many entities from the corpus — even for the entities it does include, there may be possible triples which are true in the world but are missing from the KB. As an example, these triples are in the KB:\n",
    "\n",
    "```\n",
    "(founders, SpaceX, Elon_Musk)\n",
    "(founders, Tesla_Motors, Elon_Musk)\n",
    "(worked_at, Elon_Musk, Tesla_Motors)\n",
    "```\n",
    "\n",
    "but this one is not:\n",
    "\n",
    "```\n",
    "(worked_at, Elon_Musk, SpaceX)\n",
    "```\n",
    "\n",
    "In fact, the whole point of developing methods for automatic relation extraction is to extend existing KBs (and build new ones) by identifying new relational triples from natural language text. If our KBs were complete, we wouldn't have anything to do.\n",
    "\n",
    "\\[ [top](#Relation-extraction-using-distant-supervision) \\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem formulation\n",
    "\n",
    "With our data assets in hand, it's time to provide a precise formulation of the prediction problem we aim to solve. We need to specify:\n",
    "\n",
    "- What is the input to the prediction?\n",
    "    - Is it a specific pair of entity _mentions_ in a specific context?\n",
    "    - Or is it a pair of _entities_, apart from any specific mentions?\n",
    "- What is the output of the prediction?\n",
    "    - Do we need to predict at most one relation label? (This is [multi-class classification](https://en.wikipedia.org/wiki/Multiclass_classification).)\n",
    "    - Or can we predict multiple relation labels? (This is [multi-label classification](https://en.wikipedia.org/wiki/Multi-label_classification).)\n",
    "\n",
    "### Joining the corpus and the KB\n",
    "\n",
    "In order to leverage the distant supervision paradigm, we'll need to connect information in the corpus with information in the KB. There are two possibilities, depending on how we formulate our prediction problem:\n",
    "\n",
    "- __Use the KB to generate labels for the corpus.__ If our problem is to classify a pair of entity _mentions_ in a specific example in the corpus, then we can use the KB to provide labels for training examples. Labeling specific examples is how the fully supervised paradigm works, so it's the obvious way to think about leveraging distant supervision as well. Although it can be made to work, it's not actually the preferred approach.\n",
    "- __Use the corpus to generate features for entity pairs.__ If instead our problem is to classify a pair of _entities_, then we can use all the examples from the corpus where those two entities co-occur to generate a feature representation describing the entity pair. This is the approach taken by [Mintz et al. 2009](https://www.aclweb.org/anthology/P09-1113), and it's the approach we'll pursue here.\n",
    "\n",
    "So we'll formulate our prediction problem such that the input is a pair of entities, and the goal is to predict what relation(s) the pair belongs to. The KB will provide the labels, and the corpus will provide the features.\n",
    "\n",
    "Let's determine how many examples we have for each triple in the KB. We'll compute averages per relation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                             examples\n",
      "relation               examples    triples    /triple\n",
      "--------               --------    -------    -------\n",
      "adjoins                   85660       2140      40.03\n",
      "author                    15822       3316       4.77\n",
      "capital                   12520        637      19.65\n",
      "contains                  99572      22489       4.43\n",
      "film_performance          11195       4958       2.26\n",
      "founders                   8061       2404       3.35\n",
      "genre                      1941       1012       1.92\n",
      "has_sibling               12332       3280       3.76\n",
      "has_spouse                16188       3774       4.29\n",
      "is_a                       6955       3153       2.21\n",
      "nationality                4649       1981       2.35\n",
      "parents                    5387       2013       2.68\n",
      "place_of_birth             2214       1388       1.60\n",
      "place_of_death             2047       1031       1.99\n",
      "profession                 2876       1526       1.88\n",
      "worked_at                  4494       1473       3.05\n"
     ]
    }
   ],
   "source": [
    "def count_examples(corpus, kb):\n",
    "    counter = Counter()\n",
    "    for rel in all_relations:\n",
    "        for kbt in kb.get_triples_for_relation(rel):\n",
    "            # count examples in both forward and reverse directions\n",
    "            counter[rel] += len(corpus.get_examples_for_entities(kbt.sbj, kbt.obj))\n",
    "            counter[rel] += len(corpus.get_examples_for_entities(kbt.obj, kbt.sbj))\n",
    "    # report results\n",
    "    print('{:20s} {:>10s} {:>10s} {:>10s}'.format('', '', '', 'examples'))\n",
    "    print('{:20s} {:>10s} {:>10s} {:>10s}'.format('relation', 'examples', 'triples', '/triple'))\n",
    "    print('{:20s} {:>10s} {:>10s} {:>10s}'.format('--------', '--------', '-------', '-------'))\n",
    "    for rel in all_relations:\n",
    "        nx = counter[rel]\n",
    "        nt = len(kb.get_triples_for_relation(rel))\n",
    "        print('{:20s} {:10d} {:10d} {:10.2f}'.format(rel, nx, nt, 1.0 * nx / nt))\n",
    "        \n",
    "count_examples(corpus, kb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For most relations, the total number of examples is fairly large, so we can be optimistic about learning what linguistic patterns express a given relation. However, for individual entity pairs, the number of examples is often quite low. Of course, more data would be better — much better! But more data could quickly become unwieldy to work with in a notebook like this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Negative instances\n",
    "\n",
    "By joining the corpus to the KB, we can obtain abundant positive instances for each relation. But a classifier cannot be trained on positive instances alone. In order to apply the distant supervision paradigm, we will also need some negative instances — that is, entity pairs which do not belong to any known relation. If you like, you can think of these entity pairs as being assigned to a special relation called `NO_RELATION`. We can find plenty of such pairs by searching for examples in the corpus which contain two entities which do not belong to any relation in the KB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_unrelated_pairs(corpus, kb):\n",
    "    unrelated_pairs = set()\n",
    "    for ex in corpus.get_examples():\n",
    "        if kb.get_triples_for_entities(ex.entity_1, ex.entity_2):\n",
    "            continue\n",
    "        if kb.get_triples_for_entities(ex.entity_2, ex.entity_1):\n",
    "            continue\n",
    "        unrelated_pairs.add((ex.entity_1, ex.entity_2))\n",
    "        unrelated_pairs.add((ex.entity_2, ex.entity_1))\n",
    "    return unrelated_pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 301073 unrelated pairs, including:\n",
      "    ('Legnago', 'Peschiera_del_Garda')\n",
      "    ('New_York', 'Dancehall')\n",
      "    ('Saranya_Ponvannan', 'Thoothukudi')\n",
      "    ('Information_architecture', 'Editing')\n",
      "    ('Franco', 'Spanish_Civil_War')\n",
      "    ('Ricky_Martin', 'Juan_Gabriel')\n",
      "    ('Anatolia', 'Israel')\n",
      "    ('Randy_Myers', 'Dan_Plesac')\n",
      "    ('Carcass_Island', 'Bleaker_Island')\n",
      "    ('Jo_Dee_Messina', 'Neal_McCoy')\n"
     ]
    }
   ],
   "source": [
    "unrelated_pairs = find_unrelated_pairs(corpus, kb)\n",
    "print('Found {} unrelated pairs, including:'.format(len(unrelated_pairs)))\n",
    "for pair in list(unrelated_pairs)[:10]:\n",
    "    print('   ', pair)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's a lot of negative instances! In fact, because these negative instances far outnumber our positive instances (that is, the triples in our KB), when we train models we'll wind up downsampling the negative instances substantially.\n",
    "\n",
    "Remember, though, that some of these supposedly negative instances may be false negatives. Our KB is not complete. A pair of entities might be related in real life even if they don't appear together in the KB."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-label classification\n",
    "\n",
    "A given pair of entities can belong to more than one relation. In fact, this is quite common in our KB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_relation_combinations(kb):\n",
    "    counter = Counter()\n",
    "    for sbj, obj in kb.get_all_entity_pairs():\n",
    "        rels = tuple(sorted(set([kbt.rel for kbt in kb.get_triples_for_entities(sbj, obj)])))\n",
    "        if len(rels) > 1:\n",
    "            counter[rels] += 1\n",
    "    counts = sorted([(count, key) for key, count in counter.items()], reverse=True)\n",
    "    print('The most common relation combinations are:')\n",
    "    for count, key in counts:\n",
    "        print('{:10d} {}'.format(count, key))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The most common relation combinations are:\n",
      "      1526 ('is_a', 'profession')\n",
      "       495 ('capital', 'contains')\n",
      "       183 ('place_of_birth', 'place_of_death')\n",
      "        76 ('nationality', 'place_of_birth')\n",
      "        11 ('nationality', 'place_of_death')\n",
      "        11 ('adjoins', 'contains')\n",
      "         8 ('has_sibling', 'has_spouse')\n",
      "         3 ('nationality', 'place_of_birth', 'place_of_death')\n",
      "         2 ('parents', 'worked_at')\n",
      "         1 ('nationality', 'worked_at')\n",
      "         1 ('has_spouse', 'parents')\n",
      "         1 ('author', 'founders')\n"
     ]
    }
   ],
   "source": [
    "count_relation_combinations(kb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While a few of those combinations look like data errors, most look natural and intuitive. Multiple relations per entity pair is a commonplace phenomenon.\n",
    "\n",
    "This observation strongly suggests formulating our prediction problem as [multi-label classification](https://en.wikipedia.org/wiki/Multi-label_classification). We could instead treat it as [multi-class classification](https://en.wikipedia.org/wiki/Multiclass_classification) — and indeed, [Mintz et al. 2009](https://www.aclweb.org/anthology/P09-1113) did so — but if we do, we'll be faced with the problem of assigning a single relation label to entity pairs which actually belong to multiple relations. It's not obvious how best to do this (and Mintz et al. 2009 did not make their method clear).\n",
    "\n",
    "There are a number of ways to approach multi-label classification, but the most obvious is the [binary relevance method](https://en.wikipedia.org/wiki/Multi-label_classification#Problem_transformation_methods), which just factors multi-label classification over _n_ labels into _n_ independent binary classification problems, one for each label. A disadvantage of this approach is that, by treating the binary classification problems as independent, it fails to exploit correlations between labels. But it has the great virtue of simplicity, and it will suffice for our purposes.\n",
    "\n",
    "So our problem will be to take as input an entity pair and a candidate relation (label), and to return a binary prediction as to whether the entity pair belongs to the relation. Since a KB triple is precisely a relation and a pair of entities, we could say equivalently that our prediction problem amounts to binary classification of KB triples. Given a candidate KB triple, do we predict that it is valid?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building datasets\n",
    "\n",
    "We're now in a position to write a function to build datasets suitable for training and evaluating predictive models. It will have the following characteristics:\n",
    "\n",
    "- Because we've formulated our problem as multi-label classification, and we'll be training separate models for each relation, we won't build a single dataset. Instead, we'll build a dataset for each relation, and our return value will be a map from relation names to datasets.\n",
    "- The dataset for each relation will consist of two parallel lists:\n",
    "  - A list of candidate `KBTriples` which combine the given relation with a pair of entities.\n",
    "  - A corresponding list of boolean labels indicating whether the given `KBTriple` belongs to the KB.\n",
    "- The dataset for each relation will include `KBTriples` derived from two sources:\n",
    "  - Positive instances will be drawn from the KB.\n",
    "  - Negative instances will be sampled from unrelated entity pairs, as described above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_datasets(corpus, kb, include_positive=True, sampling_rate=0.1, seed=1):\n",
    "    unrelated_pairs = find_unrelated_pairs(corpus, kb)\n",
    "    random.seed(seed)\n",
    "    unrelated_pairs = random.sample(unrelated_pairs, int(sampling_rate * len(unrelated_pairs)))\n",
    "    kbts_by_rel = defaultdict(list)\n",
    "    labels_by_rel = defaultdict(list)\n",
    "    for index, rel in enumerate(all_relations):\n",
    "        if include_positive:\n",
    "            for kbt in kb.get_triples_for_relation(rel):\n",
    "                kbts_by_rel[rel].append(kbt)\n",
    "                labels_by_rel[rel].append(True)\n",
    "        for sbj, obj in unrelated_pairs:\n",
    "            kbts_by_rel[rel].append(KBTriple(rel, sbj, obj))\n",
    "            labels_by_rel[rel].append(False)        \n",
    "    return kbts_by_rel, labels_by_rel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\[ [top](#Relation-extraction-using-distant-supervision) \\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "Before we start building models, let's set up a test harness that allows us to measure a model's performance. This may seem backwards, but it's analogous to the software engineering paradigm of [test-driven development](https://en.wikipedia.org/wiki/Test-driven_development): first, define success; then, pursue it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting the data\n",
    "\n",
    "Whenever building a model from data, it's good practice to partition the data into a multiple _splits_ — minimally, a training split on which to train the model, and a test split on which to evaluate it. In fact, we'll go a bit further, and define four splits:\n",
    "\n",
    "- __The `tiny` split (1%).__ It's often useful to carve out a tiny chunk of data to use in place of training or test data during development. Of course, any quantitative results obtained by evaluating on the `tiny` split are nearly meaningless, but because evaluations run extremely fast, using this split is a good way to flush out bugs during iterative cycles of code development.\n",
    "- __The `train` split (69%).__ We'll use the majority of our data for training models, both during development and at final evaluation. Experiments with the `train` split may take longer to run, but they'll have much greater statistical power.\n",
    "- __The `dev` split (15%).__ We'll use the `dev` split as test data for intermediate (formative) evaluations during development. During routine experiments, all evaluations should use the `dev` split.\n",
    "- __The `test` split (15%).__ We'll reserve the `test` split for our final (summative) evaluation at the conclusion of our work. Running evaluations on the `test` split before you are ready to conclude your work is methodologically unsound and intellectually dishonest!\n",
    "\n",
    "Splitting our data assets is somewhat more complicated than in many other NLP problems, because we have both a corpus and KB. In order to minimize leakage of information from training data into test data, we'd like to split both the corpus and the KB. And in order to maximize the value of a finite quantity of data, we'd like to align the corpus splits and KB splits as closely as possible. In an ideal world, each split would have its own hermetically-sealed universe of entities, the corpus for that split would contain only examples mentioning those entities, and the KB for that split would contain only triples involving those entities. However, that ideal is not quite achievable in practice. In order to get as close as possible, we'll follow this plan:\n",
    "\n",
    "- First, we'll split the set of entities which appear as the subject in some KB triple.\n",
    "- Then, we'll split the set of KB triples based on their subject entity.\n",
    "- Finally, we'll split the set of corpus examples.\n",
    "  - If the first entity in the example has already been assigned to a split, we'll assign the example to the same split.\n",
    "  - Alternatively, if the second entity has already been assigned to a split, we'll assign the example to the same split.\n",
    "  - Otherwise, we'll assign the example to a split randomly.\n",
    "  \n",
    "<!-- \\[ TODO: figure out whether we actually need to split the _corpus_ -- any lift from testing on train corpus? \\] -->\n",
    "\n",
    "Here's code to implement the splits. It's OK to skip past the details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_corpus_and_kb(\n",
    "        split_names=['tiny', 'train', 'dev', 'test'],\n",
    "        split_fracs=[0.01, 0.69, 0.15, 0.15],   \n",
    "        seed=1):\n",
    "    if len(split_names) != len(split_fracs):\n",
    "        raise ValueError('split_names and split_fracs must be of equal length')\n",
    "    if sum(split_fracs) != 1.0:\n",
    "        raise ValueError('split_fracs must sum to 1')\n",
    "    n = len(split_fracs) # for convenience only\n",
    "    \n",
    "    def split_list(xs):\n",
    "        xs = sorted(xs) # sorted for reproducibility\n",
    "        if seed:\n",
    "            random.seed(seed)\n",
    "        random.shuffle(xs)\n",
    "        split_points = [0] + [int(round(frac * len(xs))) for frac in np.cumsum(split_fracs)]\n",
    "        return [xs[split_points[i]:split_points[i + 1]] for i in range(n)]\n",
    "    \n",
    "    # first, split the entities that appear as subjects in the KB\n",
    "    sbjs = list(set([kbt.sbj for kbt in kb.get_triples()]))\n",
    "    sbj_splits = split_list(sbjs)\n",
    "    sbj_split_dict = dict([(sbj, i) for i, split in enumerate(sbj_splits) for sbj in split])\n",
    "    \n",
    "    # next, split the KB triples based on their subjects\n",
    "    kbt_splits = [[kbt for kbt in kb.get_triples() if sbj_split_dict[kbt.sbj] == i] for i in range(n)]\n",
    "    \n",
    "    # now split examples based on the entities they contain\n",
    "    ex_splits = [[] for i in range(n + 1)] # include an extra split\n",
    "    for ex in corpus.get_examples():\n",
    "        if ex.entity_1 in sbj_split_dict:\n",
    "            # if entity_1 is a sbj in the KB, assign example to split of that sbj\n",
    "            ex_splits[sbj_split_dict[ex.entity_1]].append(ex)\n",
    "        elif ex.entity_2 in sbj_split_dict:\n",
    "            # if entity_2 is a sbj in the KB, assign example to split of that sbj\n",
    "            ex_splits[sbj_split_dict[ex.entity_2]].append(ex)\n",
    "        else:\n",
    "            # otherwise, put in extra split to be redistributed\n",
    "            ex_splits[-1].append(ex)\n",
    "    # reallocate the examples that weren't assigned to a split on first pass\n",
    "    extra_ex_splits = split_list(ex_splits[-1])\n",
    "    ex_splits = [ex_splits[i] + extra_ex_splits[i] for i in range(n)]\n",
    "    \n",
    "    # create a Corpus and a KB for each split\n",
    "    data = {}\n",
    "    for i in range(n):\n",
    "        data[split_names[i]] = {'corpus': Corpus(ex_splits[i]), 'kb': KB(kbt_splits[i])}\n",
    "    data['all'] = {'corpus': corpus, 'kb': kb}\n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great. Let's use it to create the splits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'tiny': {'corpus': Corpus with 3458 examples, 'kb': KB with 425 triples},\n",
       " 'train': {'corpus': Corpus with 290042 examples, 'kb': KB with 40158 triples},\n",
       " 'dev': {'corpus': Corpus with 57241 examples, 'kb': KB with 7939 triples},\n",
       " 'test': {'corpus': Corpus with 63382 examples, 'kb': KB with 8053 triples},\n",
       " 'all': {'corpus': Corpus with 414123 examples, 'kb': KB with 56575 triples}}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = split_corpus_and_kb(seed=1)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now we can use `data['train']['corpus']` to refer to the training corpus, or `data['dev']['kb']` to refer to the dev KB.\n",
    "\n",
    "As a convenience, let's add a function for creating datasets for a specific split:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_datasets_for_split(split, include_positive=True, sampling_rate=0.1, seed=1):\n",
    "    return build_datasets(data[split]['corpus'], data[split]['kb'], include_positive, sampling_rate, seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choosing evaluation metrics\n",
    "\n",
    "Because we've formulated our prediction problem as a family of binary classification problems, one for each relation (label), choosing evaluation metrics is pretty straightforward. The standard metrics for evaluating binary classification are [precision and recall](https://en.wikipedia.org/wiki/Precision_and_recall), which are more meaningful than simple accuracy, particularly in problems with a highly biased label distribution (like ours). We'll compute and report precision and recall separately for each relation (label). There are only two wrinkles:\n",
    "\n",
    "0. __How best to combine precision and recall into a single metric.__ Having two evaluation metrics is often inconvenient. If we're considering a change to our model which improves precision but degrades recall, should we take it? To drive an iterative development process, it's useful to have a single metric on which to hill-climb. For binary classification, the standard answer is the [F<sub>1</sub>-score](https://en.wikipedia.org/wiki/F1_score), which is the harmonic mean of precision and recall. However, the F<sub>1</sub>-score gives equal weight to precision and recall. For our purposes, precision is probably more important than recall. If we're extracting new relation triples from (massively abundant) text on the web in order to augment a knowledge base, it's probably more important that the triples we extract are correct (precision) than that we extract all the triples we could (recall). Accordingly, instead of the F<sub>1</sub>-score, we'll use the F<sub>0.5</sub>-score, which gives precision twice as much weight as recall.\n",
    "\n",
    "0. __How to aggregate metrics across relations (labels).__ Reporting metrics separately for each relation is great, but in order to drive iterative development, we'd also like to have summary metrics which aggregate across all relations. There are two possible ways to do it: _micro-averaging_ will give equal weight to all problem instances, and thus give greater weight to relations with more instances, while _macro-averaging_ will give equal weight to all relations, and thus give lesser weight to problem instances in relations with more instances. Because the number of problem instances per relation is, to some degree, an accident of our data collection methodology, we'll choose macro-averaging.\n",
    "\n",
    "Thus, while every evaluation will report lots of metrics, when we need a single metric on which to hill-climb, it will be the macro-averaged F<sub>0.5</sub>-score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running evaluations\n",
    "\n",
    "It's time to write some code to run evaluations and report results. This is now straightforward. The `evaluate()` function takes as inputs:\n",
    "\n",
    "- `classifier`, which is just a function that takes a list of `KBTriples` and returns a list of boolean predictions;\n",
    "- `test_split`, the split on which to evaluate the classifier, `dev` by default;\n",
    "- `verbose`, a boolean indicating whether to print output.\n",
    "\n",
    "The other functions below are just helper functions to `evaluate()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_statistics_header():\n",
    "    print('{:20s} {:>10s} {:>10s} {:>10s} {:>10s} {:>10s}'.format(\n",
    "        'relation', 'precision', 'recall', 'f-score', 'support', 'size'))\n",
    "    print('{:20s} {:>10s} {:>10s} {:>10s} {:>10s} {:>10s}'.format(\n",
    "        '-' * 18, '-' * 9, '-' * 9, '-' * 9, '-' * 9, '-' * 9))\n",
    "\n",
    "def print_statistics_row(rel, result):\n",
    "    print('{:20s} {:10.3f} {:10.3f} {:10.3f} {:10d} {:10d}'.format(rel, *result))\n",
    "\n",
    "def print_statistics_footer(avg_result):\n",
    "    print('{:20s} {:>10s} {:>10s} {:>10s} {:>10s} {:>10s}'.format(\n",
    "        '-' * 18, '-' * 9, '-' * 9, '-' * 9, '-' * 9, '-' * 9))\n",
    "    print('{:20s} {:10.3f} {:10.3f} {:10.3f} {:10d} {:10d}'.format('macro-average', *avg_result))\n",
    "\n",
    "def macro_average_results(results):\n",
    "    avg_result = [np.average([r[i] for r in results.values()]) for i in range(3)]\n",
    "    avg_result.append(np.sum([r[3] for r in results.values()]))\n",
    "    avg_result.append(np.sum([r[4] for r in results.values()]))\n",
    "    return avg_result\n",
    "    \n",
    "def evaluate(classifier, test_split='dev', verbose=True):\n",
    "    test_kbts_by_rel, true_labels_by_rel = build_datasets_for_split(test_split)\n",
    "    results = {}\n",
    "    if verbose:\n",
    "        print_statistics_header()\n",
    "    for rel in all_relations:\n",
    "        pred_labels = classifier(test_kbts_by_rel[rel])\n",
    "        stats = precision_recall_fscore_support(true_labels_by_rel[rel], pred_labels, beta=0.5)\n",
    "        stats = [stat[1] for stat in stats]  # stats[1] is the stat for label True\n",
    "        stats.append(len(pred_labels)) # number of examples\n",
    "        results[rel] = stats\n",
    "        if verbose:\n",
    "            print_statistics_row(rel, results[rel])\n",
    "    avg_result = macro_average_results(results)\n",
    "    if verbose:\n",
    "        print_statistics_footer(avg_result)\n",
    "    return avg_result[2]  # return f_0.5 score as summary statistic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating a random-guessing strategy\n",
    "\n",
    "In order to validate our evaluation framework, and to set a floor under expected results for future evaluations, let's implement and evaluate a random-guessing strategy. The random guesser is a classifier which completely ignores its input, and simply flips a coin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lift(f):\n",
    "    return lambda xs: [f(x) for x in xs]\n",
    "\n",
    "def make_random_classifier(p=0.50):\n",
    "    def random_classify(kb_triple):\n",
    "        return random.random() < p\n",
    "    return lift(random_classify)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   0.058      0.515      0.070        303       5319\n",
      "author                    0.088      0.508      0.106        480       5496\n",
      "capital                   0.019      0.539      0.024         89       5105\n",
      "contains                  0.349      0.502      0.371       2667       7683\n",
      "film_performance          0.138      0.491      0.162        822       5838\n",
      "founders                  0.064      0.482      0.078        359       5375\n",
      "genre                     0.039      0.608      0.048        166       5182\n",
      "has_sibling               0.092      0.493      0.109        513       5529\n",
      "has_spouse                0.110      0.530      0.130        575       5591\n",
      "is_a                      0.099      0.547      0.119        494       5510\n",
      "nationality               0.054      0.463      0.066        311       5327\n",
      "parents                   0.062      0.502      0.075        325       5341\n",
      "place_of_birth            0.040      0.488      0.049        217       5233\n",
      "place_of_death            0.028      0.490      0.034        145       5161\n",
      "profession                0.052      0.563      0.063        245       5261\n",
      "worked_at                 0.047      0.544      0.058        228       5244\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.084      0.517      0.098       7939      88195\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.09757501010273492"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(make_random_classifier())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results are not too surprising. Recall is generally around 0.50, which makes sense: on any given example with label `True`, we are 50% likely to guess the right label. But precision is very poor, because most labels are not `True`, and because our classifier is completely ignorant of the features of specific problem instances. Accordingly, the F<sub>0.5</sub>-score is also very poor — first because even the equally-weighted F<sub>1</sub>-score is always closer to the lesser of precision and recall, and second because the F<sub>0.5</sub>-score weights precision twice as much as recall.\n",
    "\n",
    "Actually, the most remarkable result in this table is the comparatively good performance for the `contains` relation! What does this result tell us about the data?\n",
    "\n",
    "\\[ [top](#Relation-extraction-using-distant-supervision) \\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A simple baseline model\n",
    "\n",
    "It shouldn't be too hard to do better than random guessing. But for now, let's aim low — let's use the data we have in the easiest and most obvious way, and see how far that gets us.\n",
    "\n",
    "We start from the intuition that the words between two entity mentions frequently tell us how they're related. For example, in the phrase \"SpaceX was founded by Elon Musk\", the words \"was founded by\" indicate that the `founders` relation holds between the first entity mentioned and the second. Likewise, in the phrase \"Elon Musk established SpaceX\", the word \"established\" indicates the `founders` relation holds between the second entity mentioned and the first.\n",
    "\n",
    "So let's write some code to find the most common phrases that appear between the two entity mentions for each relation. As the examples illustrate, we need to make sure to consider both directions: that is, where the subject of the relation appears as the first mention, and where it appears as the second."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "adjoins              fwd         8461 ,\n",
      "adjoins              fwd         5633 and\n",
      "adjoins              fwd          993 , and\n",
      "adjoins              fwd         5599 ,\n",
      "adjoins              fwd         3780 and\n",
      "adjoins              fwd          680 , and\n",
      "author               fwd         1214 by\n",
      "author               fwd          155 ,\n",
      "author               fwd          130 , by\n",
      "author               fwd         1106 's\n",
      "author               fwd          294 ‘ s\n",
      "author               fwd          175 ’ s\n",
      "capital              fwd           37 ,\n",
      "capital              fwd           19 in\n",
      "capital              fwd           18 (\n",
      "capital              fwd         3711 ,\n",
      "capital              fwd          178 in\n",
      "capital              fwd           87 , the capital of\n",
      "contains             fwd          460 's\n",
      "contains             fwd          355 ,\n",
      "contains             fwd          250 (\n",
      "contains             fwd        25095 ,\n",
      "contains             fwd         5603 in\n",
      "contains             fwd          668 in the\n",
      "film_performance     fwd          286 in\n",
      "film_performance     fwd          200 's\n",
      "film_performance     fwd          115 film\n",
      "film_performance     fwd          213 with\n",
      "film_performance     fwd          152 , starring\n",
      "film_performance     fwd          115 opposite\n",
      "founders             fwd           98 founder\n",
      "founders             fwd           59 co-founder\n",
      "founders             fwd           57 ,\n",
      "founders             fwd          180 's\n",
      "founders             fwd          104 of\n",
      "founders             fwd           77 ‘ s\n",
      "genre                fwd           28 , a\n",
      "genre                fwd           13 in 1994 , he became a central figure in the\n",
      "genre                fwd           11 is a\n",
      "genre                fwd          122 ,\n",
      "genre                fwd           62 series\n",
      "genre                fwd           23 \n",
      "has_sibling          fwd         1369 and\n",
      "has_sibling          fwd          614 ,\n",
      "has_sibling          fwd          139 , and\n",
      "has_sibling          fwd          930 and\n",
      "has_sibling          fwd          460 ,\n",
      "has_sibling          fwd           94 , and\n",
      "has_spouse           fwd         2029 and\n",
      "has_spouse           fwd          375 ,\n",
      "has_spouse           fwd          112 and his wife\n",
      "has_spouse           fwd         1382 and\n",
      "has_spouse           fwd          271 ,\n",
      "has_spouse           fwd           78 and his wife\n",
      "is_a                 fwd          120 ,\n",
      "is_a                 fwd           77 and\n",
      "is_a                 fwd           34 , a\n",
      "is_a                 fwd          252 ,\n",
      "is_a                 fwd          175 and\n",
      "is_a                 fwd           81 \n",
      "nationality          fwd          331 of\n",
      "nationality          fwd           79 in\n",
      "nationality          fwd           34 of the\n",
      "nationality          fwd           57 ,\n",
      "nationality          fwd           27 by\n",
      "nationality          fwd           25 under\n",
      "parents              fwd           77 , son of\n",
      "parents              fwd           50 and\n",
      "parents              fwd           44 ,\n",
      "parents              fwd          177 and\n",
      "parents              fwd          167 ,\n",
      "parents              fwd           47 and his son\n",
      "place_of_birth       fwd           90 of\n",
      "place_of_birth       fwd           64 was born in\n",
      "place_of_birth       fwd           37 in\n",
      "place_of_birth       fwd           17 ,\n",
      "place_of_birth       fwd           16 by\n",
      "place_of_birth       fwd           11 under\n",
      "place_of_death       fwd           73 in\n",
      "place_of_death       fwd           63 of\n",
      "place_of_death       fwd           17 at\n",
      "place_of_death       fwd           12 ,\n",
      "place_of_death       fwd           10 under\n",
      "place_of_death       fwd            9 mayor\n",
      "profession           fwd           65 ,\n",
      "profession           fwd           27 , a\n",
      "profession           fwd           20 and\n",
      "profession           fwd          114 ,\n",
      "profession           fwd           74 \n",
      "profession           fwd           24 and\n",
      "worked_at            fwd          103 of\n",
      "worked_at            fwd           82 at\n",
      "worked_at            fwd           66 's\n",
      "worked_at            fwd           37 ,\n",
      "worked_at            fwd           30 founder\n",
      "worked_at            fwd           25 co-founder\n"
     ]
    }
   ],
   "source": [
    "def find_common_middles(split='train', top_k=3, show_output=False):\n",
    "    corpus = data[split]['corpus']\n",
    "    kb = data[split]['kb']\n",
    "    mids_by_rel = {\n",
    "        'fwd': defaultdict(lambda: defaultdict(int)),\n",
    "        'rev': defaultdict(lambda: defaultdict(int)),\n",
    "    }\n",
    "    for rel in all_relations:\n",
    "        for kbt in kb.get_triples_for_relation(rel):\n",
    "            for ex in corpus.get_examples_for_entities(kbt.sbj, kbt.obj):\n",
    "                mids_by_rel['fwd'][rel][ex.middle] += 1\n",
    "            for ex in corpus.get_examples_for_entities(kbt.obj, kbt.sbj):\n",
    "                mids_by_rel['rev'][rel][ex.middle] += 1\n",
    "    def most_frequent(mid_counter):\n",
    "        return sorted([(cnt, mid) for mid, cnt in mid_counter.items()], reverse=True)[:top_k]\n",
    "    for rel in all_relations:\n",
    "        for dir in ['fwd', 'rev']:\n",
    "            top = most_frequent(mids_by_rel[dir][rel])\n",
    "            if show_output:\n",
    "                for cnt, mid in top:\n",
    "                    print('{:20s} {:5s} {:10d} {:s}'.format(rel, 'fwd', cnt, mid))\n",
    "            mids_by_rel[dir][rel] = set([mid for cnt, mid in top])\n",
    "    return mids_by_rel\n",
    "\n",
    "_ = find_common_middles(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few observations here:\n",
    "\n",
    "- Some of the most frequent middles are natural and intuitive. For example, \", son of\" indicates a forward `parents` relation, while \"and his son\" indicates a reverse `parents` relation.\n",
    "- Punctuation and stop words such as \"and\" and \"of\" are extremely common. Unlike some other NLP applications, it's probably a bad idea to throw these away — they carry lots of useful information.\n",
    "- However, punctuation and stop words tend to be highly ambiguous. For example, a bare comma is a likely middle for almost every relation in at least one direction.\n",
    "- A few of the results reflect quirks of the dataset. For example, the appearance of the phrase \"in 1994 , he became a central figure in the\" as a common middle for the `genre` relation reflects both the relative scarcity of examples for that relation, and an unfortunate tendency of the Wikilinks dataset to include duplicate or near-duplicate source documents. (That middle connects the entities [Ready to Die](https://en.wikipedia.org/wiki/Ready_to_Die) — the first studio album by the Notorious B.I.G. — and [East Coast hip hop](https://en.wikipedia.org/wiki/East_Coast_hip_hop).)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_top_k_middles_classifier(train_split='train', top_k=3):\n",
    "    corpus = data[train_split]['corpus']\n",
    "    top_k_mids_by_rel = find_common_middles(split=train_split, top_k=top_k)\n",
    "    def classify(kb_triple):\n",
    "        fwd_mids = top_k_mids_by_rel['fwd'][kb_triple.rel]\n",
    "        rev_mids = top_k_mids_by_rel['rev'][kb_triple.rel]\n",
    "        for ex in corpus.get_examples_for_entities(kb_triple.sbj, kb_triple.obj):\n",
    "            if ex.middle in fwd_mids:\n",
    "                return True\n",
    "        for ex in corpus.get_examples_for_entities(kb_triple.obj, kb_triple.sbj):\n",
    "            if ex.middle in rev_mids:\n",
    "                return True\n",
    "        return False\n",
    "    return lift(classify)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   0.339      0.406      0.350        303       5319\n",
      "author                    0.246      0.058      0.150        480       5496\n",
      "capital                   0.094      0.191      0.105         89       5105\n",
      "contains                  0.522      0.066      0.218       2667       7683\n",
      "film_performance          0.222      0.002      0.012        822       5838\n",
      "founders                  0.200      0.061      0.138        359       5375\n",
      "genre                     0.000      0.000      0.000        166       5182\n",
      "has_sibling               0.322      0.222      0.295        513       5529\n",
      "has_spouse                0.385      0.249      0.347        575       5591\n",
      "is_a                      0.021      0.010      0.018        494       5510\n",
      "nationality               0.108      0.039      0.079        311       5327\n",
      "parents                   0.088      0.068      0.083        325       5341\n",
      "place_of_birth            0.049      0.023      0.040        217       5233\n",
      "place_of_death            0.011      0.007      0.010        145       5161\n",
      "profession                0.009      0.008      0.009        245       5261\n",
      "worked_at                 0.072      0.031      0.057        228       5244\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.168      0.090      0.119       7939      88195\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.11941265374393456"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(train_top_k_middles_classifier())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not surprisingly, the performance of even this extremely simplistic model is noticeably better than random guessing. Of course, recall is much worse across the board, but precision and F<sub>0.5</sub>-score are sometimes much better. We observe big gains especially on `adjoins`, `author`, `contains`, `founders`, `has_sibling`, and `has spouse`. Then again, at least one relation actually got worse. (Can you offer any explanation for that?)\n",
    "\n",
    "Admittedly, performance is still not great in absolute terms. However, we should have modest expectations for performance on this task — we are unlikely ever to get anywhere near perfect precision with perfect recall. Why?\n",
    "\n",
    "- High precision will be hard to achieve because the KB is incomplete: some entity pairs that are related in the world — and in the corpus — may simply be missing from the KB.\n",
    "- High recall will be hard to achieve because the corpus is finite: some entity pairs that are related in the KB may not have any examples in the corpus.\n",
    "\n",
    "Because of these unavoidable obstacles, what matters is not so much absolute performance, but relative performance of different approaches.\n",
    "\n",
    "__Exercise:__ What's the optimal value for `top_k`, the number of most frequent middles to consider? What choice maximizes our chosen figure of merit, the macro-averaged F<sub>0.5</sub>-score?\n",
    "\n",
    "\\[ [top](#Relation-extraction-using-distant-supervision) \\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a classifier\n",
    "\n",
    "OK, it's time to get (halfway) serious. Let's apply real machine learning to train a classifier on the training data, and see how it performs on the test data. We'll begin with one of the simplest machine learning setups: a bag-of-words feature representation, and a linear model trained using logistic regression.\n",
    "\n",
    "Just like we did in the unit on [supervised sentiment analysis](https://github.com/cgpotts/cs224u/blob/master/sst_02_hand_built_features.ipynb), we'll leverage the `sklearn` library, and we'll introduce functions for featurizing instances, training models, making predictions, and evaluating results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Featurizers\n",
    "\n",
    "Featurizers are functions which define the feature representation for our model. The primary input to a featurizer will be the `KBTriple` for which we are generating features. But since our features will be derived from corpus examples containing the entities of the `KBTriple`, we must also pass in a reference to a `Corpus`. And in order to make it easy to combine different featurizers, we'll also pass in a feature counter to hold the results.\n",
    "\n",
    "Here's an implementation for a very simple bag-of-words featurizer. It finds all the corpus examples containing the two entities in the `KBTriple`, breaks the phrase appearing between the two entity mentions into words, and counts the words. Note that it makes no distinction between \"forward\" and \"reverse\" examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simple_bag_of_words_featurizer(kbt, corpus, feature_counter):\n",
    "    for ex in corpus.get_examples_for_entities(kbt.sbj, kbt.obj):\n",
    "        for word in ex.middle.split(' '):\n",
    "            feature_counter[word] += 1\n",
    "    for ex in corpus.get_examples_for_entities(kbt.obj, kbt.sbj):\n",
    "        for word in ex.middle.split(' '):\n",
    "            feature_counter[word] += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can experiment with adding new kinds of features just by implementing additional featurizers, following `simple_bag_of_words_featurizer` as an example.\n",
    "\n",
    "Now, in order to apply machine learning algorithms such as those provided by `sklearn`, we need a way to convert datasets of `KBTriple`s into feature matrices. The function `featurize_datasets()` achieves that. It takes in a collection of `KBTriple`s grouped by relation, and returns a corresponding collection of feature matrices grouped by relation. It also needs a `Corpus` from which to extract features, and a list of featurizers to generate the features. Finally, it accepts a vectorizer as an optional argument. At training time, we won't supply a vectorizer, so this code will create a new one; at test time, we'll supply the vectorizer we created at training time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def featurize_datasets(\n",
    "        kbts_by_rel,\n",
    "        corpus,\n",
    "        featurizers=[simple_bag_of_words_featurizer],\n",
    "        vectorizer=None):\n",
    "    # Create feature counters for all instances (kbts).\n",
    "    feat_counters_by_rel = defaultdict(list)\n",
    "    for rel, kbts in kbts_by_rel.items():\n",
    "        for kbt in kbts:\n",
    "            feature_counter = Counter()\n",
    "            for featurizer in featurizers:\n",
    "                featurizer(kbt, corpus, feature_counter)\n",
    "            feat_counters_by_rel[rel].append(feature_counter)\n",
    "    feat_matrices_by_rel = defaultdict(list)\n",
    "    # If we haven't been given a Vectorizer, create one and fit it to all the feature counters.\n",
    "    if vectorizer == None:\n",
    "        vectorizer = DictVectorizer(sparse=True)\n",
    "        def traverse_dicts():\n",
    "            for dict_list in feat_counters_by_rel.values():\n",
    "                for d in dict_list:\n",
    "                    yield d\n",
    "        vectorizer.fit(traverse_dicts())\n",
    "    # Now use the Vectorizer to transform feature dictionaries into feature matrices.\n",
    "    for rel, feat_counters in feat_counters_by_rel.items():\n",
    "        feat_matrices_by_rel[rel] = vectorizer.transform(feat_counters)\n",
    "    return feat_matrices_by_rel, vectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experiments\n",
    "\n",
    "Now we need some functions to train models, make predictions, and evaluate the results. We'll start with `train_models()`. This function takes as arguments a data split on which to train, a list of featurizers, and model factory, which is a function which initializes an `sklearn` classifier. It returns a dictionary holding the featurizers, the vectorizer that was used to generate the training matrix, and a dictionary holding the trained models, one per relation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_models(\n",
    "        split='train',\n",
    "        featurizers=[simple_bag_of_words_featurizer],\n",
    "        model_factory=lambda: LogisticRegression(fit_intercept=True),\n",
    "        verbose=True):\n",
    "    if verbose: print('Building datasets')\n",
    "    train_o, train_y = build_datasets_for_split(split=split)\n",
    "    if verbose: print('Featurizing')\n",
    "    train_X, vectorizer = featurize_datasets(train_o, data[split]['corpus'], featurizers)\n",
    "    models = {}\n",
    "    if verbose: print('Training models')\n",
    "    for rel in all_relations:\n",
    "        models[rel] = model_factory()\n",
    "        models[rel].fit(train_X[rel], train_y[rel])\n",
    "    if verbose: print('Training complete\\n')\n",
    "    return {\n",
    "        'featurizers': featurizers,\n",
    "        'vectorizer': vectorizer,\n",
    "        'models': models,\n",
    "    }        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next comes `predict()`. This function takes as arguments a test split, a list of featurizers, the vectorizer that was used during training, and a dictionary holding the models, one per relation. It returns two parallel dictionaries: one holding the predictions (grouped by relation), the other holding the true labels (again, grouped by prediction)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(split, featurizers, vectorizer, models):\n",
    "    test_o, test_y = build_datasets_for_split(split=split)\n",
    "    test_X, _ = featurize_datasets(test_o, data[split]['corpus'], featurizers, vectorizer=vectorizer)\n",
    "    predictions = {}\n",
    "    for rel in all_relations:\n",
    "        predictions[rel] = models[rel].predict(test_X[rel])\n",
    "    return predictions, test_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now `evaluate_predictions()`. This function takes as arguments the parallel dictionaries of predictions and true labels produced by `predict()`. It prints summary statistics for each relation, including precision, recall, and F<sub>0.5</sub>-score, and it returns the macro-averaged F<sub>0.5</sub>-score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_predictions(predictions, test_y, verbose=True):\n",
    "    results = {}  # one result row for each relation\n",
    "    if verbose:\n",
    "        print_statistics_header()\n",
    "    for rel in all_relations:\n",
    "        stats = precision_recall_fscore_support(test_y[rel], predictions[rel], beta=0.5)\n",
    "        stats = [stat[1] for stat in stats]  # stats[1] is the stat for label True\n",
    "        stats.append(len(test_y[rel]))\n",
    "        results[rel] = stats\n",
    "        if verbose:\n",
    "            print_statistics_row(rel, results[rel])\n",
    "    avg_result = macro_average_results(results)\n",
    "    if verbose:\n",
    "        print_statistics_footer(avg_result)\n",
    "    return avg_result[2]  # return f_0.5 score as summary statistic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we introduce `experiment()`, which simply chains together `train_models()`, `predict()`, and `evaluate_predictions()`. For convenience, this function returns the output of `train_models()` as its result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def experiment(\n",
    "        train_split='train',\n",
    "        test_split='dev',\n",
    "        featurizers=[simple_bag_of_words_featurizer],\n",
    "        model_factory=lambda: LogisticRegression(fit_intercept=True),\n",
    "        verbose=True):\n",
    "    train_result = train_models(train_split, featurizers, model_factory, verbose)\n",
    "    predictions, test_y = predict(test_split,\n",
    "                                  featurizers,\n",
    "                                  train_result['vectorizer'],\n",
    "                                  train_result['models'])\n",
    "    evaluate_predictions(predictions, test_y, verbose)\n",
    "    return train_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running `experiment()` in its default configuration will give us a baseline result for machine-learned models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building datasets\n",
      "Featurizing\n",
      "Training models\n",
      "Training complete\n",
      "\n",
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   0.873      0.452      0.736        303       5319\n",
      "author                    0.790      0.550      0.727        480       5496\n",
      "capital                   0.575      0.258      0.462         89       5105\n",
      "contains                  0.744      0.609      0.712       2667       7683\n",
      "film_performance          0.812      0.572      0.749        822       5838\n",
      "founders                  0.731      0.379      0.617        359       5375\n",
      "genre                     0.591      0.235      0.453        166       5182\n",
      "has_sibling               0.847      0.238      0.560        513       5529\n",
      "has_spouse                0.916      0.358      0.698        575       5591\n",
      "is_a                      0.706      0.243      0.511        494       5510\n",
      "nationality               0.500      0.145      0.335        311       5327\n",
      "parents                   0.860      0.585      0.786        325       5341\n",
      "place_of_birth            0.639      0.212      0.455        217       5233\n",
      "place_of_death            0.447      0.117      0.286        145       5161\n",
      "profession                0.629      0.159      0.396        245       5261\n",
      "worked_at                 0.682      0.254      0.511        228       5244\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.709      0.335      0.562       7939      88195\n"
     ]
    }
   ],
   "source": [
    "_ = experiment()\n",
    "# _ = experiment(train_split='tiny', test_split='tiny') # better for rapid development"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Considering how vanilla our model is, these results are quite surprisingly good! We see huge gains for every relation over our `top_k_middles_classifier`. This strong performance is a powerful testament to the effectiveness of even the simplest forms of machine learning.\n",
    "\n",
    "But there is still much more we can do. To make further gains, we must not treat the model as a black box. We must open it up and get visibility into what it has learned, and more importantly, where it still falls down."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examining the trained models\n",
    "\n",
    "One important way to gain understanding of our trained model is to inspect the model weights. What features are strong positive indicators for each relation, and what features are strong negative indicators?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def examine_model_weights(\n",
    "        train_split='train',\n",
    "        featurizers=[simple_bag_of_words_featurizer],\n",
    "        model_factory=lambda: LogisticRegression(fit_intercept=True),\n",
    "        k=3,\n",
    "        verbose=True):\n",
    "    train_result = train_models(train_split, featurizers, model_factory, verbose)\n",
    "    feature_names = train_result['vectorizer'].get_feature_names()\n",
    "    for rel, model in train_result['models'].items():\n",
    "        print('Highest and lowest feature weights for relation {}:\\n'.format(rel))\n",
    "        sorted_weights = sorted([(wgt, idx) for idx, wgt in enumerate(model.coef_[0])], reverse=True)\n",
    "        for wgt, idx in sorted_weights[:k]:\n",
    "            print('{:10.3f} {}'.format(wgt, feature_names[idx]))\n",
    "        print('{:>10s} {}'.format('.....', '.....'))\n",
    "        for wgt, idx in sorted_weights[-k:]:\n",
    "            print('{:10.3f} {}'.format(wgt, feature_names[idx]))\n",
    "        print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building datasets\n",
      "Featurizing\n",
      "Training models\n",
      "Training complete\n",
      "\n",
      "Highest and lowest feature weights for relation adjoins:\n",
      "\n",
      "     2.456 Córdoba\n",
      "     2.220 nearby\n",
      "     2.170 Taluks\n",
      "     ..... .....\n",
      "    -1.230 crest\n",
      "    -1.268 for\n",
      "    -1.651 ``\n",
      "\n",
      "Highest and lowest feature weights for relation author:\n",
      "\n",
      "     2.865 book\n",
      "     2.786 poem\n",
      "     2.754 author\n",
      "     ..... .....\n",
      "    -2.428 or\n",
      "    -2.995 1818\n",
      "    -4.542 1945\n",
      "\n",
      "Highest and lowest feature weights for relation capital:\n",
      "\n",
      "     3.970 capital\n",
      "     1.894 km\n",
      "     1.682 posted\n",
      "     ..... .....\n",
      "    -1.494 German\n",
      "    -1.554 occupation\n",
      "    -1.687 during\n",
      "\n",
      "Highest and lowest feature weights for relation contains:\n",
      "\n",
      "     2.197 Cyprus\n",
      "     2.083 Channel\n",
      "     1.983 attended\n",
      "     ..... .....\n",
      "    -2.242 Taipei\n",
      "    -2.287 who\n",
      "    -3.995 occupation\n",
      "\n",
      "Highest and lowest feature weights for relation film_performance:\n",
      "\n",
      "     4.185 starring\n",
      "     3.782 alongside\n",
      "     3.426 co-starring\n",
      "     ..... .....\n",
      "    -1.945 or\n",
      "    -2.287 Anjaani\n",
      "    -2.287 Anjaana\n",
      "\n",
      "Highest and lowest feature weights for relation founders:\n",
      "\n",
      "     4.123 founder\n",
      "     3.881 founded\n",
      "     3.288 co-founder\n",
      "     ..... .....\n",
      "    -1.644 ousted\n",
      "    -1.684 GM\n",
      "    -1.954 novel\n",
      "\n",
      "Highest and lowest feature weights for relation genre:\n",
      "\n",
      "     3.368 series\n",
      "     2.847 \n",
      "     2.759 movie\n",
      "     ..... .....\n",
      "    -1.454 and\n",
      "    -1.510 Channel\n",
      "    -1.763 at\n",
      "\n",
      "Highest and lowest feature weights for relation has_sibling:\n",
      "\n",
      "     5.191 brother\n",
      "     3.761 sister\n",
      "     3.163 Marlon\n",
      "     ..... .....\n",
      "    -1.461 engineer\n",
      "    -1.745 bollywood\n",
      "    -1.799 1862-1942\n",
      "\n",
      "Highest and lowest feature weights for relation has_spouse:\n",
      "\n",
      "     5.050 wife\n",
      "     4.237 husband\n",
      "     4.203 married\n",
      "     ..... .....\n",
      "    -1.664 engineer\n",
      "    -1.882 44\n",
      "    -1.966 Sir\n",
      "\n",
      "Highest and lowest feature weights for relation is_a:\n",
      "\n",
      "     4.790 stage\n",
      "     3.468 \n",
      "     2.878 family\n",
      "     ..... .....\n",
      "    -2.000 Pavo\n",
      "    -2.438 sunfish\n",
      "    -2.992 Latidae\n",
      "\n",
      "Highest and lowest feature weights for relation nationality:\n",
      "\n",
      "     2.490 born\n",
      "     1.867 president\n",
      "     1.859 wooed\n",
      "     ..... .....\n",
      "    -1.649 or\n",
      "    -1.699 foreign\n",
      "    -2.171 1961\n",
      "\n",
      "Highest and lowest feature weights for relation parents:\n",
      "\n",
      "     5.155 son\n",
      "     4.802 daughter\n",
      "     4.455 father\n",
      "     ..... .....\n",
      "    -1.836 bollywood\n",
      "    -1.965 1862-1942\n",
      "    -2.054 Thomas\n",
      "\n",
      "Highest and lowest feature weights for relation place_of_birth:\n",
      "\n",
      "     3.848 born\n",
      "     3.026 mayor\n",
      "     2.951 birthplace\n",
      "     ..... .....\n",
      "    -1.320 David\n",
      "    -1.349 and\n",
      "    -1.545 or\n",
      "\n",
      "Highest and lowest feature weights for relation place_of_death:\n",
      "\n",
      "     2.431 assassinated\n",
      "     2.081 where\n",
      "     2.071 executed\n",
      "     ..... .....\n",
      "    -1.127 and\n",
      "    -1.149 or\n",
      "    -1.179 ;\n",
      "\n",
      "Highest and lowest feature weights for relation profession:\n",
      "\n",
      "     3.776 \n",
      "     2.680 philosopher\n",
      "     2.598 American\n",
      "     ..... .....\n",
      "    -1.278 in\n",
      "    -1.292 on\n",
      "    -1.303 at\n",
      "\n",
      "Highest and lowest feature weights for relation worked_at:\n",
      "\n",
      "     3.585 professor\n",
      "     3.214 president\n",
      "     2.788 employee\n",
      "     ..... .....\n",
      "    -1.547 1961\n",
      "    -1.705 or\n",
      "    -2.359 ousted\n",
      "\n"
     ]
    }
   ],
   "source": [
    "examine_model_weights()\n",
    "# examine_model_weights(train_split='tiny') # better for rapid development"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By and large, the high-weight features for each relation are pretty intuitive — they are words that are used to express the relation in question. (The counter-intuitive results merit a bit of investigation!)\n",
    "\n",
    "The low-weight features (that is, features with large negative weights) may be a bit harder to understand. In some cases, however, they can be interpreted as features which indicate some _other_ relation which is anti-correlated with the target relation. (As an example, \"directed\" is a negative indicator for the `author` relation.)\n",
    "\n",
    "__Exercise:__ Investigate one of the counter-intuitive high-weight features. Find the training examples which caused the feature to be included. Given the training data, does it make sense that this feature is a good predictor for the target relation?\n",
    "\n",
    "<!--\n",
    "- SPOILER: Using `penalty='l1'` results in somewhat less intuitive feature weights, and about the same performance.\n",
    "- SPOILER: Using `penalty='l1', C=0.1` results in much more intuitive feature weights, but much worse performance.\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discovering new relation instances\n",
    "\n",
    "Another way to gain insight into our trained models is to use them to discover new relation instances that don't currently appear in the KB. In fact, this is the whole point of building a relation extraction system: to extend an existing KB (or build a new one) using knowledge extracted from natural language text at scale. Can the models we've trained do this effectively?\n",
    "\n",
    "Because the goal is to discover new relation instances which are _true_ but _absent from the KB_, we can't evalute this capability automatically. But we can generate candidate KB triples and manually evaluate them for correctness.\n",
    "\n",
    "To do this, we'll start from corpus examples containing pairs of entities which do not belong to any relation in the KB (earlier, we described these as \"negative examples\"). We'll then apply our trained models to each pair of entities, and sort the results by probability assigned by the model, in order to find the most likely new instances for each relation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_new_relation_instances(\n",
    "        train_split='train',\n",
    "        test_split='dev',\n",
    "        featurizers=[simple_bag_of_words_featurizer],\n",
    "        model_factory=lambda: LogisticRegression(fit_intercept=True),\n",
    "        k=10,\n",
    "        verbose=True):\n",
    "\n",
    "    # train models\n",
    "    train_result = train_models(train_split, featurizers, model_factory, verbose)\n",
    "\n",
    "    # build datasets for negative instances only\n",
    "    neg_o, neg_y = build_datasets_for_split(test_split, include_positive=False, sampling_rate=1.0)\n",
    "    neg_X, _ = featurize_datasets(neg_o,\n",
    "                                  data[test_split]['corpus'],\n",
    "                                  featurizers,\n",
    "                                  train_result['vectorizer'])\n",
    "\n",
    "    # report highest confidence predictions\n",
    "    for rel, model in train_result['models'].items():\n",
    "        print('Highest probability examples for relation {}:\\n'.format(rel))\n",
    "        probs = model.predict_proba(neg_X[rel])\n",
    "        probs = [prob[1] for prob in probs] # probability for class True\n",
    "        sorted_probs = sorted([(p, idx) for idx, p in enumerate(probs)], reverse=True)\n",
    "        for p, idx in sorted_probs[:k]:\n",
    "            print('{:10.3f} {}'.format(p, neg_o[rel][idx]))\n",
    "        print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building datasets\n",
      "Featurizing\n",
      "Training models\n",
      "Training complete\n",
      "\n",
      "Highest probability examples for relation adjoins:\n",
      "\n",
      "     1.000 KBTriple(rel='adjoins', sbj='Moon', obj='Sun')\n",
      "     1.000 KBTriple(rel='adjoins', sbj='Sun', obj='Moon')\n",
      "     1.000 KBTriple(rel='adjoins', sbj='Maharashtra', obj='India')\n",
      "     1.000 KBTriple(rel='adjoins', sbj='India', obj='Maharashtra')\n",
      "     1.000 KBTriple(rel='adjoins', sbj='Europe', obj='Great_Britain')\n",
      "     1.000 KBTriple(rel='adjoins', sbj='Great_Britain', obj='Europe')\n",
      "     1.000 KBTriple(rel='adjoins', sbj='Kashmir', obj='India')\n",
      "     1.000 KBTriple(rel='adjoins', sbj='India', obj='Kashmir')\n",
      "     1.000 KBTriple(rel='adjoins', sbj='Sarawak', obj='Malaysia')\n",
      "     1.000 KBTriple(rel='adjoins', sbj='Malaysia', obj='Sarawak')\n",
      "\n",
      "Highest probability examples for relation author:\n",
      "\n",
      "     1.000 KBTriple(rel='author', sbj='Aldous_Huxley', obj='The_Doors_of_Perception')\n",
      "     1.000 KBTriple(rel='author', sbj='Systema_Naturae', obj='Carl_Linnaeus')\n",
      "     1.000 KBTriple(rel='author', sbj='Carl_Linnaeus', obj='Systema_Naturae')\n",
      "     1.000 KBTriple(rel='author', sbj='The_Doors_of_Perception', obj='Aldous_Huxley')\n",
      "     1.000 KBTriple(rel='author', sbj='Charlie_and_the_Chocolate_Factory', obj='Roald_Dahl')\n",
      "     1.000 KBTriple(rel='author', sbj='Roald_Dahl', obj='Charlie_and_the_Chocolate_Factory')\n",
      "     1.000 KBTriple(rel='author', sbj='Psychological_Types', obj='Carl_Jung')\n",
      "     1.000 KBTriple(rel='author', sbj='Carl_Jung', obj='Psychological_Types')\n",
      "     1.000 KBTriple(rel='author', sbj='Neil_Gaiman', obj='American_Gods')\n",
      "     1.000 KBTriple(rel='author', sbj='American_Gods', obj='Neil_Gaiman')\n",
      "\n",
      "Highest probability examples for relation capital:\n",
      "\n",
      "     1.000 KBTriple(rel='capital', sbj='Rome', obj='Italy')\n",
      "     1.000 KBTriple(rel='capital', sbj='Italy', obj='Rome')\n",
      "     1.000 KBTriple(rel='capital', sbj='Chernobyl_Nuclear_Power_Plant', obj='Ukraine')\n",
      "     1.000 KBTriple(rel='capital', sbj='Ukraine', obj='Chernobyl_Nuclear_Power_Plant')\n",
      "     1.000 KBTriple(rel='capital', sbj='Maharashtra', obj='India')\n",
      "     1.000 KBTriple(rel='capital', sbj='India', obj='Maharashtra')\n",
      "     1.000 KBTriple(rel='capital', sbj='Blarney', obj='Republic_of_Ireland')\n",
      "     1.000 KBTriple(rel='capital', sbj='Republic_of_Ireland', obj='Blarney')\n",
      "     1.000 KBTriple(rel='capital', sbj='Johor', obj='Johor_Bahru')\n",
      "     1.000 KBTriple(rel='capital', sbj='Johor_Bahru', obj='Johor')\n",
      "\n",
      "Highest probability examples for relation contains:\n",
      "\n",
      "     1.000 KBTriple(rel='contains', sbj='Rome', obj='Italy')\n",
      "     1.000 KBTriple(rel='contains', sbj='India', obj='Uttar_Pradesh')\n",
      "     1.000 KBTriple(rel='contains', sbj='Uttar_Pradesh', obj='India')\n",
      "     1.000 KBTriple(rel='contains', sbj='Chernobyl_Nuclear_Power_Plant', obj='Ukraine')\n",
      "     1.000 KBTriple(rel='contains', sbj='Roman_Empire', obj='Rome')\n",
      "     1.000 KBTriple(rel='contains', sbj='Rome', obj='Roman_Empire')\n",
      "     1.000 KBTriple(rel='contains', sbj='Maharashtra', obj='India')\n",
      "     1.000 KBTriple(rel='contains', sbj='Isle_of_Wight', obj='Ryde')\n",
      "     1.000 KBTriple(rel='contains', sbj='Ukraine', obj='Chernobyl_Nuclear_Power_Plant')\n",
      "     1.000 KBTriple(rel='contains', sbj='Ryde', obj='Isle_of_Wight')\n",
      "\n",
      "Highest probability examples for relation film_performance:\n",
      "\n",
      "     1.000 KBTriple(rel='film_performance', sbj='Hong_Kong', obj='Shanghai_Noon')\n",
      "     1.000 KBTriple(rel='film_performance', sbj='Shanghai_Noon', obj='Hong_Kong')\n",
      "     1.000 KBTriple(rel='film_performance', sbj='Salman_Khan', obj='Tere_Naam')\n",
      "     1.000 KBTriple(rel='film_performance', sbj='Tere_Naam', obj='Salman_Khan')\n",
      "     1.000 KBTriple(rel='film_performance', sbj='The_Pink_Panther_2', obj='Harald_Zwart')\n",
      "     1.000 KBTriple(rel='film_performance', sbj='Harald_Zwart', obj='The_Pink_Panther_2')\n",
      "     1.000 KBTriple(rel='film_performance', sbj='Robin_Williams', obj='Francis_Ford_Coppola')\n",
      "     1.000 KBTriple(rel='film_performance', sbj='Francis_Ford_Coppola', obj='Robin_Williams')\n",
      "     1.000 KBTriple(rel='film_performance', sbj='Angelina_Jolie', obj='Gia')\n",
      "     1.000 KBTriple(rel='film_performance', sbj='Gia', obj='Angelina_Jolie')\n",
      "\n",
      "Highest probability examples for relation founders:\n",
      "\n",
      "     1.000 KBTriple(rel='founders', sbj='Church_of_Scientology', obj='L._Ron_Hubbard')\n",
      "     1.000 KBTriple(rel='founders', sbj='L._Ron_Hubbard', obj='Church_of_Scientology')\n",
      "     1.000 KBTriple(rel='founders', sbj='Ishmael', obj='Abraham')\n",
      "     1.000 KBTriple(rel='founders', sbj='Abraham', obj='Ishmael')\n",
      "     1.000 KBTriple(rel='founders', sbj='Adam_Weishaupt', obj='Illuminati')\n",
      "     1.000 KBTriple(rel='founders', sbj='Illuminati', obj='Adam_Weishaupt')\n",
      "     1.000 KBTriple(rel='founders', sbj='Insect', obj='Lepidoptera')\n",
      "     1.000 KBTriple(rel='founders', sbj='Lepidoptera', obj='Insect')\n",
      "     1.000 KBTriple(rel='founders', sbj='Austria', obj='Gaston_Glock')\n",
      "     1.000 KBTriple(rel='founders', sbj='Gaston_Glock', obj='Austria')\n",
      "\n",
      "Highest probability examples for relation genre:\n",
      "\n",
      "     1.000 KBTriple(rel='genre', sbj='Cartoon_Cartoons', obj=\"Dexter's_Laboratory\")\n",
      "     1.000 KBTriple(rel='genre', sbj=\"Dexter's_Laboratory\", obj='Cartoon_Cartoons')\n",
      "     1.000 KBTriple(rel='genre', sbj='Taylor_York', obj='All_We_Know_Is_Falling')\n",
      "     1.000 KBTriple(rel='genre', sbj='All_We_Know_Is_Falling', obj='Taylor_York')\n",
      "     1.000 KBTriple(rel='genre', sbj='Lanner_falcon', obj='Falcon')\n",
      "     1.000 KBTriple(rel='genre', sbj='Falcon', obj='Lanner_falcon')\n",
      "     0.999 KBTriple(rel='genre', sbj='Meg_Griffin', obj='Family_Guy')\n",
      "     0.999 KBTriple(rel='genre', sbj='Family_Guy', obj='Meg_Griffin')\n",
      "     0.978 KBTriple(rel='genre', sbj='Tattoo_artist', obj='Tattoo_artist')\n",
      "     0.977 KBTriple(rel='genre', sbj='The_Nude_Bomb', obj='Get_Smart')\n",
      "\n",
      "Highest probability examples for relation has_sibling:\n",
      "\n",
      "     1.000 KBTriple(rel='has_sibling', sbj='Ishmael', obj='Abraham')\n",
      "     1.000 KBTriple(rel='has_sibling', sbj='Abraham', obj='Ishmael')\n",
      "     1.000 KBTriple(rel='has_sibling', sbj='Larry_Page', obj='Sergey_Brin')\n",
      "     1.000 KBTriple(rel='has_sibling', sbj='Sergey_Brin', obj='Larry_Page')\n",
      "     1.000 KBTriple(rel='has_sibling', sbj='Abraham', obj='Isaac')\n",
      "     1.000 KBTriple(rel='has_sibling', sbj='Isaac', obj='Abraham')\n",
      "     1.000 KBTriple(rel='has_sibling', sbj='Friedrich_Engels', obj='Karl_Marx')\n",
      "     1.000 KBTriple(rel='has_sibling', sbj='Karl_Marx', obj='Friedrich_Engels')\n",
      "     1.000 KBTriple(rel='has_sibling', sbj='Janet_Leigh', obj='Jamie_Lee_Curtis')\n",
      "     1.000 KBTriple(rel='has_sibling', sbj='Jamie_Lee_Curtis', obj='Janet_Leigh')\n",
      "\n",
      "Highest probability examples for relation has_spouse:\n",
      "\n",
      "     1.000 KBTriple(rel='has_spouse', sbj='Larry_Page', obj='Sergey_Brin')\n",
      "     1.000 KBTriple(rel='has_spouse', sbj='Sergey_Brin', obj='Larry_Page')\n",
      "     1.000 KBTriple(rel='has_spouse', sbj='Anne_Boleyn', obj='Elizabeth_I_of_England')\n",
      "     1.000 KBTriple(rel='has_spouse', sbj='Elizabeth_I_of_England', obj='Anne_Boleyn')\n",
      "     1.000 KBTriple(rel='has_spouse', sbj='Ishmael', obj='Abraham')\n",
      "     1.000 KBTriple(rel='has_spouse', sbj='Abraham', obj='Ishmael')\n",
      "     1.000 KBTriple(rel='has_spouse', sbj='Friedrich_Engels', obj='Karl_Marx')\n",
      "     1.000 KBTriple(rel='has_spouse', sbj='Karl_Marx', obj='Friedrich_Engels')\n",
      "     1.000 KBTriple(rel='has_spouse', sbj='Denver', obj='Isidor_Straus')\n",
      "     1.000 KBTriple(rel='has_spouse', sbj='Isidor_Straus', obj='Denver')\n",
      "\n",
      "Highest probability examples for relation is_a:\n",
      "\n",
      "     1.000 KBTriple(rel='is_a', sbj='Bee', obj='Apidae')\n",
      "     1.000 KBTriple(rel='is_a', sbj='Apidae', obj='Bee')\n",
      "     1.000 KBTriple(rel='is_a', sbj='Insect', obj='Lepidoptera')\n",
      "     1.000 KBTriple(rel='is_a', sbj='Lepidoptera', obj='Insect')\n",
      "     1.000 KBTriple(rel='is_a', sbj='Corvidae', obj='Passerine')\n",
      "     1.000 KBTriple(rel='is_a', sbj='Passerine', obj='Corvidae')\n",
      "     1.000 KBTriple(rel='is_a', sbj='Hibiscus', obj='Malvaceae')\n",
      "     1.000 KBTriple(rel='is_a', sbj='Malvaceae', obj='Hibiscus')\n",
      "     1.000 KBTriple(rel='is_a', sbj='Malvaceae', obj='Okra')\n",
      "     1.000 KBTriple(rel='is_a', sbj='Okra', obj='Malvaceae')\n",
      "\n",
      "Highest probability examples for relation nationality:\n",
      "\n",
      "     1.000 KBTriple(rel='nationality', sbj='Matale_District', obj='Sri_Lanka')\n",
      "     1.000 KBTriple(rel='nationality', sbj='Sri_Lanka', obj='Matale_District')\n",
      "     1.000 KBTriple(rel='nationality', sbj='New_Zealand', obj='North_Island')\n",
      "     1.000 KBTriple(rel='nationality', sbj='North_Island', obj='New_Zealand')\n",
      "     1.000 KBTriple(rel='nationality', sbj='Systema_Naturae', obj='Carl_Linnaeus')\n",
      "     1.000 KBTriple(rel='nationality', sbj='Carl_Linnaeus', obj='Systema_Naturae')\n",
      "     1.000 KBTriple(rel='nationality', sbj='San_Francisco_Bay_Area', obj='California')\n",
      "     1.000 KBTriple(rel='nationality', sbj='California', obj='San_Francisco_Bay_Area')\n",
      "     0.999 KBTriple(rel='nationality', sbj='Illinois', obj='United_States_Senate')\n",
      "     0.999 KBTriple(rel='nationality', sbj='United_States_Senate', obj='Illinois')\n",
      "\n",
      "Highest probability examples for relation parents:\n",
      "\n",
      "     1.000 KBTriple(rel='parents', sbj='Ishmael', obj='Abraham')\n",
      "     1.000 KBTriple(rel='parents', sbj='Abraham', obj='Isaac')\n",
      "     1.000 KBTriple(rel='parents', sbj='Isaac', obj='Abraham')\n",
      "     1.000 KBTriple(rel='parents', sbj='Abraham', obj='Ishmael')\n",
      "     1.000 KBTriple(rel='parents', sbj='Kim_Jong-un', obj='Kim_Jong-il')\n",
      "     1.000 KBTriple(rel='parents', sbj='Kim_Jong-il', obj='Kim_Jong-un')\n",
      "     1.000 KBTriple(rel='parents', sbj='Anne_Boleyn', obj='Elizabeth_I_of_England')\n",
      "     1.000 KBTriple(rel='parents', sbj='Elizabeth_I_of_England', obj='Anne_Boleyn')\n",
      "     1.000 KBTriple(rel='parents', sbj='Charlemagne', obj='Louis_the_Pious')\n",
      "     1.000 KBTriple(rel='parents', sbj='Louis_the_Pious', obj='Charlemagne')\n",
      "\n",
      "Highest probability examples for relation place_of_birth:\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     1.000 KBTriple(rel='place_of_birth', sbj='Matale_District', obj='Sri_Lanka')\n",
      "     1.000 KBTriple(rel='place_of_birth', sbj='Sri_Lanka', obj='Matale_District')\n",
      "     1.000 KBTriple(rel='place_of_birth', sbj='New_Zealand', obj='North_Island')\n",
      "     1.000 KBTriple(rel='place_of_birth', sbj='North_Island', obj='New_Zealand')\n",
      "     1.000 KBTriple(rel='place_of_birth', sbj='San_Francisco_Bay_Area', obj='California')\n",
      "     1.000 KBTriple(rel='place_of_birth', sbj='California', obj='San_Francisco_Bay_Area')\n",
      "     1.000 KBTriple(rel='place_of_birth', sbj='Illinois', obj='United_States_Senate')\n",
      "     1.000 KBTriple(rel='place_of_birth', sbj='United_States_Senate', obj='Illinois')\n",
      "     1.000 KBTriple(rel='place_of_birth', sbj='Ishmael', obj='Abraham')\n",
      "     1.000 KBTriple(rel='place_of_birth', sbj='Abraham', obj='Ishmael')\n",
      "\n",
      "Highest probability examples for relation place_of_death:\n",
      "\n",
      "     1.000 KBTriple(rel='place_of_death', sbj='Ishmael', obj='Abraham')\n",
      "     1.000 KBTriple(rel='place_of_death', sbj='Abraham', obj='Ishmael')\n",
      "     1.000 KBTriple(rel='place_of_death', sbj='Systema_Naturae', obj='Carl_Linnaeus')\n",
      "     1.000 KBTriple(rel='place_of_death', sbj='Carl_Linnaeus', obj='Systema_Naturae')\n",
      "     1.000 KBTriple(rel='place_of_death', sbj='Matale_District', obj='Sri_Lanka')\n",
      "     1.000 KBTriple(rel='place_of_death', sbj='Sri_Lanka', obj='Matale_District')\n",
      "     1.000 KBTriple(rel='place_of_death', sbj='New_Zealand', obj='North_Island')\n",
      "     1.000 KBTriple(rel='place_of_death', sbj='North_Island', obj='New_Zealand')\n",
      "     1.000 KBTriple(rel='place_of_death', sbj='Chernobyl_Nuclear_Power_Plant', obj='Ukraine')\n",
      "     1.000 KBTriple(rel='place_of_death', sbj='Ukraine', obj='Chernobyl_Nuclear_Power_Plant')\n",
      "\n",
      "Highest probability examples for relation profession:\n",
      "\n",
      "     1.000 KBTriple(rel='profession', sbj='Ishmael', obj='Abraham')\n",
      "     1.000 KBTriple(rel='profession', sbj='Abraham', obj='Ishmael')\n",
      "     0.999 KBTriple(rel='profession', sbj='Aldous_Huxley', obj='Eyeless_in_Gaza')\n",
      "     0.999 KBTriple(rel='profession', sbj='Eyeless_in_Gaza', obj='Aldous_Huxley')\n",
      "     0.995 KBTriple(rel='profession', sbj='Actor', obj='New_York')\n",
      "     0.995 KBTriple(rel='profession', sbj='New_York', obj='Actor')\n",
      "     0.994 KBTriple(rel='profession', sbj='Phuket', obj='Tilly_Smith')\n",
      "     0.994 KBTriple(rel='profession', sbj='Tilly_Smith', obj='Phuket')\n",
      "     0.992 KBTriple(rel='profession', sbj='Actor', obj='Screenwriter')\n",
      "     0.992 KBTriple(rel='profession', sbj='Screenwriter', obj='Actor')\n",
      "\n",
      "Highest probability examples for relation worked_at:\n",
      "\n",
      "     1.000 KBTriple(rel='worked_at', sbj='Matale_District', obj='Sri_Lanka')\n",
      "     1.000 KBTriple(rel='worked_at', sbj='Sri_Lanka', obj='Matale_District')\n",
      "     1.000 KBTriple(rel='worked_at', sbj='New_Zealand', obj='North_Island')\n",
      "     1.000 KBTriple(rel='worked_at', sbj='North_Island', obj='New_Zealand')\n",
      "     1.000 KBTriple(rel='worked_at', sbj='Church_of_Scientology', obj='L._Ron_Hubbard')\n",
      "     1.000 KBTriple(rel='worked_at', sbj='L._Ron_Hubbard', obj='Church_of_Scientology')\n",
      "     1.000 KBTriple(rel='worked_at', sbj='Austria', obj='Gaston_Glock')\n",
      "     1.000 KBTriple(rel='worked_at', sbj='Gaston_Glock', obj='Austria')\n",
      "     1.000 KBTriple(rel='worked_at', sbj='Adam_Weishaupt', obj='Illuminati')\n",
      "     1.000 KBTriple(rel='worked_at', sbj='Illuminati', obj='Adam_Weishaupt')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "find_new_relation_instances()\n",
    "# find_new_relation_instances(train_split='tiny', test_split='tiny') # for rapid development"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are actually some good discoveries here! The predictions for the `author` relation seem especially good. Of course, there are also plenty of bad results, and a few that are downright comical. We may hope that as we improve our models and optimize performance in our automatic evaluations, the results we observe in this manual evaluation improve as well.\n",
    "\n",
    "__Exercise:__ Note that every time we predict that a given relation holds between entities `X` and `Y`, we also predict, with equal confidence, that it holds between `Y` and `X`. Why? How could we fix this?\n",
    "\n",
    "\\[ [top](#Relation-extraction-using-distant-supervision) \\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "Our current model is quite rudimentary — it's merely a starting point for further exploration. This section lists a number of suggestions for next steps. Pursue whatever ideas look most promising! Your immediate goal is to optimize macro-averaged F<sub>0.5</sub>-score — but don't blinker yourself. Consider other ways of evaluating performance, and remember that the ultimate goal is to extract new relational triples.\n",
    "\n",
    "\n",
    "### Experimental methodology\n",
    "\n",
    "- Write code that facilitates _error analysis_ — that is, code that enables you to inspect, analyze, and categorize specific classification errors. Error analysis is the best avenue to understanding how to improve your model.\n",
    "- Our code for building datasets provides the ability to downsample negative examples, and we've used a default sampling rate of 10%. Is that the best choice? What are the consequences of sampling at a higher or lower rate?\n",
    "\n",
    "### Feature representation\n",
    "\n",
    "- Add a feature that indicates the length of the middle.\n",
    "- Augment the bag-of-words representation to include bigrams or trigrams (not just unigrams).\n",
    "- Introduce features based on the entity mentions themselves. <!-- \\[SPOILER: it helps a lot, maybe 4% in F-score. And combines nicely with the directional features.\\] -->\n",
    "- Experiment with features based on the context outside (rather than between) the two entity mentions — that is, the words before the first mention, or after the second.\n",
    "- Try adding features which capture syntactic information, such as the dependency-path features used by Mintz et al. 2009. The [NLTK](https://www.nltk.org/) toolkit contains a variety of [parsing algorithms](http://www.nltk.org/api/nltk.parse.html) that may help.\n",
    "- The bag-of-words representation does not permit generalization across word categories such as names of people, places, or companies. Can we do better using word embeddings such as [GloVe](https://nlp.stanford.edu/projects/glove/)?\n",
    "\n",
    "### Model selection\n",
    "\n",
    "\n",
    "\n",
    "- The `LogisticRegression` model in `sklearn` does L2 regularization by default. Try using L1 regularization instead. <!-- \\[SPOILER: doesn't make huge difference\\] -->\n",
    "- Whether you're using L1 or L2 regularization, you may get better results by tuning the regularization parameter. <!-- \\[SPOILER: `C=8.0`, i.e. less regularization, helps a little\\] -->\n",
    "- Experiment with different model types. `sklearn` makes this very easy: it provides implementations for everything from [elastic nets](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNetCV.html) to [SVMs](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) to [gradient boosting](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html).\n",
    "- Explore ways to predict the relations that hold between a given pair of entities jointly, instead of independently, in order to exploit the correlations between relations. This could be done using an ensemble or hierarchial model, or a neural architecture.\n",
    "- Over the last few years, neural sequence models such as LSTMs have yielded dramatic gains on many NLP tasks. Investigate whether they help here.\n",
    "\n",
    "\\[ [top](#Relation-extraction-using-distant-supervision) \\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework 3\n",
    "\n",
    "The purpose of this homework is to begin exploring larger, more diverse feature space, to figure out which ones lead to improved classifiers.\n",
    "\n",
    "As a reminder, our baseline classifier (using `simple_bag_of_words_featurizer` looks like this) when run on the `dev` set with the standard `model_factory`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building datasets\n",
      "Featurizing\n",
      "Training models\n",
      "Training complete\n",
      "\n",
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   0.873      0.452      0.736        303       5319\n",
      "author                    0.790      0.550      0.727        480       5496\n",
      "capital                   0.575      0.258      0.462         89       5105\n",
      "contains                  0.744      0.609      0.712       2667       7683\n",
      "film_performance          0.812      0.572      0.749        822       5838\n",
      "founders                  0.731      0.379      0.617        359       5375\n",
      "genre                     0.591      0.235      0.453        166       5182\n",
      "has_sibling               0.847      0.238      0.560        513       5529\n",
      "has_spouse                0.916      0.358      0.698        575       5591\n",
      "is_a                      0.706      0.243      0.511        494       5510\n",
      "nationality               0.500      0.145      0.335        311       5327\n",
      "parents                   0.860      0.585      0.786        325       5341\n",
      "place_of_birth            0.639      0.212      0.455        217       5233\n",
      "place_of_death            0.447      0.117      0.286        145       5161\n",
      "profession                0.629      0.159      0.396        245       5261\n",
      "worked_at                 0.682      0.254      0.511        228       5244\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.709      0.335      0.562       7939      88195\n"
     ]
    }
   ],
   "source": [
    "baseline = experiment(featurizers=[simple_bag_of_words_featurizer])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Questions 1–3: Directional unigram features [3 points]\n",
    "\n",
    "The current bag-of-words representation makes no distinction between \"forward\" and \"reverse\" examples. But, intuitively, there is big difference between _X and his son Y_ and _Y and his son X_. This question asks you to modify `simple_bag_of_words_featurizer` to capture these differences. \n",
    "\n",
    "__To submit:__\n",
    "\n",
    "1. A feature function `directional_bag_of_words_featurizer` that is just like `simple_bag_of_words_featurizer` except that it distinguishes \"forward\" and \"reverse\". To do this, you just need to mark each word feature for whether it is derived from a subject–object example or from an object–subject example. The precise nature of the mark you add for the two cases doesn't make a difference to the model.\n",
    "\n",
    "2. The macro-average F-score on the `dev` set that you obtain from running `experiment` with `directional_bag_of_words_featurizer` as the only featurizer. (Aside from this, use all the default values for `experiment`.)\n",
    "\n",
    "3. `experiment` returns some of the core objects used in the experiment. How many feature names does the `vectorizer` have for the experiment run in the previous step? (Note: we're partly asking you to figure out how to get this value by using the sklearn documentation, so please don't ask how to do it on Piazza!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KBTriple(rel='contains', sbj='Brickfields', obj='Kuala_Lumpur_Sentral_railway_station')"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kbt = next(kb.get_triples())\n",
    "kbt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Example(entity_1='Marche', entity_2='Ancona', left='2011 History of Town and Charity Background History of the Town Servigliano is a comune ( municipality ) in the Province of Fermo in the Italian region', mention_1='Marche', middle=', located about 60 km south of', mention_2='Ancona', right='. As far back as 1914 , with the imminent prospect of Italy entering the war , a large holding camp was built in Servigliano for the', left_POS='2011/CD History/NN of/IN Town/NNP and/CC Charity/NNP Background/NNP History/NN of/IN the/DT Town/NNP Servigliano/NNP is/VBZ a/DT comune/NN -LRB-/-LRB- municipality/NN -RRB-/-RRB- in/IN the/DT Province/NNP of/IN Fermo/NNP in/IN the/DT Italian/JJ region/NN', mention_1_POS='Marche/NNP', middle_POS=',/, located/JJ about/RB 60/CD km/NN south/NN of/IN', mention_2_POS='Ancona/NNP', right_POS='./. As/IN far/RB back/RB as/IN 1914/CD ,/, with/IN the/DT imminent/JJ prospect/NN of/IN Italy/NNP entering/VBG the/DT war/NN ,/, a/DT large/JJ holding/VBG camp/NN was/VBD built/VBN in/IN Servigliano/NNP for/IN the/DT')"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = next(corpus.get_examples())\n",
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def directional_bag_of_words_featurizer(kbt, corpus, feature_counter):\n",
    "    for ex in corpus.get_examples_for_entities(kbt.sbj, kbt.obj):\n",
    "        for word in ex.middle.split(' '):\n",
    "            feature_counter['FWD' + word] += 1\n",
    "    for ex in corpus.get_examples_for_entities(kbt.obj, kbt.sbj):\n",
    "        for word in ex.middle.split(' '):\n",
    "            feature_counter['BWD' + word] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building datasets\n",
      "Featurizing\n",
      "Training models\n",
      "Training complete\n",
      "\n",
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   0.881      0.515      0.772        303       5319\n",
      "author                    0.831      0.625      0.780        480       5496\n",
      "capital                   0.558      0.270      0.460         89       5105\n",
      "contains                  0.765      0.670      0.744       2667       7683\n",
      "film_performance          0.855      0.645      0.803        822       5838\n",
      "founders                  0.775      0.432      0.669        359       5375\n",
      "genre                     0.739      0.307      0.577        166       5182\n",
      "has_sibling               0.860      0.240      0.567        513       5529\n",
      "has_spouse                0.897      0.362      0.692        575       5591\n",
      "is_a                      0.757      0.271      0.557        494       5510\n",
      "nationality               0.570      0.209      0.424        311       5327\n",
      "parents                   0.870      0.575      0.789        325       5341\n",
      "place_of_birth            0.655      0.263      0.504        217       5233\n",
      "place_of_death            0.600      0.166      0.393        145       5161\n",
      "profession                0.737      0.229      0.510        245       5261\n",
      "worked_at                 0.725      0.289      0.557        228       5244\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.755      0.379      0.612       7939      88195\n"
     ]
    }
   ],
   "source": [
    "directional = experiment(featurizers = [directional_bag_of_words_featurizer])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "43279"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(directional['vectorizer'].feature_names_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Questions 4–5: The part-of-speech tags of the \"middle\" words [3 points]\n",
    "\n",
    "Our corpus distribution contains part-of-speech (POS) tagged versions of the core text spans. Let's begin to explore whether there is information in these sequences, focusing on `middle_POS`.\n",
    "\n",
    "__To submit:__\n",
    "\n",
    "1. A feature function `middle_bigram_pos_tag_featurizer` that is just like `simple_bag_of_words_featurizer` except that it creates a feature for bigram POS sequences. For example, given \n",
    "\n",
    "  `The/DT dog/N napped/V`\n",
    "  \n",
    "   we obtain the list of bigram POS sequences\n",
    "  \n",
    "   `['<s> DT', 'DT N', 'N V', 'V </s>']`. \n",
    "   \n",
    "   Don't forget the start and end tags, to model those environments properly!\n",
    "\n",
    "2. The macro-average F-score on the `dev` set that you obtain from running `experiment` with `middle_bigram_pos_tag_featurizer` as the only featurizer. (Aside from this, use all the default values for `experiment`.)\n",
    "\n",
    "Note: To parse `middle_POS`, one splits on whitespace to get the `word/TAG` pairs. Each of these pairs `s` can be parsed with `s.rsplit('/', 1)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [],
   "source": [
    "def middle_bigram_pos_tag_featurizer(kbt, corpus, feature_counter):\n",
    "    for ex in corpus.get_examples_for_entities(kbt.sbj, kbt.obj):\n",
    "        mid_words = ['<S>'] + [x.rsplit('/', 1)[-1] for x in ex.middle_POS.split(' ')] + ['</S>']\n",
    "        for idx in range(len(mid_words)-1):\n",
    "            feature_counter[mid_words[idx] + ' ' + mid_words[idx+1]] += 1\n",
    "    for ex in corpus.get_examples_for_entities(kbt.obj, kbt.sbj):\n",
    "        mid_words = ['<S>'] + [x.rsplit('/', 1)[-1] for x in ex.middle_POS.split(' ')] + ['</S>']\n",
    "        for idx in range(len(mid_words)-1):\n",
    "            feature_counter[mid_words[idx] + ' ' + mid_words[idx+1]] += 1            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building datasets\n",
      "Featurizing\n",
      "Training models\n",
      "Training complete\n",
      "\n",
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   0.890      0.426      0.730        303       5319\n",
      "author                    0.695      0.338      0.574        480       5496\n",
      "capital                   0.433      0.146      0.311         89       5105\n",
      "contains                  0.704      0.642      0.691       2667       7683\n",
      "film_performance          0.746      0.425      0.648        822       5838\n",
      "founders                  0.573      0.175      0.394        359       5375\n",
      "genre                     0.654      0.205      0.455        166       5182\n",
      "has_sibling               0.649      0.144      0.382        513       5529\n",
      "has_spouse                0.723      0.254      0.528        575       5591\n",
      "is_a                      0.652      0.178      0.426        494       5510\n",
      "nationality               0.271      0.042      0.129        311       5327\n",
      "parents                   0.698      0.298      0.551        325       5341\n",
      "place_of_birth            0.550      0.101      0.292        217       5233\n",
      "place_of_death            0.433      0.090      0.245        145       5161\n",
      "profession                0.677      0.171      0.426        245       5261\n",
      "worked_at                 0.519      0.118      0.310        228       5244\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.617      0.235      0.443       7939      88195\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "middle_bigram_pos_tag = experiment(featurizers = [middle_bigram_pos_tag_featurizer])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Questions 6–7: Bag of Synsets [3 points]\n",
    "\n",
    "The following allows you to use NLTK's WordNet API to get the synsets compatible with _dog_ as used as a noun:\n",
    "\n",
    "```\n",
    "from nltk.corpus import wordnet as wn\n",
    "dog = wn.synsets('dog', pos='n')\n",
    "```\n",
    "\n",
    "This question asks you to create synset-based features from the word/tag pairs in `middle_POS`.\n",
    "\n",
    "To convert the tags in the corpus to WordNet tags:\n",
    "\n",
    "| Tag begins with | WordNet `pos` value |\n",
    "|-----------------|---------------------|\n",
    "| `N`             | `'n'`               |\n",
    "| `V`             | `'v'`               |\n",
    "| `J`             | `'a'`               |\n",
    "| `R`             | `'r'`               |\n",
    "| Otherwise       | `None`              |\n",
    "\n",
    "__To submit:__\n",
    "\n",
    "1. A feature function `synset_featurizer` that is just like `simple_bag_of_words_featurizer` except that it creates a count dictionary where the keys are synsets, as derived from the unigrams in the `middle_POS` field using `wn.synsets`. Stringify the synsets with `str` so that they can be `dict` keys. Use the table above to convert tags to `pos` arguments usable by `wn.synsets`.\n",
    "\n",
    "2. The macro-average F-score on the `dev` set that you obtain from running `experiment` with `synset_featurizer` as the only featurizer. (Aside from this, use all the default values for `experiment`.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import wordnet as wn\n",
    "dog = wn.synsets('dog', pos='n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "pos_map = {\n",
    "        'N': 'n',\n",
    "        'V': 'v',\n",
    "        'J': 'a',\n",
    "        'R': 'r'\n",
    "    }\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "def synset_featurizer(kbt, corpus, feature_counter):\n",
    "    pos_map = {\n",
    "        'N': 'n',\n",
    "        'V': 'v',\n",
    "        'J': 'a',\n",
    "        'R': 'r'\n",
    "    }\n",
    "    \n",
    "    for ex in corpus.get_examples_for_entities(kbt.sbj, kbt.obj):\n",
    "        mid_words_pos = [x.rsplit('/', 1) for x in ex.middle_POS.split(' ')]\n",
    "        for pair in mid_words_pos:\n",
    "            if len(pair) != 2:\n",
    "                continue\n",
    "            else:\n",
    "                word, pos = pair\n",
    "            for sysnet_key in wn.synsets(word, pos=None if pos not in pos_map else pos_map[pos]):\n",
    "                feature_counter[str(sysnet_key)] += 1\n",
    "                \n",
    "    for ex in corpus.get_examples_for_entities(kbt.obj, kbt.sbj):\n",
    "        mid_words_pos = [x.rsplit('/', 1) for x in ex.middle_POS.split(' ')]\n",
    "        for pair in mid_words_pos:\n",
    "            if len(pair) != 2:\n",
    "                continue\n",
    "            else:\n",
    "                word, pos = pair\n",
    "            for sysnet_key in wn.synsets(word, pos=None if pos not in pos_map else pos_map[pos]):\n",
    "                feature_counter[str(sysnet_key)] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building datasets\n",
      "Featurizing\n",
      "Training models\n",
      "Training complete\n",
      "\n",
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   0.823      0.429      0.695        303       5319\n",
      "author                    0.800      0.458      0.696        480       5496\n",
      "capital                   0.617      0.326      0.523         89       5105\n",
      "contains                  0.736      0.595      0.703       2667       7683\n",
      "film_performance          0.787      0.536      0.720        822       5838\n",
      "founders                  0.731      0.401      0.628        359       5375\n",
      "genre                     0.507      0.229      0.408        166       5182\n",
      "has_sibling               0.792      0.201      0.499        513       5529\n",
      "has_spouse                0.833      0.320      0.631        575       5591\n",
      "is_a                      0.607      0.235      0.461        494       5510\n",
      "nationality               0.447      0.148      0.318        311       5327\n",
      "parents                   0.836      0.582      0.769        325       5341\n",
      "place_of_birth            0.518      0.203      0.395        217       5233\n",
      "place_of_death            0.380      0.131      0.275        145       5161\n",
      "profession                0.561      0.151      0.363        245       5261\n",
      "worked_at                 0.614      0.237      0.466        228       5244\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.662      0.324      0.534       7939      88195\n"
     ]
    }
   ],
   "source": [
    "synset_exp = experiment(featurizers = [synset_featurizer])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Questions 8–9: Bringing them all together [2 points]\n",
    "\n",
    "Run `experiment` with all `directional_bag_of_words_featurizer`, `middle_bigram_pos_tag_featurizer`, and `synset_featurizer` together as the featurizers. Let's see if all this work paid off in terms of raw peformance!\n",
    "\n",
    "__To submit:__\n",
    "\n",
    "1. The macro-average F-score on the `dev` set that you obtain from running this experiment. (Aside from `featurizers`, use all the default values for `experiment`.)\n",
    "\n",
    "1. The number of feature names contained in the `vectorizer` for this experiment.\n",
    "\n",
    "Note: You'll see that this is a *very* large model. We want you to submit with the default value for `model_factory`, but it is worth trying variants that are more suitable for these inputs – `penalty=\"l1\"` for `LogisticRegression` is a good starting point, as it will give 0 weight to many uninformative features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\[ [top](#Relation-extraction-using-distant-supervision) \\]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building datasets\n",
      "Featurizing\n",
      "Training models\n",
      "Training complete\n",
      "\n",
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   0.843      0.515      0.748        303       5319\n",
      "author                    0.798      0.658      0.766        480       5496\n",
      "capital                   0.659      0.303      0.534         89       5105\n",
      "contains                  0.798      0.691      0.774       2667       7683\n",
      "film_performance          0.858      0.689      0.817        822       5838\n",
      "founders                  0.756      0.510      0.690        359       5375\n",
      "genre                     0.818      0.434      0.695        166       5182\n",
      "has_sibling               0.819      0.265      0.578        513       5529\n",
      "has_spouse                0.853      0.372      0.678        575       5591\n",
      "is_a                      0.719      0.362      0.601        494       5510\n",
      "nationality               0.503      0.238      0.412        311       5327\n",
      "parents                   0.841      0.588      0.775        325       5341\n",
      "place_of_birth            0.500      0.267      0.426        217       5233\n",
      "place_of_death            0.368      0.145      0.282        145       5161\n",
      "profession                0.800      0.327      0.620        245       5261\n",
      "worked_at                 0.645      0.351      0.552        228       5244\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.724      0.420      0.622       7939      88195\n",
      "CPU times: user 2min 18s, sys: 2.17 s, total: 2min 21s\n",
      "Wall time: 2min 32s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "hw_all_exp = experiment(featurizers = [directional_bag_of_words_featurizer, middle_bigram_pos_tag_featurizer, synset_featurizer])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "77314"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(hw_all_exp['vectorizer'].feature_names_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bake-off\n",
    "\n",
    "The goal of the bake-off for this unit is very simple: to achieve the best macro-averaged F<sub>0.5</sub>-score on the `test` split. The [Next steps](#Next-steps) section suggested a number of possible strategies for improving the baseline model, and [Homework 3](#Homework-3) explores several more. But of course these are by no means exhaustive. You can surely come up with lots of additional ideas. The sky is the limit!\n",
    "\n",
    "There's only one strict rule here: __you must not evaluate on the `test` split until you ready to report your bake-off results__. All evaluations during development must be on the `dev` split. To do otherwise would be methodologically unsound and intellectually dishonest!\n",
    "\n",
    "Your bake-off submission should include:\n",
    "\n",
    "- Your macro-averaged F<sub>0.5</sub>-score on the `test` split.\n",
    "- A brief description of the strategies you employed to achieve this result.\n",
    "\n",
    "Submission URL: https://goo.gl/forms/ohqzpnHMwHIT7f642\n",
    "\n",
    "\\[ [top](#Relation-extraction-using-distant-supervision) \\]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "import utils\n",
    "import scipy\n",
    "import re\n",
    "vsmdata_home = 'vsmdata'\n",
    "\n",
    "glove_home = os.path.join(vsmdata_home, 'glove.6B')\n",
    "glove_lookup = utils.glove2dict(os.path.join(glove_home, 'glove.6B.300d.txt'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [],
   "source": [
    "def directional_middle_bigram_pos_tag_featurizer(kbt, corpus, feature_counter):\n",
    "    for ex in corpus.get_examples_for_entities(kbt.sbj, kbt.obj):\n",
    "        mid_words = ['<S>'] + [x.rsplit('/', 1)[-1] for x in ex.middle_POS.split(' ')] + ['</S>']\n",
    "        for idx in range(len(mid_words)-1):\n",
    "            feature_counter['FWD' + mid_words[idx] + ' ' + mid_words[idx+1]] += 1\n",
    "    for ex in corpus.get_examples_for_entities(kbt.obj, kbt.sbj):\n",
    "        mid_words = ['<S>'] + [x.rsplit('/', 1)[-1] for x in ex.middle_POS.split(' ')] + ['</S>']\n",
    "        for idx in range(len(mid_words)-1):\n",
    "            feature_counter['BWD' + mid_words[idx] + ' ' + mid_words[idx+1]] += 1            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "def glove_dist(kbt, corpus, feature_counter):\n",
    "    sbj_vec = np.mean([glove_lookup[w] for w in kbt.sbj.lower().split('_') if w in glove_lookup], axis=0)\n",
    "    obj_vec = np.mean([glove_lookup[w] for w in kbt.obj.lower().split('_') if w in glove_lookup], axis=0)\n",
    "        \n",
    "    dist = scipy.spatial.distance.cosine(sbj_vec, obj_vec)\n",
    "    feature_counter['glove_dist_nan'] = int(np.isnan(dist))\n",
    "    if feature_counter['glove_dist_nan']:\n",
    "        feature_counter['glove_vec_dist'] = 0\n",
    "    else:\n",
    "        feature_counter['glove_vec_dist'] = dist\n",
    "\n",
    "def get_avg_glove_vec(s):\n",
    "    return np.mean([glove_lookup[w] for w in re.findall(r\"[^\\W\\d_]+|\\d+\", s.lower()) if w in glove_lookup], axis=0)\n",
    "\n",
    "def all_glove_dist(kbt, corpus, feature_counter):\n",
    "    sbj_vec = get_avg_glove_vec(kbt.sbj)\n",
    "    obj_vec = get_avg_glove_vec(kbt.obj)\n",
    "    \n",
    "    left_vec = []\n",
    "    right_vec = []\n",
    "    for ex in corpus.get_examples_for_entities(kbt.sbj, kbt.obj):\n",
    "        left_vec.append(get_avg_glove_vec(ex.left))\n",
    "        right_vec.append(get_avg_glove_vec(ex.right))\n",
    "    left_vec = np.mean([x for x in left_vec if not np.isnan(np.min(x))], axis=0)\n",
    "    right_vec = np.mean([x for x in right_vec if not np.isnan(np.min(x))], axis=0)\n",
    "\n",
    "    \n",
    "    dist = scipy.spatial.distance.cosine(sbj_vec, obj_vec)\n",
    "    feature_counter['glove_dist_nan'] = int(np.isnan(dist))\n",
    "    feature_counter['glove_vec_dist'] = 0 if feature_counter['glove_dist_nan'] else dist\n",
    "\n",
    "    dist = scipy.spatial.distance.cosine(sbj_vec, right_vec)\n",
    "    feature_counter['glove_dist_sbj_right_nan'] = int(np.isnan(dist))\n",
    "    feature_counter['glove_vec_sbj_right_dist'] = 0 if feature_counter['glove_dist_sbj_right_nan'] else dist\n",
    "    \n",
    "    dist = scipy.spatial.distance.cosine(obj_vec, left_vec)\n",
    "    feature_counter['glove_dist_obj_left_nan'] = int(np.isnan(dist))\n",
    "    feature_counter['glove_vec_obj_left_dist'] = 0 if feature_counter['glove_dist_obj_left_nan'] else dist\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "def middle_size_dist(kbt, corpus, feature_counter):\n",
    "    max_len = 10\n",
    "    middle_word_size = [0] * (max_len + 1)\n",
    "    avg_len = []\n",
    "    for ex in corpus.get_examples_for_entities(kbt.sbj, kbt.obj):\n",
    "        l = ex.middle.count(' ')\n",
    "        l_str = str(min(l, max_len))\n",
    "        feature_counter['mid_len_dist_' + l_str] += 1\n",
    "        avg_len.append(l)\n",
    "\n",
    "    feature_counter['avg_mid_len'] = 0 if np.isnan(np.mean(avg_len)) else np.mean(avg_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building datasets\n",
      "Featurizing\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/itsmrlin/workspace/cs224u/venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2957: RuntimeWarning: Mean of empty slice.\n",
      "  out=out, **kwargs)\n",
      "/Users/itsmrlin/workspace/cs224u/venv/lib/python3.6/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  ret = ret.dtype.type(ret / rcount)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training models\n",
      "Training complete\n",
      "\n",
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   0.854      0.578      0.779        303       5319\n",
      "author                    0.827      0.675      0.791        480       5496\n",
      "capital                   0.714      0.337      0.584         89       5105\n",
      "contains                  0.744      0.838      0.761       2667       7683\n",
      "film_performance          0.872      0.755      0.846        822       5838\n",
      "founders                  0.747      0.543      0.695        359       5375\n",
      "genre                     0.796      0.446      0.688        166       5182\n",
      "has_sibling               0.739      0.507      0.677        513       5529\n",
      "has_spouse                0.840      0.402      0.690        575       5591\n",
      "is_a                      0.719      0.383      0.611        494       5510\n",
      "nationality               0.596      0.318      0.508        311       5327\n",
      "parents                   0.846      0.662      0.802        325       5341\n",
      "place_of_birth            0.595      0.304      0.499        217       5233\n",
      "place_of_death            0.382      0.200      0.323        145       5161\n",
      "profession                0.791      0.371      0.645        245       5261\n",
      "worked_at                 0.705      0.377      0.601        228       5244\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.735      0.481      0.656       7939      88195\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "all_exp = experiment(featurizers = [directional_bag_of_words_featurizer,\n",
    "                                    directional_middle_bigram_pos_tag_featurizer,\n",
    "                                    synset_featurizer,\n",
    "                                    all_glove_dist, middle_size_dist])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building datasets\n",
      "Featurizing\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/itsmrlin/workspace/cs224u/venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2957: RuntimeWarning: Mean of empty slice.\n",
      "  out=out, **kwargs)\n",
      "/Users/itsmrlin/workspace/cs224u/venv/lib/python3.6/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  ret = ret.dtype.type(ret / rcount)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training models\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.2548            0.67s\n",
      "         2           0.2365            0.60s\n",
      "         3           0.2149            0.58s\n",
      "         4           0.1968            0.56s\n",
      "         5           0.1864            0.59s\n",
      "         6           0.1819            0.57s\n",
      "         7           0.1695            0.57s\n",
      "         8           0.1623            0.58s\n",
      "         9           0.1595            0.57s\n",
      "        10           0.1536            0.58s\n",
      "        20           0.1110            0.48s\n",
      "        30           0.0831            0.42s\n",
      "        40           0.0643            0.38s\n",
      "        50           0.0547            0.37s\n",
      "        60           0.0469            0.38s\n",
      "        70           0.0317            0.32s\n",
      "        80           0.0276            0.22s\n",
      "        90           0.0227            0.11s\n",
      "       100           0.0189            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.3908            0.79s\n",
      "         2           0.3451            0.77s\n",
      "         3           0.3117            0.75s\n",
      "         4           0.2842            0.72s\n",
      "         5           0.2605            0.73s\n",
      "         6           0.2383            0.73s\n",
      "         7           0.2155            0.80s\n",
      "         8           0.2016            0.80s\n",
      "         9           0.1909            0.79s\n",
      "        10           0.1801            0.77s\n",
      "        20           0.1243            0.67s\n",
      "        30           0.0993            0.57s\n",
      "        40           0.0843            0.46s\n",
      "        50           0.0734            0.38s\n",
      "        60           0.0637            0.28s\n",
      "        70           0.0561            0.20s\n",
      "        80           0.0290            0.13s\n",
      "        90           0.0241            0.07s\n",
      "       100           0.0207            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.0261            0.55s\n",
      "         2           0.0236            0.54s\n",
      "         3           0.0213            0.56s\n",
      "         4           0.0193            0.56s\n",
      "         5           0.0174            0.58s\n",
      "         6           0.0158            0.60s\n",
      "         7           0.0143            0.59s\n",
      "         8           0.0129            0.58s\n",
      "         9           0.0117            0.56s\n",
      "        10           0.0106            0.54s\n",
      "        20           0.0039            0.47s\n",
      "        30           0.0014            0.40s\n",
      "        40           0.0006            0.30s\n",
      "        50           0.0003            0.23s\n",
      "        60           0.0002            0.17s\n",
      "        70           0.0002            0.12s\n",
      "        80           0.0001            0.07s\n",
      "        90           0.0001            0.03s\n",
      "       100           0.0001            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.8411            0.68s\n",
      "         2           0.7774            0.82s\n",
      "         3           0.7292            0.74s\n",
      "         4           0.6877            0.70s\n",
      "         5           0.6548            0.68s\n",
      "         6           0.6270            0.67s\n",
      "         7           0.6016            0.65s\n",
      "         8           0.5796            0.63s\n",
      "         9           0.5615            0.62s\n",
      "        10           0.5458            0.61s\n",
      "        20           0.4342            0.58s\n",
      "        30           0.3912            0.48s\n",
      "        40           0.3584            0.44s\n",
      "        50           0.3240            0.35s\n",
      "        60           0.3032            0.28s\n",
      "        70           0.2851            0.21s\n",
      "        80           0.2707            0.14s\n",
      "        90           0.2565            0.07s\n",
      "       100           0.2425            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.8099            0.97s\n",
      "         2           0.7499            0.92s\n",
      "         3           0.7079            0.91s\n",
      "         4           0.6545            0.87s\n",
      "         5           0.6160            0.84s\n",
      "         6           0.5921            0.80s\n",
      "         7           0.5547            0.77s\n",
      "         8           0.5231            0.74s\n",
      "         9           0.4956            0.72s\n",
      "        10           0.4728            0.71s\n",
      "        20           0.3410            0.57s\n",
      "        30           0.2803            0.50s\n",
      "        40           0.2431            0.43s\n",
      "        50           0.2186            0.36s\n",
      "        60           0.1959            0.31s\n",
      "        70           0.1813            0.24s\n",
      "        80           0.1623            0.16s\n",
      "        90           0.1505            0.09s\n",
      "       100           0.1399            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.4222            0.86s\n",
      "         2           0.3859            1.00s\n",
      "         3           0.3593            3.02s\n",
      "         4           0.3371            3.28s\n",
      "         5           0.3177            2.85s\n",
      "         6           0.3000            2.55s\n",
      "         7           0.2901            2.29s\n",
      "         8           0.2814            2.10s\n",
      "         9           0.2736            2.03s\n",
      "        10           0.2675            1.94s\n",
      "        20           0.2305            1.14s\n",
      "        30           0.1911            0.80s\n",
      "        40           0.1667            0.59s\n",
      "        50           0.1545            0.44s\n",
      "        60           0.1396            0.33s\n",
      "        70           0.1254            0.24s\n",
      "        80           0.0982            0.15s\n",
      "        90           0.0812            0.07s\n",
      "       100           0.0734            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.2014            0.85s\n",
      "         2           0.1571            0.69s\n",
      "         3           0.1297            0.63s\n",
      "         4           0.1241            0.59s\n",
      "         5           0.1193            0.56s\n",
      "         6           0.1150            0.54s\n",
      "         7           0.1114            0.52s\n",
      "         8           0.1077            0.51s\n",
      "         9           0.1054            0.50s\n",
      "        10           0.1016            0.52s\n",
      "        20           0.0579            0.53s\n",
      "        30           0.0423            0.50s\n",
      "        40           0.0325            0.43s\n",
      "        50           0.0256            0.33s\n",
      "        60           0.0206            0.25s\n",
      "        70           0.0167            0.18s\n",
      "        80           0.0134            0.12s\n",
      "        90           0.0107            0.06s\n",
      "       100           0.0087            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.3599            0.56s\n",
      "         2           0.3231            0.56s\n",
      "         3           0.2986            0.57s\n",
      "         4           0.2794            0.57s\n",
      "         5           0.2646            0.63s\n",
      "         6           0.2494            0.63s\n",
      "         7           0.2389            0.64s\n",
      "         8           0.2304            0.64s\n",
      "         9           0.2230            0.64s\n",
      "        10           0.2162            0.65s\n",
      "        20           0.1629            0.62s\n",
      "        30           0.1314            0.60s\n",
      "        40           0.1189            0.49s\n",
      "        50           0.1086            0.43s\n",
      "        60           0.0993            0.37s\n",
      "        70           0.0923            0.27s\n",
      "        80           0.0862            0.19s\n",
      "        90           0.0813            0.09s\n",
      "       100           0.0757            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.7059            0.71s\n",
      "         2           0.6766            0.69s\n",
      "         3           0.6543            0.67s\n",
      "         4           0.6288            0.69s\n",
      "         5           0.6105            0.73s\n",
      "         6           0.5951            0.77s\n",
      "         7           0.5783            0.79s\n",
      "         8           0.5667            0.79s\n",
      "         9           0.5506            0.85s\n",
      "        10           0.5400            0.96s\n",
      "        20           0.4403            0.80s\n",
      "        30           0.4001            0.79s\n",
      "        40           0.3624            0.65s\n",
      "        50           0.3212            0.50s\n",
      "        60           0.2996            0.37s\n",
      "        70           0.2801            0.28s\n",
      "        80           0.2637            0.19s\n",
      "        90           0.2457            0.09s\n",
      "       100           0.2342            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.5051            0.71s\n",
      "         2           0.4854            0.65s\n",
      "         3           0.4703            0.64s\n",
      "         4           0.4539            0.61s\n",
      "         5           0.4354            0.58s\n",
      "         6           0.4207            0.56s\n",
      "         7           0.4132            0.55s\n",
      "         8           0.4067            0.54s\n",
      "         9           0.4015            0.53s\n",
      "        10           0.3963            0.51s\n",
      "        20           0.3527            0.43s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        30           0.3245            0.37s\n",
      "        40           0.2894            0.36s\n",
      "        50           0.2554            0.31s\n",
      "        60           0.2318            0.24s\n",
      "        70           0.2051            0.18s\n",
      "        80           0.1891            0.12s\n",
      "        90           0.1718            0.06s\n",
      "       100           0.1582            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.3578            0.90s\n",
      "         2           0.3396            0.82s\n",
      "         3           0.3274            0.80s\n",
      "         4           0.3142            0.74s\n",
      "         5           0.3002            0.74s\n",
      "         6           0.2960            0.73s\n",
      "         7           0.2894            0.76s\n",
      "         8           0.2867            0.80s\n",
      "         9           0.2833            0.79s\n",
      "        10           0.2780            0.78s\n",
      "        20           0.2202            0.86s\n",
      "        30           0.1839            0.82s\n",
      "        40           0.1560            0.64s\n",
      "        50           0.1397            0.52s\n",
      "        60           0.1244            0.40s\n",
      "        70           0.1126            0.28s\n",
      "        80           0.1043            0.18s\n",
      "        90           0.0963            0.08s\n",
      "       100           0.0878            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.1419            0.48s\n",
      "         2           0.0984            0.49s\n",
      "         3           0.0774            0.51s\n",
      "         4           0.0712            0.54s\n",
      "         5           0.0657            0.59s\n",
      "         6           0.0610            0.62s\n",
      "         7           0.0569            0.63s\n",
      "         8           0.0532            0.66s\n",
      "         9           0.0434            0.65s\n",
      "        10           0.0419            0.64s\n",
      "        20           0.0224            0.54s\n",
      "        30           0.0128            0.44s\n",
      "        40           0.0084            0.40s\n",
      "        50           0.0055            0.36s\n",
      "        60           0.0038            0.28s\n",
      "        70           0.0028            0.20s\n",
      "        80           0.0021            0.13s\n",
      "        90           0.0015            0.06s\n",
      "       100           0.0012            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.2158            0.53s\n",
      "         2           0.1719            0.52s\n",
      "         3           0.1528            0.54s\n",
      "         4           0.1484            0.54s\n",
      "         5           0.1445            0.54s\n",
      "         6           0.1382            0.53s\n",
      "         7           0.1319            0.52s\n",
      "         8           0.1266            0.51s\n",
      "         9           0.1220            0.50s\n",
      "        10           0.1166            0.51s\n",
      "        20           0.0884            0.52s\n",
      "        30           0.0744            0.44s\n",
      "        40           0.0651            0.35s\n",
      "        50           0.0594            0.28s\n",
      "        60           0.0535            0.21s\n",
      "        70           0.0496            0.16s\n",
      "        80           0.0391            0.11s\n",
      "        90           0.0178            0.05s\n",
      "       100           0.0159            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.1280            0.45s\n",
      "         2           0.0850            0.45s\n",
      "         3           0.0834            0.43s\n",
      "         4           0.0819            0.42s\n",
      "         5           0.0806            0.42s\n",
      "         6           0.0795            0.42s\n",
      "         7           0.0784            0.42s\n",
      "         8           0.0775            0.42s\n",
      "         9           0.0766            0.41s\n",
      "        10           0.0759            0.41s\n",
      "        20           0.0435            0.38s\n",
      "        30           0.0395            0.40s\n",
      "        40           0.0065            0.37s\n",
      "        50           0.0037            0.30s\n",
      "        60           0.0026            0.24s\n",
      "        70           0.0021            0.17s\n",
      "        80           0.0019            0.11s\n",
      "        90           0.0014            0.05s\n",
      "       100           0.0010            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.2880            0.78s\n",
      "         2           0.2705            0.73s\n",
      "         3           0.2356            0.69s\n",
      "         4           0.2293            0.67s\n",
      "         5           0.2241            0.64s\n",
      "         6           0.2197            0.61s\n",
      "         7           0.2158            0.59s\n",
      "         8           0.1868            0.58s\n",
      "         9           0.1838            0.56s\n",
      "        10           0.1804            0.54s\n",
      "        20           0.1594            0.51s\n",
      "        30           0.1266            0.47s\n",
      "        40           0.1170            0.40s\n",
      "        50           0.1110            0.32s\n",
      "        60           0.0581            0.24s\n",
      "        70           0.0525            0.17s\n",
      "        80           0.0239            0.11s\n",
      "        90           0.0203            0.06s\n",
      "       100           0.0179            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.2758            0.60s\n",
      "         2           0.2537            0.58s\n",
      "         3           0.2246            0.58s\n",
      "         4           0.2055            0.63s\n",
      "         5           0.1980            0.64s\n",
      "         6           0.1917            0.63s\n",
      "         7           0.1861            0.65s\n",
      "         8           0.1811            0.64s\n",
      "         9           0.1767            0.62s\n",
      "        10           0.1728            0.62s\n",
      "        20           0.1488            0.50s\n",
      "        30           0.1157            0.41s\n",
      "        40           0.0858            0.34s\n",
      "        50           0.0610            0.27s\n",
      "        60           0.0564            0.21s\n",
      "        70           0.0231            0.15s\n",
      "        80           0.0190            0.10s\n",
      "        90           0.0158            0.05s\n",
      "       100           0.0128            0.00s\n",
      "Training complete\n",
      "\n",
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   1.000      1.000      1.000         15        373\n",
      "author                    1.000      1.000      1.000         28        386\n",
      "capital                   1.000      1.000      1.000          5        363\n",
      "contains                  0.986      0.872      0.960         78        436\n",
      "film_performance          1.000      0.944      0.988         72        430\n",
      "founders                  1.000      0.889      0.976         27        385\n",
      "genre                     1.000      1.000      1.000         12        370\n",
      "has_sibling               1.000      0.913      0.981         23        381\n",
      "has_spouse                1.000      0.654      0.904         52        410\n",
      "is_a                      1.000      0.677      0.913         31        389\n",
      "nationality               1.000      0.762      0.941         21        379\n",
      "parents                   1.000      1.000      1.000          9        367\n",
      "place_of_birth            1.000      1.000      1.000         12        370\n",
      "place_of_death            1.000      1.000      1.000          7        365\n",
      "profession                1.000      1.000      1.000         16        374\n",
      "worked_at                 1.000      1.000      1.000         17        375\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.999      0.919      0.979        425       6153\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'featurizers': [<function __main__.directional_bag_of_words_featurizer(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.directional_middle_bigram_pos_tag_featurizer(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.synset_featurizer(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.all_glove_dist(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.middle_size_dist(kbt, corpus, feature_counter)>],\n",
       " 'vectorizer': DictVectorizer(dtype=<class 'numpy.float64'>, separator='=', sort=True,\n",
       "         sparse=True),\n",
       " 'models': {'adjoins': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'author': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'capital': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'contains': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'film_performance': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'founders': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'genre': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'has_sibling': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'has_spouse': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'is_a': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'nationality': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'parents': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'place_of_birth': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'place_of_death': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'profession': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'worked_at': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False)}}"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment(train_split='tiny', test_split='tiny',\n",
    "           featurizers = [directional_bag_of_words_featurizer,\n",
    "                            directional_middle_bigram_pos_tag_featurizer,\n",
    "                            synset_featurizer,\n",
    "                            all_glove_dist, middle_size_dist],\n",
    "           model_factory=lambda: sklearn.ensemble.GradientBoostingClassifier(verbose=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building datasets\n",
      "Featurizing\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/itsmrlin/workspace/cs224u/venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2957: RuntimeWarning: Mean of empty slice.\n",
      "  out=out, **kwargs)\n",
      "/Users/itsmrlin/workspace/cs224u/venv/lib/python3.6/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  ret = ret.dtype.type(ret / rcount)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training models\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.3990            2.24m\n",
      "         2           0.3726            2.34m\n",
      "         3           0.3525            2.27m\n",
      "         4           0.3365            2.21m\n",
      "         5           0.3244            2.20m\n",
      "         6           0.3146            2.19m\n",
      "         7           0.3062            2.21m\n",
      "         8           0.2993            2.26m\n",
      "         9           0.2928            2.25m\n",
      "        10           0.2879            2.26m\n",
      "        20           0.2548            2.13m\n",
      "        30           0.2381            1.82m\n",
      "        40           0.2259            1.50m\n",
      "        50           0.2166            1.26m\n",
      "        60           0.2116            1.00m\n",
      "        70           0.2044           44.84s\n",
      "        80           0.1994           29.85s\n",
      "        90           0.1945           14.64s\n",
      "       100           0.1921            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.5417            2.85m\n",
      "         2           0.4983            3.36m\n",
      "         3           0.4668            3.10m\n",
      "         4           0.4439            2.89m\n",
      "         5           0.4241            2.81m\n",
      "         6           0.4076            2.82m\n",
      "         7           0.3943            2.80m\n",
      "         8           0.3838            2.76m\n",
      "         9           0.3745            2.73m\n",
      "        10           0.3661            2.63m\n",
      "        20           0.3139            2.24m\n",
      "        30           0.2588            1.86m\n",
      "        40           0.2365            1.61m\n",
      "        50           0.2196            1.36m\n",
      "        60           0.2065            1.09m\n",
      "        70           0.1979           47.92s\n",
      "        80           0.1904           31.71s\n",
      "        90           0.1836           15.68s\n",
      "       100           0.1795            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.1415            1.99m\n",
      "         2           0.1369            2.06m\n",
      "         3           0.1348            2.34m\n",
      "         4           0.1320            2.27m\n",
      "         5           0.1297            2.21m\n",
      "         6           0.1269            2.24m\n",
      "         7           0.1249            2.22m\n",
      "         8           0.1220            2.19m\n",
      "         9           0.1193            2.17m\n",
      "        10           0.1174            2.18m\n",
      "        20           0.1025            1.81m\n",
      "        30           0.0901            1.52m\n",
      "        40           0.0835            1.27m\n",
      "        50           0.0792            1.05m\n",
      "        60           0.0754           50.09s\n",
      "        70           0.0739           37.21s\n",
      "        80           0.0727           24.61s\n",
      "        90           0.0711           12.21s\n",
      "       100           0.0704            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           1.3149            4.61m\n",
      "         2           1.2700            4.54m\n",
      "         3           1.2314            4.43m\n",
      "         4           1.1989            4.36m\n",
      "         5           1.1714            4.34m\n",
      "         6           1.1447            4.34m\n",
      "         7           1.1212            4.33m\n",
      "         8           1.1012            4.30m\n",
      "         9           1.0850            4.25m\n",
      "        10           1.0691            4.20m\n",
      "        20           0.9732            3.71m\n",
      "        30           0.9187            3.23m\n",
      "        40           0.8847            2.75m\n",
      "        50           0.8597            2.29m\n",
      "        60           0.8391            1.83m\n",
      "        70           0.8221            1.37m\n",
      "        80           0.8056           55.55s\n",
      "        90           0.7912           28.63s\n",
      "       100           0.7802            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.7080            2.79m\n",
      "         2           0.6697            2.91m\n",
      "         3           0.6408            2.87m\n",
      "         4           0.6175            2.80m\n",
      "         5           0.5926            2.86m\n",
      "         6           0.5781            2.86m\n",
      "         7           0.5644            2.79m\n",
      "         8           0.5475            2.76m\n",
      "         9           0.5344            2.81m\n",
      "        10           0.5222            2.76m\n",
      "        20           0.4402            2.45m\n",
      "        30           0.3927            2.12m\n",
      "        40           0.3628            1.83m\n",
      "        50           0.3420            1.54m\n",
      "        60           0.3244            1.20m\n",
      "        70           0.3114           55.05s\n",
      "        80           0.2997           36.14s\n",
      "        90           0.2900           17.99s\n",
      "       100           0.2814            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.4724            2.22m\n",
      "         2           0.4532            2.13m\n",
      "         3           0.4399            2.09m\n",
      "         4           0.4256            2.10m\n",
      "         5           0.4176            2.08m\n",
      "         6           0.4088            2.06m\n",
      "         7           0.4030            2.03m\n",
      "         8           0.3947            2.03m\n",
      "         9           0.3892            2.08m\n",
      "        10           0.3826            2.05m\n",
      "        20           0.3474            1.87m\n",
      "        30           0.3262            1.80m\n",
      "        40           0.3078            1.57m\n",
      "        50           0.2946            1.37m\n",
      "        60           0.2823            1.07m\n",
      "        70           0.2749           48.26s\n",
      "        80           0.2662           31.77s\n",
      "        90           0.2621           15.75s\n",
      "       100           0.2547            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.2460            2.46m\n",
      "         2           0.2347            2.24m\n",
      "         3           0.2268            2.17m\n",
      "         4           0.2220            2.11m\n",
      "         5           0.2188            2.07m\n",
      "         6           0.2162            2.03m\n",
      "         7           0.2136            2.02m\n",
      "         8           0.2114            1.98m\n",
      "         9           0.2098            1.94m\n",
      "        10           0.2077            1.93m\n",
      "        20           0.1891            1.72m\n",
      "        30           0.1756            1.49m\n",
      "        40           0.1665            1.27m\n",
      "        50           0.1581            1.07m\n",
      "        60           0.1476           52.77s\n",
      "        70           0.1371           40.38s\n",
      "        80           0.1295           27.27s\n",
      "        90           0.1249           13.69s\n",
      "       100           0.1225            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.5358            2.75m\n",
      "         2           0.4941            2.69m\n",
      "         3           0.4621            2.51m\n",
      "         4           0.4385            2.45m\n",
      "         5           0.4183            2.40m\n",
      "         6           0.4024            2.35m\n",
      "         7           0.3885            2.31m\n",
      "         8           0.3768            2.28m\n",
      "         9           0.3675            2.25m\n",
      "        10           0.3587            2.23m\n",
      "        20           0.3098            2.01m\n",
      "        30           0.2885            1.72m\n",
      "        40           0.2742            1.44m\n",
      "        50           0.2654            1.19m\n",
      "        60           0.2583           56.66s\n",
      "        70           0.2545           42.30s\n",
      "        80           0.2480           28.06s\n",
      "        90           0.2392           13.95s\n",
      "       100           0.2327            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.6110            2.32m\n",
      "         2           0.5787            2.24m\n",
      "         3           0.5574            2.27m\n",
      "         4           0.5416            2.26m\n",
      "         5           0.5270            2.23m\n",
      "         6           0.5152            2.21m\n",
      "         7           0.5051            2.17m\n",
      "         8           0.4982            2.15m\n",
      "         9           0.4889            2.12m\n",
      "        10           0.4823            2.12m\n",
      "        20           0.4389            1.88m\n",
      "        30           0.3811            1.64m\n",
      "        40           0.3651            1.41m\n",
      "        50           0.3460            1.18m\n",
      "        60           0.3340           56.66s\n",
      "        70           0.3244           42.37s\n",
      "        80           0.3179           28.25s\n",
      "        90           0.3127           14.13s\n",
      "       100           0.3064            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.5851            2.38m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         2           0.5721            2.32m\n",
      "         3           0.5634            2.28m\n",
      "         4           0.5566            2.24m\n",
      "         5           0.5499            2.19m\n",
      "         6           0.5437            2.15m\n",
      "         7           0.5394            2.13m\n",
      "         8           0.5340            2.10m\n",
      "         9           0.5272            2.10m\n",
      "        10           0.5230            2.08m\n",
      "        20           0.4877            1.81m\n",
      "        30           0.4644            1.57m\n",
      "        40           0.4467            1.34m\n",
      "        50           0.4307            1.12m\n",
      "        60           0.4129           53.56s\n",
      "        70           0.4017           40.37s\n",
      "        80           0.3831           27.19s\n",
      "        90           0.3758           13.58s\n",
      "       100           0.3641            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.4149            2.08m\n",
      "         2           0.4051            2.08m\n",
      "         3           0.3947            2.03m\n",
      "         4           0.3897            2.00m\n",
      "         5           0.3809            1.97m\n",
      "         6           0.3739            1.95m\n",
      "         7           0.3679            1.94m\n",
      "         8           0.3621            1.93m\n",
      "         9           0.3576            1.92m\n",
      "        10           0.3550            1.90m\n",
      "        20           0.3284            1.68m\n",
      "        30           0.3117            1.47m\n",
      "        40           0.2974            1.26m\n",
      "        50           0.2861            1.06m\n",
      "        60           0.2790           50.99s\n",
      "        70           0.2718           38.29s\n",
      "        80           0.2657           25.76s\n",
      "        90           0.2600           13.09s\n",
      "       100           0.2544            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.3470            2.24m\n",
      "         2           0.3222            2.19m\n",
      "         3           0.2994            2.15m\n",
      "         4           0.2859            2.13m\n",
      "         5           0.2706            2.11m\n",
      "         6           0.2583            2.10m\n",
      "         7           0.2483            2.07m\n",
      "         8           0.2417            2.05m\n",
      "         9           0.2341            2.03m\n",
      "        10           0.2292            2.00m\n",
      "        20           0.1870            1.81m\n",
      "        30           0.1690            1.70m\n",
      "        40           0.1524            1.51m\n",
      "        50           0.1438            1.25m\n",
      "        60           0.1369           59.37s\n",
      "        70           0.1298           43.83s\n",
      "        80           0.1229           28.79s\n",
      "        90           0.1185           14.26s\n",
      "       100           0.1168            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.3108            2.12m\n",
      "         2           0.3030            2.08m\n",
      "         3           0.2957            2.07m\n",
      "         4           0.2901            2.06m\n",
      "         5           0.2837            2.04m\n",
      "         6           0.2797            2.01m\n",
      "         7           0.2767            2.01m\n",
      "         8           0.2744            1.99m\n",
      "         9           0.2715            1.97m\n",
      "        10           0.2687            1.94m\n",
      "        20           0.2465            1.74m\n",
      "        30           0.2328            1.61m\n",
      "        40           0.2245            1.41m\n",
      "        50           0.2159            1.18m\n",
      "        60           0.2088           57.45s\n",
      "        70           0.1991           42.52s\n",
      "        80           0.1915           28.15s\n",
      "        90           0.1841           14.16s\n",
      "       100           0.1785            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.2642            1.97m\n",
      "         2           0.2558            2.12m\n",
      "         3           0.2506            2.09m\n",
      "         4           0.2467            2.14m\n",
      "         5           0.2424            2.21m\n",
      "         6           0.2400            2.20m\n",
      "         7           0.2383            2.15m\n",
      "         8           0.2362            2.10m\n",
      "         9           0.2337            2.05m\n",
      "        10           0.2323            2.02m\n",
      "        20           0.2195            1.83m\n",
      "        30           0.2097            1.62m\n",
      "        40           0.2005            1.39m\n",
      "        50           0.1942            1.15m\n",
      "        60           0.1862           54.20s\n",
      "        70           0.1775           40.80s\n",
      "        80           0.1681           27.28s\n",
      "        90           0.1597           13.89s\n",
      "       100           0.1532            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.3409            3.17m\n",
      "         2           0.3331            3.00m\n",
      "         3           0.3279            2.77m\n",
      "         4           0.3244            2.58m\n",
      "         5           0.3171            2.49m\n",
      "         6           0.3116            2.41m\n",
      "         7           0.3073            2.38m\n",
      "         8           0.3015            2.44m\n",
      "         9           0.2992            2.43m\n",
      "        10           0.2961            2.39m\n",
      "        20           0.2781            2.10m\n",
      "        30           0.2645            1.80m\n",
      "        40           0.2549            1.51m\n",
      "        50           0.2179            1.26m\n",
      "        60           0.2123           59.58s\n",
      "        70           0.2053           44.09s\n",
      "        80           0.1981           29.47s\n",
      "        90           0.1902           14.67s\n",
      "       100           0.1822            0.00s\n",
      "      Iter       Train Loss   Remaining Time \n",
      "         1           0.3193            2.77m\n",
      "         2           0.3022            2.72m\n",
      "         3           0.2915            2.61m\n",
      "         4           0.2820            2.62m\n",
      "         5           0.2750            2.64m\n",
      "         6           0.2714            2.58m\n",
      "         7           0.2665            2.51m\n",
      "         8           0.2630            2.48m\n",
      "         9           0.2591            2.47m\n",
      "        10           0.2564            2.47m\n",
      "        20           0.2318            2.20m\n",
      "        30           0.2172            1.86m\n",
      "        40           0.2074            1.57m\n",
      "        50           0.1985            1.29m\n",
      "        60           0.1902            1.04m\n",
      "        70           0.1849           46.15s\n",
      "        80           0.1810           30.40s\n",
      "        90           0.1727           14.97s\n",
      "       100           0.1650            0.00s\n",
      "Training complete\n",
      "\n",
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   0.893      0.525      0.783        303       5319\n",
      "author                    0.883      0.673      0.831        480       5496\n",
      "capital                   0.564      0.348      0.502         89       5105\n",
      "contains                  0.801      0.637      0.762       2667       7683\n",
      "film_performance          0.890      0.652      0.830        822       5838\n",
      "founders                  0.859      0.407      0.703        359       5375\n",
      "genre                     0.716      0.289      0.553        166       5182\n",
      "has_sibling               0.834      0.569      0.763        513       5529\n",
      "has_spouse                0.849      0.383      0.683        575       5591\n",
      "is_a                      0.842      0.259      0.581        494       5510\n",
      "nationality               0.770      0.215      0.508        311       5327\n",
      "parents                   0.860      0.603      0.792        325       5341\n",
      "place_of_birth            0.750      0.249      0.535        217       5233\n",
      "place_of_death            0.594      0.131      0.348        145       5161\n",
      "profession                0.712      0.192      0.462        245       5261\n",
      "worked_at                 0.724      0.311      0.573        228       5244\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.784      0.403      0.638       7939      88195\n",
      "CPU times: user 44min 47s, sys: 19.5 s, total: 45min 6s\n",
      "Wall time: 47min 13s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'featurizers': [<function __main__.directional_bag_of_words_featurizer(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.directional_middle_bigram_pos_tag_featurizer(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.synset_featurizer(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.all_glove_dist(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.middle_size_dist(kbt, corpus, feature_counter)>],\n",
       " 'vectorizer': DictVectorizer(dtype=<class 'numpy.float64'>, separator='=', sort=True,\n",
       "         sparse=True),\n",
       " 'models': {'adjoins': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'author': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'capital': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'contains': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'film_performance': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'founders': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'genre': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'has_sibling': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'has_spouse': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'is_a': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'nationality': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'parents': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'place_of_birth': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'place_of_death': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'profession': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False),\n",
       "  'worked_at': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "                learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "                max_features=None, max_leaf_nodes=None,\n",
       "                min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                min_samples_leaf=1, min_samples_split=2,\n",
       "                min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "                presort='auto', random_state=None, subsample=1.0,\n",
       "                verbose=True, warm_start=False)}}"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "experiment(featurizers = [directional_bag_of_words_featurizer,\n",
    "                            directional_middle_bigram_pos_tag_featurizer,\n",
    "                            synset_featurizer,\n",
    "                            all_glove_dist, middle_size_dist],\n",
    "           model_factory=lambda: sklearn.ensemble.GradientBoostingClassifier(verbose=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building datasets\n",
      "Featurizing\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/itsmrlin/workspace/cs224u/venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2957: RuntimeWarning: Mean of empty slice.\n",
      "  out=out, **kwargs)\n",
      "/Users/itsmrlin/workspace/cs224u/venv/lib/python3.6/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  ret = ret.dtype.type(ret / rcount)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training models\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:   13.9s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    9.4s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    7.6s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:   37.0s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:   10.8s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    9.9s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    7.7s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:   13.3s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:   20.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:   16.9s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:   12.3s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:   10.5s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:   10.3s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:   10.3s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:   11.4s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    8.7s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training complete\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/itsmrlin/workspace/cs224u/venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2957: RuntimeWarning: Mean of empty slice.\n",
      "  out=out, **kwargs)\n",
      "/Users/itsmrlin/workspace/cs224u/venv/lib/python3.6/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  ret = ret.dtype.type(ret / rcount)\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   0.786      0.531      0.717        311       5391\n",
      "author                    0.822      0.772      0.812        491       5571\n",
      "capital                   0.603      0.469      0.571         81       5161\n",
      "contains                  0.785      0.770      0.782       2878       7958\n",
      "film_performance          0.884      0.688      0.836        785       5865\n",
      "founders                  0.785      0.629      0.748        313       5393\n",
      "genre                     0.569      0.311      0.488        132       5212\n",
      "has_sibling               0.857      0.650      0.805        534       5614\n",
      "has_spouse                0.848      0.640      0.796        594       5674\n",
      "is_a                      0.755      0.461      0.670        462       5542\n",
      "nationality               0.822      0.566      0.754        286       5366\n",
      "parents                   0.854      0.763      0.834        321       5401\n",
      "place_of_birth            0.840      0.461      0.722        217       5297\n",
      "place_of_death            0.815      0.323      0.625        164       5244\n",
      "profession                0.741      0.352      0.607        244       5324\n",
      "worked_at                 0.829      0.504      0.734        240       5320\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.787      0.556      0.719       8053      89333\n",
      "CPU times: user 9min 35s, sys: 7.31 s, total: 9min 42s\n",
      "Wall time: 10min 26s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  10 out of  10 | elapsed:    0.1s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'featurizers': [<function __main__.directional_bag_of_words_featurizer(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.directional_middle_bigram_pos_tag_featurizer(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.synset_featurizer(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.all_glove_dist(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.middle_size_dist(kbt, corpus, feature_counter)>],\n",
       " 'vectorizer': DictVectorizer(dtype=<class 'numpy.float64'>, separator='=', sort=True,\n",
       "         sparse=True),\n",
       " 'models': {'adjoins': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'author': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'capital': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'contains': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'film_performance': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'founders': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'genre': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'has_sibling': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'has_spouse': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'is_a': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'nationality': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'parents': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'place_of_birth': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'place_of_death': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'profession': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False),\n",
       "  'worked_at': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "              max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_samples_split=2,\n",
       "              min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "              oob_score=False, random_state=None, verbose=True,\n",
       "              warm_start=False)}}"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "experiment(train_split='train', test_split='test',\n",
    "           featurizers = [directional_bag_of_words_featurizer,\n",
    "                            directional_middle_bigram_pos_tag_featurizer,\n",
    "                            synset_featurizer,\n",
    "                            all_glove_dist, middle_size_dist],\n",
    "          model_factory=lambda: sklearn.ensemble.RandomForestClassifier(verbose=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building datasets\n",
      "Featurizing\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/itsmrlin/workspace/cs224u/venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2957: RuntimeWarning: Mean of empty slice.\n",
      "  out=out, **kwargs)\n",
      "/Users/itsmrlin/workspace/cs224u/venv/lib/python3.6/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  ret = ret.dtype.type(ret / rcount)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training models\n",
      "Training complete\n",
      "\n",
      "relation              precision     recall    f-score    support       size\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "adjoins                   0.845      0.492      0.739        311       5391\n",
      "author                    0.829      0.654      0.787        491       5571\n",
      "capital                   0.733      0.407      0.632         81       5161\n",
      "contains                  0.750      0.803      0.760       2878       7958\n",
      "film_performance          0.859      0.746      0.834        785       5865\n",
      "founders                  0.713      0.508      0.660        313       5393\n",
      "genre                     0.619      0.394      0.556        132       5212\n",
      "has_sibling               0.754      0.545      0.700        534       5614\n",
      "has_spouse                0.825      0.372      0.663        594       5674\n",
      "is_a                      0.791      0.294      0.591        462       5542\n",
      "nationality               0.589      0.301      0.494        286       5366\n",
      "parents                   0.842      0.648      0.794        321       5401\n",
      "place_of_birth            0.649      0.332      0.545        217       5297\n",
      "place_of_death            0.591      0.317      0.504        164       5244\n",
      "profession                0.703      0.369      0.595        244       5324\n",
      "worked_at                 0.741      0.454      0.658        240       5320\n",
      "------------------    ---------  ---------  ---------  ---------  ---------\n",
      "macro-average             0.740      0.477      0.657       8053      89333\n",
      "CPU times: user 6min 34s, sys: 7.37 s, total: 6min 41s\n",
      "Wall time: 7min 23s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'featurizers': [<function __main__.directional_bag_of_words_featurizer(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.directional_middle_bigram_pos_tag_featurizer(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.synset_featurizer(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.all_glove_dist(kbt, corpus, feature_counter)>,\n",
       "  <function __main__.middle_size_dist(kbt, corpus, feature_counter)>],\n",
       " 'vectorizer': DictVectorizer(dtype=<class 'numpy.float64'>, separator='=', sort=True,\n",
       "         sparse=True),\n",
       " 'models': {'adjoins': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'author': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'capital': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'contains': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'film_performance': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'founders': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'genre': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'has_sibling': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'has_spouse': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'is_a': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'nationality': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'parents': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'place_of_birth': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'place_of_death': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'profession': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False),\n",
       "  'worked_at': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "            intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "            penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "            verbose=0, warm_start=False)}}"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "experiment(train_split='train', test_split='test',\n",
    "           featurizers = [directional_bag_of_words_featurizer,\n",
    "                            directional_middle_bigram_pos_tag_featurizer,\n",
    "                            synset_featurizer,\n",
    "                            all_glove_dist, middle_size_dist])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}