hardware/max_storage_temp_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Extracting Maximum Storage Temperatures for Transistors from PDF Datasheets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "We will walk through the process of using `Fonduer` to extract relations from [**richly formatted** data](https://hazyresearch.github.io/snorkel/blog/fonduer.html), where information is conveyed via combinations of textual, structural, tabular, and visual expressions, as seen in webpages, business reports, product specifications, and scientific literature.\n",
    "\n",
    "In this tutorial, we use `Fonduer` to identify mentions of the maximum storage temperature of transistors (e.g. `150°C`) in a corpus of transistor datasheets from [Digikey.com](https://www.digikey.com/products/en/discrete-semiconductor-products/transistors-bipolar-bjt-single/276).\n",
    "\n",
    "The tutorial is broken into several parts, each covering a phase of the `Fonduer` pipeline (as outlined in the [paper](https://arxiv.org/abs/1703.05028)), and the iterative KBC process:\n",
    "\n",
    "1. KBC Initialization\n",
    "2. Candidate Generation and Multimodal Featurization\n",
    "3. Probabilistic Relation Classification\n",
    "4. Error Analysis and Iterative KBC\n",
    "\n",
    "In addition, we show how users can iteratively improve labeling functions to improve relation extraction quality.\n",
    "\n",
    "# Phase 1: KBC Initialization\n",
    "\n",
    "In this first phase of `Fonduer`'s pipeline, `Fonduer` uses a user-specified _schema_ to initialize a relational database where the output KB will be stored. Furthermore, `Fonduer` iterates over its input _corpus_ and transforms each document into a unified data model, which captures the variability and multimodality of richly formatted data. This unified data model then serves as an intermediate representation used in the rest of the phases.\n",
    "\n",
    "This preprocessed data is saved to a database. The connection string to the database is provided to the `Meta` object, which will initialize a PostgreSQL database for parallel execution.\n",
    "\n",
    "We initialize several variables for convenience that define what the database should be called and what level of parallelization the `Fonduer` pipeline will be run with.\n",
    "\n",
    "If you're running this tutorial interactively on your own machine, you'll need to create a new PostgreSQL database named `stg_temp_max`.\n",
    "\n",
    "If you already have the database `stg_temp_max` in your postgresql, please uncomment the first line to drop it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#! dropdb --if-exists stg_temp_max\n",
    "! createdb stg_temp_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "import sys\n",
    "import logging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PARALLEL = 4  # assuming a quad-core machine\n",
    "ATTRIBUTE = \"stg_temp_max\"\n",
    "conn_string = 'postgresql://localhost:5432/' + ATTRIBUTE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Parsing and Transforming the Input Documents into Unified Data Models\n",
    "\n",
    "We first initialize a `Meta` object, which manages the connection to the database automatically, and enables us to save intermediate results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer import Meta, init_logging\n",
    "\n",
    "# Configure logging for Fonduer\n",
    "init_logging(log_dir=\"logs\")\n",
    "\n",
    "session = Meta.init(conn_string).Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we load the corpus of datasheets and transform them into the unified data model. Each datasheet has a PDF and HTML representation (in this example, the HTML is created using Adobe Acrobat). Both representations are used in conjunction to create a robust unified data model with textual, structural, tabular, and visual modality information. Note that since each document is independent of each other, we can parse the documents in parallel. We depend on PostgreSQL for this functionality.\n",
    "\n",
    "### Configuring an `HTMLDocPreprocessor`\n",
    "We start by setting the paths to where our documents are stored, and defining a `HTMLDocPreprocessor` to read in the documents found in the specified paths. `max_docs` specified the number of documents to parse. For the sake of this tutorial, we only look at 100 documents.\n",
    "\n",
    "**Note that you need to have run `download_data.sh` before executing these next steps or you won't have the documents needed for the tutorial.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.parser.preprocessors import HTMLDocPreprocessor\n",
    "from fonduer.parser import Parser\n",
    "\n",
    "docs_path = 'data/html/'\n",
    "pdf_path = 'data/pdf/'\n",
    "\n",
    "max_docs = 100\n",
    "doc_preprocessor = HTMLDocPreprocessor(docs_path, max_docs=max_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuring a `Parser`\n",
    "Next, we configure a `Parser`, which serves as our `CorpusParser` for PDF documents. We use [spaCy](https://spacy.io/) as a preprocessing tool to split our documents into sentences and tokens, and to provide annotations such as part-of-speech tags and dependency parse structures for these sentences. In addition, we can specify which modality information to include in the unified data model for each document. Below, we enable all modality information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_parser = Parser(session, structural=True, lingual=True, visual=True, pdf_path=pdf_path)\n",
    "%time corpus_parser.apply(doc_preprocessor, parallelism=PARALLEL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use simple database queries (written in the syntax of [SQLAlchemy](http://www.sqlalchemy.org/), which `Fonduer` uses) to check how many documents and sentences were parsed, or even check how many sentences are contained in each document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.parser.models import Document, Sentence\n",
    "\n",
    "print(f\"Documents: {session.query(Document).count()}\")\n",
    "print(f\"Sentences: {session.query(Sentence).count()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Dividing the Corpus into Test and Train\n",
    "\n",
    "We'll split the documents 80/10/10 into train/dev/test splits. Note that here we do this in a non-random order to preserve the consistency in the tutorial, and we reference the splits by 0/1/2 respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = session.query(Document).order_by(Document.name).all()\n",
    "ld   = len(docs)\n",
    "\n",
    "train_docs = set()\n",
    "dev_docs   = set()\n",
    "test_docs  = set()\n",
    "splits = (0.8, 0.9)\n",
    "data = [(doc.name, doc) for doc in docs]\n",
    "data.sort(key=lambda x: x[0])\n",
    "for i, (doc_name, doc) in enumerate(data):\n",
    "    if i < splits[0] * ld:\n",
    "        train_docs.add(doc)\n",
    "    elif i < splits[1] * ld:\n",
    "        dev_docs.add(doc)\n",
    "    else:\n",
    "        test_docs.add(doc)\n",
    "from pprint import pprint\n",
    "pprint([x.name for x in train_docs])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Phase 2: Mention Extraction, Candidate Extraction Multimodal Featurization\n",
    "\n",
    "Given the unified data model from Phase 1, `Fonduer` extracts relation\n",
    "candidates based on user-provided **matchers** and **throttlers**. Then,\n",
    "`Fonduer` leverages the multimodality information captured in the unified data\n",
    "model to provide multimodal features for each candidate.\n",
    "\n",
    "## 2.1 Mention Extraction\n",
    "\n",
    "The first step is to extract **mentions** from our corpus. A `mention` is the\n",
    "type of object which makes up a `candidate`. For example, if we wanted to\n",
    "extract pairs of transistor part numbers and their corresponding maximum\n",
    "storage temperatures, the transistor part number would be one `mention` while\n",
    "the temperature value would be another. These `mention`s are then combined to\n",
    "create `candidates`, where our task is to predict which `candidates` are true\n",
    "in the associated document.\n",
    "\n",
    "We first start by defining and naming our two `mention`s:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.candidates.models import mention_subclass\n",
    "\n",
    "Part = mention_subclass(\"Part\")\n",
    "Temp = mention_subclass(\"Temp\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we write **matchers** to define which spans of text in the corpus are\n",
    "instances of each entity. Matchers can leverage a variety of information from\n",
    "regular expressions, to dictionaries, to user-defined functions. Furthermore,\n",
    "different techniques can be combined to form higher quality matchers. In\n",
    "general, matchers should seek to be as precise as possible while maintaining\n",
    "complete recall. More documentation about Matchers can be found on [Read the Docs](https://fonduer.readthedocs.io/en/stable/user/candidates.html#matchers).\n",
    "\n",
    "In our case, we need to write a matcher that defines a transistor part number\n",
    "and a matcher to define a valid temperature value.\n",
    "\n",
    "### Writing a simple temperature matcher\n",
    "\n",
    "Our maximum storage temperature matcher can be a very simple regular expression\n",
    "since we know that we are looking for integers, and by inspecting a portion of\n",
    "our corpus, we see that maximum storage temperatures fall within a fairly\n",
    "narrow range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.candidates.matchers import RegexMatchSpan, DictionaryMatch, LambdaFunctionMatcher, Intersect, Union\n",
    "\n",
    "temp_matcher = RegexMatchSpan(rgx=r\"(?:[1][5-9]|20)[05]\", longest_match_only=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing an advanced transistor part matcher\n",
    "\n",
    "In contrast, transistor part numbers are complex expressions. Here, we show how transistor part numbers can leverage [naming conventions](https://en.wikipedia.org/wiki/Transistor#Part_numbering_standards.2Fspecifications) as regular expressions, and use a dictionary of known part numbers, and use user-defined functions together. First, we create a regular expression matcher for standard transistor naming conventions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Transistor Naming Conventions as Regular Expressions ###\n",
    "eeca_rgx = r'([ABC][A-Z][WXYZ]?[0-9]{3,5}(?:[A-Z]){0,5}[0-9]?[A-Z]?(?:-[A-Z0-9]{1,7})?(?:[-][A-Z0-9]{1,2})?(?:\\/DG)?)'\n",
    "jedec_rgx = r'(2N\\d{3,4}[A-Z]{0,5}[0-9]?[A-Z]?)'\n",
    "jis_rgx = r'(2S[ABCDEFGHJKMQRSTVZ]{1}[\\d]{2,4})'\n",
    "others_rgx = r'((?:NSVBC|SMBT|MJ|MJE|MPS|MRF|RCA|TIP|ZTX|ZT|ZXT|TIS|TIPL|DTC|MMBT|SMMBT|PZT|FZT|STD|BUV|PBSS|KSC|CXT|FCX|CMPT){1}[\\d]{2,4}[A-Z]{0,5}(?:-[A-Z0-9]{0,6})?(?:[-][A-Z0-9]{0,1})?)'\n",
    "\n",
    "part_rgx = '|'.join([eeca_rgx, jedec_rgx, jis_rgx, others_rgx])\n",
    "part_rgx_matcher = RegexMatchSpan(rgx=part_rgx, longest_match_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can create a matcher from a dictionary of known part numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "def get_digikey_parts_set(path):\n",
    "    \"\"\"\n",
    "    Reads in the digikey part dictionary and yeilds each part.\n",
    "    \"\"\"\n",
    "    all_parts = set()\n",
    "    with open(path, \"r\") as csvinput:\n",
    "        reader = csv.reader(csvinput)\n",
    "        for line in reader:\n",
    "            (part, url) = line\n",
    "            all_parts.add(part)\n",
    "    return all_parts\n",
    "\n",
    "### Dictionary of known transistor parts ###\n",
    "dict_path = 'data/digikey_part_dictionary.csv'\n",
    "part_dict_matcher = DictionaryMatch(d=get_digikey_parts_set(dict_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use user-defined functions to further improve our matchers. For example, here we use patterns in the document filenames as a signal for whether a span of text in a document is a valid transistor part number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from builtins import range\n",
    "\n",
    "def common_prefix_length_diff(str1, str2):\n",
    "    for i in range(min(len(str1), len(str2))):\n",
    "        if str1[i] != str2[i]:\n",
    "            return min(len(str1), len(str2)) - i\n",
    "    return 0\n",
    "\n",
    "def part_file_name_conditions(attr):\n",
    "    file_name = attr.sentence.document.name\n",
    "    if len(file_name.split('_')) != 2: return False\n",
    "    if attr.get_span()[0] == '-': return False\n",
    "    name = attr.get_span().replace('-', '')\n",
    "    return any(char.isdigit() for char in name) and any(char.isalpha() for char in name) and common_prefix_length_diff(file_name.split('_')[1], name) <= 2\n",
    "\n",
    "add_rgx = '^[A-Z0-9\\-]{5,15}$'\n",
    "\n",
    "part_file_name_lambda_matcher = LambdaFunctionMatcher(func=part_file_name_conditions)\n",
    "part_file_name_matcher = Intersect(RegexMatchSpan(rgx=add_rgx, longest_match_only=True), part_file_name_lambda_matcher)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can union all of these matchers together to form our final part matcher."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "part_matcher = Union(part_rgx_matcher, part_dict_matcher, part_file_name_matcher)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two matchers define each entity in our relation schema."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a Mention's `MentionSpace`\n",
    "\n",
    "Next, in order to define the \"space\" of all mentions that are even considered\n",
    "from the document, we need to define a `MentionSpace` for each component of the\n",
    "relation we wish to extract. Fonduer provides a default `MentionSpace` for you\n",
    "to use, but you can also extend the default `MentionSpace` depending on your\n",
    "needs.\n",
    "\n",
    "In the case of transistor part numbers, the `MentionSpace` can be quite complex\n",
    "due to the need to handle implicit part numbers that are implied in text like\n",
    "\"BC546A/B/C...BC548A/B/C\", which refers to 9 unique part numbers. To handle\n",
    "these, we consider all n-grams up to 3 words long.\n",
    "\n",
    "In contrast, the `MentionSpace` for temperature values is simpler: we only need\n",
    "to process different Unicode representations of a (`-`), and don't need to look\n",
    "at more than two words at a time.\n",
    "\n",
    "When no special preprocessing like this is needed, we could have used the\n",
    "default `Ngrams` class provided by `fonduer`. For example, if we were looking\n",
    "to match polarities, which only take the form of \"NPN\" or \"PNP\", we could've\n",
    "used `ngrams = MentionNgrams(n_max=1)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hardware_spaces import MentionNgramsPart, MentionNgramsTemp\n",
    "    \n",
    "part_ngrams = MentionNgramsPart(parts_by_doc=None, n_max=3)\n",
    "temp_ngrams = MentionNgramsTemp(n_max=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Mention Extraction \n",
    "\n",
    "Next, we create a `MentionExtractor` to extract the mentions from all of\n",
    "our documents based on the `MentionSpace` and matchers we defined above.\n",
    "\n",
    "View the API for the MentionExtractor on [ReadTheDocs](https://fonduer.readthedocs.io/en/stable/user/candidates.html#fonduer.candidates.MentionExtractor).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.candidates import MentionExtractor \n",
    "\n",
    "mention_extractor = MentionExtractor(\n",
    "    session, [Part, Temp], [part_ngrams, temp_ngrams], [part_matcher, temp_matcher]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we run the extractor on all of our documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.candidates.models import Mention\n",
    "\n",
    "mention_extractor.apply(docs, parallelism=PARALLEL)\n",
    "\n",
    "print(f\"Total Mentions: {session.query(Mention).count()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Candidate Extraction\n",
    "\n",
    "Now that we have both defined and extracted the Mentions that can be used to compose Candidates, we are ready to move on to extracting Candidates. Like we did with the Mentions, we first define what each candidate schema looks like. In this example, we create a candidate that is composed of a `Part` and a `Temp` mention as we defined above. We name this candidate \"PartTemp\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.candidates.models import candidate_subclass\n",
    "\n",
    "PartTemp = candidate_subclass(\"PartTemp\", [Part, Temp])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining candidate `Throttlers`\n",
    "\n",
    "Next, we need to define **throttlers**, which allow us to further prune excess candidates and avoid unnecessarily materializing invalid candidates. Throttlers, like matchers, act as hard filters, and should be created to have high precision while maintaining complete recall, if possible.\n",
    "\n",
    "Here, we create a throttler that discards candidates if they are in the same table, but the part and storage temperature are not vertically or horizontally aligned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.utils.data_model_utils import *\n",
    "import re\n",
    "\n",
    "def stg_temp_filter(c):\n",
    "    (part, attr) = c\n",
    "    if same_table((part, attr)):\n",
    "        return (is_horz_aligned((part, attr)) or is_vert_aligned((part, attr)))\n",
    "    return True\n",
    "\n",
    "temp_throttler = stg_temp_filter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the `CandidateExtractor`\n",
    "\n",
    "Now, we have all the component necessary to perform candidate extraction. We have defined the Mentions that compose each candidate and a throttler to prune away excess candidates. We now can define the `CandidateExtractor` with the candidate subclass and corresponding throttler to use.\n",
    "\n",
    "View the API for the CandidateExtractor on [ReadTheDocs](https://fonduer.readthedocs.io/en/stable/user/candidates.html#fonduer.candidates.MentionExtractor)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.candidates import CandidateExtractor\n",
    "\n",
    "\n",
    "candidate_extractor = CandidateExtractor(session, [PartTemp], throttlers=[temp_throttler])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we specified that these `Candidates` belong to the training set by specifying `split=0`; recall that we're referring to train/dev/test as splits 0/1/2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, docs in enumerate([train_docs, dev_docs, test_docs]):\n",
    "    candidate_extractor.apply(docs, split=i, parallelism=PARALLEL)\n",
    "    print(f\"Number of Candidates in split={i}: {session.query(PartTemp).filter(PartTemp.split == i).count()}\")\n",
    "\n",
    "train_cands = candidate_extractor.get_candidates(split = 0)\n",
    "dev_cands = candidate_extractor.get_candidates(split = 1)\n",
    "test_cands = candidate_extractor.get_candidates(split = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Multimodal Featurization\n",
    "Unlike dealing with plain unstructured text, `Fonduer` deals with richly formatted data, and consequently featurizes each candidate with a baseline library of multimodal features. \n",
    "\n",
    "### Featurize with `Fonduer`'s optimized Postgres Featurizer\n",
    "We now annotate the candidates in our training, dev, and test sets with features. The `Featurizer` provided by `Fonduer` allows this to be done in parallel to improve performance.\n",
    "\n",
    "View the API provided by the `Featurizer` on [ReadTheDocs](https://fonduer.readthedocs.io/en/stable/user/features.html#fonduer.features.Featurizer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.features import Featurizer\n",
    "\n",
    "featurizer = Featurizer(session, [PartTemp])\n",
    "%time featurizer.apply(split=0, train=True, parallelism=PARALLEL)\n",
    "%time F_train = featurizer.get_feature_matrices(train_cands)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(F_train[0].shape)\n",
    "%time featurizer.apply(split=1, parallelism=PARALLEL)\n",
    "%time F_dev = featurizer.get_feature_matrices(dev_cands)\n",
    "print(F_dev[0].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time featurizer.apply(split=2, parallelism=PARALLEL)\n",
    "%time F_test = featurizer.get_feature_matrices(test_cands)\n",
    "print(F_test[0].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of this phase, `Fonduer` has generated the set of candidates and the feature matrix. Note that Phase 1 and 2 are relatively static and typically are only executed once during the KBC process.\n",
    "\n",
    "# Phase 3: Probabilistic Relation Classification\n",
    "In this phase, `Fonduer` applies user-defined **labeling functions**, which express various heuristics, patterns, and [weak supervision](http://hazyresearch.github.io/snorkel/blog/weak_supervision.html) strategies to label our data, to each of the candidates to create a label matrix that is used by our data programming engine.\n",
    "\n",
    "In the wild, hand-labeled training data is rare and expensive. A common scenario is to have access to tons of unlabeled training data, and have some idea of how to label them programmatically. For example:\n",
    "* We may be able to think of text patterns that would indicate a part and polarity mention are related, for example the word \"temperature\" appearing between them.\n",
    "* We may have access to an external knowledge base that lists some pairs of parts and polarities, and can use these to noisily label some of our mention pairs.\n",
    "Our labeling functions will capture these types of strategies. We know that these labeling functions will not be perfect, and some may be quite low-quality, so we will model their accuracies with a generative model, which `Fonduer` will help us easily apply.\n",
    "\n",
    "Using data programming, we can then train machine learning models to learn which features are the most important in classifying candidates.\n",
    "\n",
    "### Loading Gold Data\n",
    "For convenience in error analysis and evaluation, we have already annotated the dev and test set for this tutorial, and we'll now load it using an externally-defined helper function: `gold`. Technically, this is also a labeling function similar to labeling functions we will be writing later. The only difference is that (gold) labels are written to `GoldLabel` table while (non-gold) labels are written to `Label` table. If you're interested in the example implementation details, please see the script we now load:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.supervision.models import GoldLabel\n",
    "from hardware_utils import gold\n",
    "\n",
    "from fonduer.supervision import Labeler\n",
    "\n",
    "docs = corpus_parser.get_documents()\n",
    "labeler = Labeler(session, [PartTemp])\n",
    "%time labeler.apply(docs=docs, lfs=[[gold]], table=GoldLabel, train=True, parallelism=PARALLEL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating Labeling Functions\n",
    "\n",
    "In `Fonduer`, our primary interface through which we provide training signal to\n",
    "the end extraction model we are training is by writing labeling functions\n",
    "(**LFs**) (as opposed to hand-labeling massive training sets).\n",
    "\n",
    "A labeling function isn't anything special. It's just a Python function that\n",
    "accepts a `Candidate` as the input argument and returns `1` if it says the\n",
    "Candidate should be marked as true, `0` if it says the `Candidate` should be\n",
    "marked as false, and `-1` if it doesn't know how to vote and abstains. In\n",
    "practice, many labeling functions are unipolar: it labels only 1s and -1s, or it\n",
    "labels only 0s and -1s.\n",
    "\n",
    "Recall that our goal is ultimately to train a high-performance classification\n",
    "model that predicts which of our Candidates are true mentions of maximum\n",
    "storage temperature relations. It turns out that we can do this by writing\n",
    "potentially low-quality labeling functions!\n",
    "\n",
    "With `Fonduer`, labeling functions can be written using intuitive patterns\n",
    "discovered by inspecting the target corpus. A library of data model utilities\n",
    "which can be used to write labeling functions are outline in [Read the\n",
    "Docs](http://fonduer.readthedocs.io/en/stable/user/data_model_utils.html). \n",
    "\n",
    "For example, inspecting several documents may reveal that storage temperatures\n",
    "are typically listed inside a table where the row header contains the word\n",
    "\"storage\". This intuitive pattern can be directly expressed as a labeling\n",
    "function. Similarly, the word \"temperature\" is an obvious positive signal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.labeling import labeling_function\n",
    "from fonduer.utils.data_model_utils import *\n",
    "import re\n",
    "\n",
    "ABSTAIN = -1\n",
    "FALSE = 0\n",
    "TRUE = 1\n",
    "\n",
    "@labeling_function()\n",
    "def LF_storage_row(c):\n",
    "    return TRUE if 'storage' in get_row_ngrams(c.temp) else ABSTAIN\n",
    "\n",
    "@labeling_function()\n",
    "def LF_temperature_row(c):\n",
    "    return TRUE if 'temperature' in get_row_ngrams(c.temp) else ABSTAIN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We express several of these simple patterns below as a set of labeling functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@labeling_function()\n",
    "def LF_operating_row(c):\n",
    "    return TRUE if 'operating' in get_row_ngrams(c.temp) else ABSTAIN\n",
    "\n",
    "@labeling_function()\n",
    "def LF_tstg_row(c):\n",
    "    return TRUE if overlap(\n",
    "        ['tstg','stg','ts'], \n",
    "        list(get_row_ngrams(c.temp))) else ABSTAIN\n",
    "\n",
    "@labeling_function()\n",
    "def LF_to_left(c):\n",
    "    return TRUE if 'to' in get_left_ngrams(c.temp, window=2) else ABSTAIN\n",
    "\n",
    "@labeling_function()\n",
    "def LF_negative_number_left(c):\n",
    "    return TRUE if any([re.match(r'-\\s*\\d+', ngram) for ngram in get_left_ngrams(c.temp, window=4)]) else ABSTAIN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we collect all of the labeling function we would like to use into a single list, which is provided as input to the `Labeler`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stg_temp_lfs = [\n",
    "    LF_storage_row,\n",
    "    LF_operating_row,\n",
    "    LF_temperature_row,\n",
    "    LF_tstg_row,\n",
    "    LF_to_left,\n",
    "    LF_negative_number_left,\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying the Labeling Functions\n",
    "\n",
    "Next, we need to actually run the LFs over all of our training candidates, producing a set of `Labels` and `LabelKeys` (just the names of the LFs) in the database. Note that this will delete any existing `Labels` and `LabelKeys` for this candidate set.\n",
    "\n",
    "View the API provided by the `Labeler` on [ReadTheDocs](https://fonduer.readthedocs.io/en/stable/user/supervision.html#fonduer.supervision.Labeler)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time labeler.apply(split=0, lfs=[stg_temp_lfs], train=True, parallelism=PARALLEL)\n",
    "%time L_train = labeler.get_label_matrices(train_cands)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also view statistics about the resulting label matrix.\n",
    "* **Coverage** is the fraction of candidates that the labeling function emits a non-zero label for.\n",
    "* **Overlap** is the fraction candidates that the labeling function emits a non-zero label for and that another labeling function emits a non-zero label for.\n",
    "* **Conflict** is the fraction candidates that the labeling function emits a non-zero label for and that another labeling function emits a conflicting non-zero label for.\n",
    "\n",
    "In addition, because we have already loaded the gold labels, we can view the emperical accuracy of these labeling functions when compared to our gold labels using the `analysis` module of [Snorkel](https://github.com/snorkel-team/snorkel)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "L_gold_train = labeler.get_gold_labels(train_cands, annotator='gold')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.labeling import LFAnalysis\n",
    "\n",
    "# Sort LFs for LFAnalysis because LFAnalysis does not sort LFs,\n",
    "# while columns of L_train are sorted alphabetically already.\n",
    "sorted_lfs = sorted(stg_temp_lfs, key=lambda lf: lf.name)\n",
    "\n",
    "LFAnalysis(L=L_train[0], lfs=sorted_lfs).lf_summary(Y=L_gold_train[0].reshape(-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitting the Generative Model\n",
    "\n",
    "Now, we'll train a model of the LFs to estimate their accuracies. Once the model is trained, we can combine the outputs of the LFs into a single, noise-aware training label set for our extractor. Intuitively, we'll model the LFs by observing how they overlap and conflict with each other. To do so, we use [Snorkel](https://github.com/snorkel-team/snorkel)'s single-task label model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.labeling.model import LabelModel\n",
    "\n",
    "gen_model = LabelModel(cardinality=2)\n",
    "%time gen_model.fit(L_train[0], n_epochs=500, log_freq=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now apply the generative model to the training candidates to get the noise-aware training label set. We'll refer to these as the training marginals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_marginals = gen_model.predict_proba(L_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll look at the distribution of the training marginals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.hist(train_marginals[:, TRUE], bins=20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the Model to Iterate on Labeling Functions\n",
    "\n",
    "Now that we have learned the generative model, we can stop here and use this to potentially debug and/or improve our labeling function set. First, we apply the LFs to our development set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labeler.apply(split=1, lfs=[stg_temp_lfs], parallelism=PARALLEL)\n",
    "%time L_dev = labeler.get_label_matrices(dev_cands)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "L_dev[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting Generative Model Performance\n",
    "\n",
    "At this point, we should be getting an F1 score of around 0.6 to 0.7 on the development set, which is pretty good! However, we should be very careful in interpreting this. Since we developed our labeling functions using this development set as a guide, and our generative model is composed of these labeling functions, we expect it to score very well here!\n",
    "\n",
    "In fact, it is probably somewhat overfit to this set. However this is fine, since in the next, we'll train a more powerful end extraction model which will generalize beyond the development set, and which we will evaluate on a blind test set (i.e. one we never looked at during development).\n",
    "\n",
    "\n",
    "### Training the Discriminative Model\n",
    "\n",
    "Now, we'll use the noisy training labels we generated in the last part to train our end extraction model. For this tutorial, we will be training a simple--but fairly effective--logistic regression model.\n",
    "\n",
    "We use the training marginals to train a discriminative model that classifies each Candidate as a true or false mention.\n",
    "\n",
    "In `Fonduer`, we use a new machine learning framework [Emmental](https://github.com/SenWu/emmental) to support all model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import emmental\n",
    "\n",
    "# Setup training config\n",
    "config = {\n",
    "    \"meta_config\": {\"verbose\": True},\n",
    "    \"model_config\": {\"model_path\": None, \"device\": 0, \"dataparallel\": False},\n",
    "    \"learner_config\": {\n",
    "        \"n_epochs\": 50,\n",
    "        \"optimizer_config\": {\"lr\": 0.001, \"l2\": 0.0},\n",
    "        \"task_scheduler\": \"round_robin\",\n",
    "    },\n",
    "    \"logging_config\": {\n",
    "        \"evaluation_freq\": 1,\n",
    "        \"counter_unit\": \"epoch\",\n",
    "        \"checkpointing\": False,\n",
    "        \"checkpointer_config\": {\n",
    "            \"checkpoint_metric\": {f\"{ATTRIBUTE}/{ATTRIBUTE}/train/loss\": \"min\"},\n",
    "            \"checkpoint_freq\": 1,\n",
    "            \"checkpoint_runway\": 2,\n",
    "            \"clear_intermediate_checkpoints\": True,\n",
    "            \"clear_all_checkpoints\": True,\n",
    "        },\n",
    "    },\n",
    "}\n",
    "\n",
    "emmental.init(Meta.log_path)\n",
    "emmental.Meta.update_config(config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Collect word counter from training data\n",
    "from fonduer.learning.utils import collect_word_counter\n",
    "\n",
    "word_counter = collect_word_counter(train_cands)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate word embedding module for LSTM model\n",
    "# (in Logistic Regression, we generate it since Fonduer dataset requires word2id dict)\n",
    "from emmental.modules.embedding_module import EmbeddingModule\n",
    "\n",
    "arity = 2\n",
    "\n",
    "# Geneate special tokens\n",
    "specials = []\n",
    "for i in range(arity):\n",
    "    specials += [f\"~~[[{i}\", f\"{i}]]~~\"]\n",
    "\n",
    "emb_layer = EmbeddingModule(\n",
    "    word_counter=word_counter, word_dim=300, specials=specials\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate dataloader for training set\n",
    "from emmental.data import EmmentalDataLoader\n",
    "from fonduer.learning.dataset import FonduerDataset\n",
    "import numpy as np\n",
    "\n",
    "# Filter out noise samples\n",
    "diffs = train_marginals.max(axis=1) - train_marginals.min(axis=1)\n",
    "train_idxs = np.where(diffs > 1e-6)[0]\n",
    "\n",
    "train_dataloader = EmmentalDataLoader(\n",
    "    task_to_label_dict={ATTRIBUTE: \"labels\"},\n",
    "    dataset=FonduerDataset(\n",
    "        ATTRIBUTE,\n",
    "        train_cands[0],\n",
    "        F_train[0],\n",
    "        emb_layer.word2id,\n",
    "        train_marginals,\n",
    "        train_idxs,\n",
    "    ),\n",
    "    split=\"train\",\n",
    "    batch_size=100,\n",
    "    shuffle=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creaet task and model, and perform training\n",
    "from emmental.model import EmmentalModel\n",
    "from fonduer.learning.task import create_task\n",
    "from emmental.learner import EmmentalLearner\n",
    "\n",
    "tasks = create_task(\n",
    "    ATTRIBUTE, 2, F_train[0].shape[1], 2, emb_layer, model=\"LogisticRegression\"\n",
    ")\n",
    "\n",
    "model = EmmentalModel(name=f\"{ATTRIBUTE}_task\")\n",
    "\n",
    "for task in tasks:\n",
    "    model.add_task(task)\n",
    "\n",
    "emmental_learner = EmmentalLearner()\n",
    "emmental_learner.learn(model, [train_dataloader])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating on the Test Set\n",
    "In this final section, we'll get the score we've been after: the performance of the extraction model on the blind test set (split 2). First, we load the test set labels and gold candidates from earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from hardware_utils import entity_level_f1\n",
    "import pickle\n",
    "pickle_file = 'data/parts_by_doc_dict.pkl'\n",
    "with open(pickle_file, 'rb') as f:\n",
    "    parts_by_doc = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we score using the discriminitive model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate dataloader for test data\n",
    "test_dataloader = EmmentalDataLoader(\n",
    "    task_to_label_dict={ATTRIBUTE: \"labels\"},\n",
    "    dataset=FonduerDataset(\n",
    "        ATTRIBUTE, test_cands[0], F_test[0], emb_layer.word2id, 2\n",
    "    ),\n",
    "    split=\"test\",\n",
    "    batch_size=100,\n",
    "    shuffle=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gold_file = 'data/hardware_tutorial_gold.csv'\n",
    "\n",
    "test_preds = model.predict(test_dataloader, return_preds=True)\n",
    "positive = np.where(np.array(test_preds[\"probs\"][ATTRIBUTE])[:, TRUE] > 0.6)\n",
    "true_pred = [test_cands[0][_] for _ in positive[0]]\n",
    "%time (TP, FP, FN) = entity_level_f1(true_pred, gold_file, ATTRIBUTE, test_docs, parts_by_doc=parts_by_doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Phase 4:  Error Analysis & Iterative KBC\n",
    "\n",
    "During the development process, we can iteratively improve the quality of our labeling functions through error analysis, without executing the full pipeline as in previous techniques. \n",
    "\n",
    "You may have noticed that our final score is about 50 F1 points. To remedy this and improve our quality, we can perform error analysis to understand what kinds of patterns we may have missed, or what issues exist with our labeling functions. Then, we can edit our set of labeling functions and rerun Phase 3, Probabilistic Relation Classification. \n",
    "\n",
    "## Error Analysis\n",
    "For example, notice that our `entity_level_f1` returns `TP`, `FP`, `FN` sets. We can also see that our recall is high, but we have low precision, so let's look at our false positivies, `FP`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there are actually only a few documents that are causing us problems. In particular, we see that `BC546-D` is giving us many false positives. So, let's inspect one of those candidates. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fonduer.utils.visualizer import Visualizer\n",
    "from hardware_utils import entity_to_candidates\n",
    "vis = Visualizer(pdf_path)\n",
    "\n",
    "# Get a list of candidates that match the FN[10] entity\n",
    "fp_cands = entity_to_candidates(FP[10], test_cands[0])\n",
    "# Display a candidate\n",
    "vis.display_candidates([fp_cands[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the candidates are boxed in blue. We see that the temperature falls within the range of numbers that our matcher for storage temperature allows to match. By inspecting candidates like this, or just by examining the problematic PDFs directly, we can notice some patterns that we can exploit as new labeling functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a list of candidates that match the FN[10] entity\n",
    "fp_cands = entity_to_candidates(FP[2], test_cands[0])\n",
    "\n",
    "# # Display this candidate\n",
    "vis.display_candidates([fp_cands[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iteratively Improving Labeling Functions\n",
    "\n",
    "From this error analysis, we may notice two important things. First, our original set of labeling functions had no labeling functions that labeled candidates a negative. This resulted in most skewing the models to accept most candidates, and hurt our precision. Second, we have now noticed that we need to focus on negatively labeling numbers that pass through our storage temperature matchers, but are not related to storage temperature.\n",
    "\n",
    "Below are a set of negative labeling functions that capture some of these patterns. For example, we label candidates an negative if the number is aligned with attributes that are not related to storage temperature, if a candidate represents a typical value, rather than a maximum value, if a temperature value is found outside of a table, and other intuitive patterns we noticed when carefully inspecting our false positives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import chain\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_test_condition_aligned(c):\n",
    "    return (\n",
    "        FALSE\n",
    "        if overlap([\"test\", \"condition\"], list(get_aligned_ngrams(c.temp)))\n",
    "        else ABSTAIN\n",
    "    )\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_collector_aligned(c):\n",
    "    return (\n",
    "        FALSE\n",
    "        if overlap(\n",
    "            [\"collector\", \"collector-current\", \"collector-base\", \"collector-emitter\"],\n",
    "            list(get_aligned_ngrams(c.temp)),\n",
    "        )\n",
    "        else ABSTAIN\n",
    "    )\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_current_aligned(c):\n",
    "    return (\n",
    "        FALSE\n",
    "        if overlap([\"current\", \"dc\", \"ic\"], list(get_aligned_ngrams(c.temp)))\n",
    "        else ABSTAIN\n",
    "    )\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_voltage_row_temp(c):\n",
    "    return (\n",
    "        FALSE\n",
    "        if overlap(\n",
    "            [\"voltage\", \"cbo\", \"ceo\", \"ebo\", \"v\"], list(get_aligned_ngrams(c.temp))\n",
    "        )\n",
    "        else ABSTAIN\n",
    "    )\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_voltage_row_part(c):\n",
    "    return (\n",
    "        FALSE\n",
    "        if overlap(\n",
    "            [\"voltage\", \"cbo\", \"ceo\", \"ebo\", \"v\"], list(get_aligned_ngrams(c.temp))\n",
    "        )\n",
    "        else ABSTAIN\n",
    "    )\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_typ_row(c):\n",
    "    return FALSE if overlap([\"typ\", \"typ.\"], list(get_row_ngrams(c.temp))) else ABSTAIN\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_complement_left_row(c):\n",
    "    return (\n",
    "        FALSE\n",
    "        if (\n",
    "            overlap(\n",
    "                [\"complement\", \"complementary\"],\n",
    "                chain.from_iterable(\n",
    "                    [get_row_ngrams(c.part), get_left_ngrams(c.part, window=10)]\n",
    "                ),\n",
    "            )\n",
    "        )\n",
    "        else ABSTAIN\n",
    "    )\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_too_many_numbers_row(c):\n",
    "    num_numbers = list(get_row_ngrams(c.temp, attrib=\"ner_tags\")).count(\"number\")\n",
    "    return FALSE if num_numbers >= 3 else ABSTAIN\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_temp_on_high_page_num(c):\n",
    "    return FALSE if c.temp.context.get_attrib_tokens(\"page\")[0] > 2 else ABSTAIN\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_temp_outside_table(c):\n",
    "    return FALSE if not c.temp.context.sentence.is_tabular() is None else ABSTAIN\n",
    "\n",
    "\n",
    "@labeling_function()\n",
    "def LF_not_temp_relevant(c):\n",
    "    return (\n",
    "        FALSE\n",
    "        if not overlap(\n",
    "            [\"storage\", \"temperature\", \"tstg\", \"stg\", \"ts\"],\n",
    "            list(get_aligned_ngrams(c.temp)),\n",
    "        )\n",
    "        else ABSTAIN\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can add these to our list of labeling functions.\n",
    "Note that `LF_too_many_numbers_row` is not included in `stg_temp_lfs_2` because this LF gives `ABSTAIN` to all the candidates in `train_cands`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stg_temp_lfs_2 = [\n",
    "    LF_test_condition_aligned,\n",
    "    LF_collector_aligned,\n",
    "    LF_current_aligned,\n",
    "    LF_voltage_row_temp,\n",
    "    LF_voltage_row_part,\n",
    "    LF_typ_row,\n",
    "    LF_complement_left_row,\n",
    "#     LF_too_many_numbers_row,\n",
    "    LF_temp_on_high_page_num,\n",
    "    LF_temp_outside_table,\n",
    "    LF_not_temp_relevant,\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And rerun labeling. Importantly, this time we use the `.update()` function to reflect the fact that we are adding new labeling functions, but do not want to throw away the computations already performed in the previous iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time labeler.update(split=0, lfs=[stg_temp_lfs_2], parallelism=PARALLEL)\n",
    "%time L_train = labeler.get_label_matrices(train_cands)\n",
    "print(L_train[0].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "LFAnalysis(L=L_train[0], lfs=sorted(stg_temp_lfs + stg_temp_lfs_2, key=lambda lf: lf.name)).lf_summary(Y=L_gold_train[0].reshape(-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can rerun probablistic relation classification, the same way we did above. We start with the generative model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gen_model = LabelModel(cardinality=2)\n",
    "\n",
    "%time gen_model.fit(L_train[0], n_epochs=500, log_freq=100)\n",
    "train_marginals = gen_model.predict_proba(L_train[0])\n",
    "plt.hist(train_marginals[:, TRUE], bins=20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we rerun the discriminitive model and see that our score has improved significantly to about 80 F1 points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Regenerate dataloader for training data based on new train_marginals\n",
    "diffs = train_marginals.max(axis=1) - train_marginals.min(axis=1)\n",
    "train_idxs = np.where(diffs > 1e-6)[0]\n",
    "\n",
    "train_dataloader = EmmentalDataLoader(\n",
    "    task_to_label_dict={ATTRIBUTE: \"labels\"},\n",
    "    dataset=FonduerDataset(\n",
    "        ATTRIBUTE,\n",
    "        train_cands[0],\n",
    "        F_train[0],\n",
    "        emb_layer.word2id,\n",
    "        train_marginals,\n",
    "        train_idxs,\n",
    "    ),\n",
    "    split=\"train\",\n",
    "    batch_size=100,\n",
    "    shuffle=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrain the model\n",
    "emmental.Meta.reset()\n",
    "emmental.init(Meta.log_path)\n",
    "emmental.Meta.update_config(config=config)\n",
    "\n",
    "tasks = create_task(\n",
    "    ATTRIBUTE, 2, F_train[0].shape[1], 2, emb_layer, model=\"LogisticRegression\"\n",
    ")\n",
    "\n",
    "model = EmmentalModel(name=f\"{ATTRIBUTE}_task\")\n",
    "\n",
    "for task in tasks:\n",
    "    model.add_task(task)\n",
    "\n",
    "emmental_learner = EmmentalLearner()\n",
    "emmental_learner.learn(model, [train_dataloader])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_preds = model.predict(test_dataloader, return_preds=True)\n",
    "positive = np.where(np.array(test_preds[\"probs\"][ATTRIBUTE])[:, TRUE] > 0.6)\n",
    "true_pred = [test_cands[0][_] for _ in positive[0]]\n",
    "%time (TP, FP, FN) = entity_level_f1(true_pred, gold_file, ATTRIBUTE, test_docs, parts_by_doc=parts_by_doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using these new LFs, we've significantly improved precision and lowered our number of false positives for an F1 score of about 80."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}