tutorials/WiC_slicing_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Snorkel Workshop: Slicing Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "To start, let's make sure that we have the right paths/environment variables set by following the instructions in `snorkel-superglue/README.md`.\n",
    "\n",
    "Specifically, ensure that (1) `snorkel` is installed and (2) `SUPERGLUEDATA` is set where [download_superglue_data.py](https://github.com/HazyResearch/snorkel-superglue/blob/staging/download_superglue_data.py) was called."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys, os\n",
    "from pathlib import Path\n",
    "\n",
    "if not \"cwd\" in globals():\n",
    "    cwd = Path(os.getcwd())\n",
    "sys.path.insert(0, str(cwd.parents[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "# Don't truncate the sentence when viewing examples\n",
    "pd.set_option('display.max_colwidth', -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, we rely heavily on the `snorkel.mtl` module, which is a great abstraction for implementing these slicing tasks. \n",
    "Intuitively, we want an API to add extra model capacity corresponding to each slice—exactly what this module flexibly provides!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.mtl.data import MultitaskDataLoader\n",
    "from snorkel.mtl.model import MultitaskModel\n",
    "from snorkel.mtl.snorkel_config import default_config as config\n",
    "from snorkel.mtl.trainer import Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import superglue_tasks\n",
    "from tokenizer import get_tokenizer\n",
    "from utils import task_dataset_to_dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the WiC dataset\n",
    "We'll be working with the [Words in Context (WiC) task](https://pilehvar.github.io/wic/). To start, let's look at a few examples. To do so, we'll convert them to dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataloaders import get_jsonl_path\n",
    "from superglue_parsers.wic import get_rows\n",
    "\n",
    "task_name = \"WiC\"\n",
    "data_dir = os.environ.get(\"SUPERGLUEDATA\", os.path.join(str(cwd.parents[0]), \"data\"))\n",
    "split = \"valid\"\n",
    "max_data_samples = None # max examples to include in dataset\n",
    "\n",
    "jsonl_path = get_jsonl_path(data_dir, task_name, split)\n",
    "wic_df = pd.DataFrame.from_records(get_rows(jsonl_path, max_data_samples=max_data_samples))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall, the WiC task is used to identify the intended meaning of specified words across multiple contexts—the `label` indicates whether the word is used in the same sense in both `sentence1` and `sentence2`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence1</th>\n",
       "      <th>sentence2</th>\n",
       "      <th>word</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Room and board .</td>\n",
       "      <td>He nailed boards across the windows .</td>\n",
       "      <td>board</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Circulate a rumor .</td>\n",
       "      <td>This letter is being circulated among the faculty .</td>\n",
       "      <td>circulate</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hook a fish .</td>\n",
       "      <td>He hooked a snake accidentally , and was so scared he dropped his rod into the water .</td>\n",
       "      <td>hook</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>For recreation he wrote poetry and solved crossword puzzles .</td>\n",
       "      <td>Drug abuse is often regarded as a form of recreation .</td>\n",
       "      <td>recreation</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Making a hobby of domesticity .</td>\n",
       "      <td>A royal family living in unpretentious domesticity .</td>\n",
       "      <td>domesticity</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                       sentence1  \\\n",
       "0  Room and board .                                                \n",
       "1  Circulate a rumor .                                             \n",
       "2  Hook a fish .                                                   \n",
       "3  For recreation he wrote poetry and solved crossword puzzles .   \n",
       "4  Making a hobby of domesticity .                                 \n",
       "\n",
       "                                                                                sentence2  \\\n",
       "0  He nailed boards across the windows .                                                    \n",
       "1  This letter is being circulated among the faculty .                                      \n",
       "2  He hooked a snake accidentally , and was so scared he dropped his rod into the water .   \n",
       "3  Drug abuse is often regarded as a form of recreation .                                   \n",
       "4  A royal family living in unpretentious domesticity .                                     \n",
       "\n",
       "          word  label  \n",
       "0  board        False  \n",
       "1  circulate    False  \n",
       "2  hook         True   \n",
       "3  recreation   True   \n",
       "4  domesticity  False  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wic_df[[\"sentence1\", \"sentence2\", \"word\", \"label\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a model using BERT\n",
    "Now, let's train a model using the Snorkel API, with the [BERT](https://arxiv.org/abs/1810.04805) model, a powerful pre-training mechanism for general language understanding.\n",
    "Thanks to folks at [huggingface](https://github.com/huggingface/pytorch-pretrained-BERT), we can use this model in PyTorch with with a simple import statement!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_model = \"bert-large-cased\"\n",
    "tokenizer_name = \"bert-large-cased\"\n",
    "batch_size = 4\n",
    "max_sequence_length = 256"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the word-piece tokenizer for the 'bert-large-cased' vocabulary\n",
    "tokenizer = get_tokenizer(tokenizer_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the style of the Snorkel `Multitask` tutorial, we'll use a few helpers to load them into PyTorch datasets that we wrap with a `MultitaskDataLoader`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataloaders import get_dataset\n",
    "\n",
    "datasets = []\n",
    "dataloaders = []\n",
    "for split in [\"train\", \"valid\"]:\n",
    "    # parse raw data and format it as a Pytorch dataset\n",
    "    dataset = get_dataset(\n",
    "        data_dir, task_name, split, tokenizer, max_data_samples, max_sequence_length\n",
    "    )\n",
    "    dataloader = MultitaskDataLoader(\n",
    "        task_to_label_dict={task_name: \"labels\"},\n",
    "        dataset=dataset,\n",
    "        split=split,\n",
    "        batch_size=batch_size,\n",
    "        shuffle=(split == \"train\"),\n",
    "    )\n",
    "    datasets.append(dataset)\n",
    "    dataloaders.append(dataloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model is fairly simple, and identical to the baseline model suggested by the SuperGLUE creators. We feed both sentences through a pre-trained BERT module, then concatenate the output of its classification token with the final representation of the target token (the word whose sense we're disambiguating) in each sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Task(name=WiC)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Construct base task\n",
    "base_task = superglue_tasks.task_funcs[task_name](bert_model)\n",
    "tasks = [base_task]\n",
    "tasks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = MultitaskModel(\n",
    "    name=f\"SuperGLUE\",\n",
    "    tasks=tasks, \n",
    "    dataparallel=False,\n",
    "    device=-1 # use CPU\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've pretrained a model for you, but feel free to uncomment this line to experiment with it yourself!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# trainer = Trainer(**config)\n",
    "# trainer.train_model(slice_model, dataloaders)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-08-03 02:32:48--  https://www.dropbox.com/s/vix9bhzy18o3wjl/WiC_bert.pth\n",
      "Resolving www.dropbox.com (www.dropbox.com)... 162.125.1.1, 2620:100:6016:1::a27d:101\n",
      "Connecting to www.dropbox.com (www.dropbox.com)|162.125.1.1|:443... connected.\n",
      "HTTP request sent, awaiting response... 301 Moved Permanently\n",
      "Location: /s/raw/vix9bhzy18o3wjl/WiC_bert.pth [following]\n",
      "--2019-08-03 02:32:48--  https://www.dropbox.com/s/raw/vix9bhzy18o3wjl/WiC_bert.pth\n",
      "Reusing existing connection to www.dropbox.com:443.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://uc5fbf1e57c6929ec88de0a85822.dl.dropboxusercontent.com/cd/0/inline/Al7aymtIUKN8POR4ob4DribD1koqNAU_hr0282L7a9GA7cFoSj629Q9q1DX7EIDdaeuFO3vNOQk7PDQdvd6hvzV8dN7wEtxHWuLSCumWVvYZ2A/file# [following]\n",
      "--2019-08-03 02:32:49--  https://uc5fbf1e57c6929ec88de0a85822.dl.dropboxusercontent.com/cd/0/inline/Al7aymtIUKN8POR4ob4DribD1koqNAU_hr0282L7a9GA7cFoSj629Q9q1DX7EIDdaeuFO3vNOQk7PDQdvd6hvzV8dN7wEtxHWuLSCumWVvYZ2A/file\n",
      "Resolving uc5fbf1e57c6929ec88de0a85822.dl.dropboxusercontent.com (uc5fbf1e57c6929ec88de0a85822.dl.dropboxusercontent.com)... 162.125.1.6, 2620:100:6016:6::a27d:106\n",
      "Connecting to uc5fbf1e57c6929ec88de0a85822.dl.dropboxusercontent.com (uc5fbf1e57c6929ec88de0a85822.dl.dropboxusercontent.com)|162.125.1.6|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 FOUND\n",
      "Location: /cd/0/inline2/Al7PburfRR6N1Vf7oK23g3yqxiOJMht6SBoMocy1un04fX-jTb_541KY31jRU4mAHSpD-ugX64sjgvykWM5DIKBPm77Ssv_kh9lyCkut5jrct5jBfj_MyBYnbZ1ZzuOfVKIfP6GoHySpFGCTL2NQme6RKZ5J7ub8bD-C-yDihxydJMREeGomVddA40CBL8XzYRKIB9tPeb4V3XvA9Hnwu0kMF6qGwex7Nv7xjHmK51aAJh2S_cqIZlBYnIOy-5sY4zwk0ZpH1Q5pMCn50GTDn-n9ypQbENNqSTKqoCpHgxxk1e0ZvL9YPcqCzkBpxcVFnJ7N_U0aIajJQmuWEEy98BHA/file [following]\n",
      "--2019-08-03 02:32:50--  https://uc5fbf1e57c6929ec88de0a85822.dl.dropboxusercontent.com/cd/0/inline2/Al7PburfRR6N1Vf7oK23g3yqxiOJMht6SBoMocy1un04fX-jTb_541KY31jRU4mAHSpD-ugX64sjgvykWM5DIKBPm77Ssv_kh9lyCkut5jrct5jBfj_MyBYnbZ1ZzuOfVKIfP6GoHySpFGCTL2NQme6RKZ5J7ub8bD-C-yDihxydJMREeGomVddA40CBL8XzYRKIB9tPeb4V3XvA9Hnwu0kMF6qGwex7Nv7xjHmK51aAJh2S_cqIZlBYnIOy-5sY4zwk0ZpH1Q5pMCn50GTDn-n9ypQbENNqSTKqoCpHgxxk1e0ZvL9YPcqCzkBpxcVFnJ7N_U0aIajJQmuWEEy98BHA/file\n",
      "Reusing existing connection to uc5fbf1e57c6929ec88de0a85822.dl.dropboxusercontent.com:443.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1334440755 (1.2G) [application/octet-stream]\n",
      "Saving to: ‘WiC_bert.pth’\n",
      "\n",
      "WiC_bert.pth        100%[===================>]   1.24G  36.3MB/s    in 35s     \n",
      "\n",
      "2019-08-03 02:33:25 (36.4 MB/s) - ‘WiC_bert.pth’ saved [1334440755/1334440755]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# If you're missing the model, uncomment this line:\n",
    "# ! wget -nc https://www.dropbox.com/s/vix9bhzy18o3wjl/WiC_bert.pth\n",
    "\n",
    "# wic_path = \"WiC_bert.pth\"\n",
    "# model.load(wic_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How well do we do on the valid set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6min 49s, sys: 980 ms, total: 6min 50s\n",
      "Wall time: 4min 14s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'WiC/SuperGLUE/valid/accuracy': 0.7460815047021944}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "model.score(dataloaders[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error analysis (to give us ideas for slicing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The key to debugging machine learning models---error analysis! let's look at a few examples that we get wrong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6min 25s, sys: 1.71 s, total: 6min 26s\n",
      "Wall time: 3min 54s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "results = model.predict(dataloaders[1], return_preds=True)\n",
    "golds, preds = results[\"golds\"][task_name], results[\"preds\"][task_name]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence1</th>\n",
       "      <th>sentence2</th>\n",
       "      <th>word</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Circulate a rumor .</td>\n",
       "      <td>This letter is being circulated among the faculty .</td>\n",
       "      <td>circulate</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Making a hobby of domesticity .</td>\n",
       "      <td>A royal family living in unpretentious domesticity .</td>\n",
       "      <td>domesticity</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>The child 's acquisition of language .</td>\n",
       "      <td>That graphite tennis racquet is quite an acquisition .</td>\n",
       "      <td>acquisition</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>They swam in the nude .</td>\n",
       "      <td>The marketing rule ' nude sells ' spread from verbal to visual mainstream media in the 20th century .</td>\n",
       "      <td>nude</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>He took the manuscript in both hands and gave it a mighty tear .</td>\n",
       "      <td>There were big tears rolling down Lisa 's cheeks .</td>\n",
       "      <td>tear</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                           sentence1  \\\n",
       "1   Circulate a rumor .                                                \n",
       "4   Making a hobby of domesticity .                                    \n",
       "5   The child 's acquisition of language .                             \n",
       "7   They swam in the nude .                                            \n",
       "16  He took the manuscript in both hands and gave it a mighty tear .   \n",
       "\n",
       "                                                                                                sentence2  \\\n",
       "1   This letter is being circulated among the faculty .                                                     \n",
       "4   A royal family living in unpretentious domesticity .                                                    \n",
       "5   That graphite tennis racquet is quite an acquisition .                                                  \n",
       "7   The marketing rule ' nude sells ' spread from verbal to visual mainstream media in the 20th century .   \n",
       "16  There were big tears rolling down Lisa 's cheeks .                                                      \n",
       "\n",
       "           word  label  \n",
       "1   circulate    False  \n",
       "4   domesticity  False  \n",
       "5   acquisition  False  \n",
       "7   nude         False  \n",
       "16  tear         False  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "incorrect_preds = golds != preds\n",
    "wic_df[incorrect_preds][[\"sentence1\", \"sentence2\", \"word\", \"label\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We notice that one particular error mode occurs when the target **word** is a _verb_. Let's investigate further...\n",
    "\n",
    "We view examples where we make the wrong prediction _and_ the target word is a verb."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence1</th>\n",
       "      <th>sentence2</th>\n",
       "      <th>word</th>\n",
       "      <th>pos</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Circulate a rumor .</td>\n",
       "      <td>This letter is being circulated among the faculty .</td>\n",
       "      <td>circulate</td>\n",
       "      <td>V</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>To clutch power .</td>\n",
       "      <td>She clutched her purse .</td>\n",
       "      <td>clutch</td>\n",
       "      <td>V</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>She used to wait down at the Dew Drop Inn .</td>\n",
       "      <td>Wait here until your car arrives .</td>\n",
       "      <td>wait</td>\n",
       "      <td>V</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>Wear gloves so your hands stay warm .</td>\n",
       "      <td>Stay with me , please .</td>\n",
       "      <td>stay</td>\n",
       "      <td>V</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>You need to push quite hard to get this door open .</td>\n",
       "      <td>Nora pushed through the crowd .</td>\n",
       "      <td>push</td>\n",
       "      <td>V</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              sentence1  \\\n",
       "1   Circulate a rumor .                                   \n",
       "45  To clutch power .                                     \n",
       "62  She used to wait down at the Dew Drop Inn .           \n",
       "78  Wear gloves so your hands stay warm .                 \n",
       "83  You need to push quite hard to get this door open .   \n",
       "\n",
       "                                              sentence2       word pos  label  \n",
       "1   This letter is being circulated among the faculty .  circulate  V   False  \n",
       "45  She clutched her purse .                             clutch     V   True   \n",
       "62  Wait here until your car arrives .                   wait       V   False  \n",
       "78  Stay with me , please .                              stay       V   True   \n",
       "83  Nora pushed through the crowd .                      push       V   True   "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_is_verb = wic_df[\"pos\"] == \"V\"\n",
    "df_wrong_and_target_is_verb = wic_df[incorrect_preds & target_is_verb]\n",
    "df_wrong_and_target_is_verb[[\"sentence1\", \"sentence2\", \"word\", \"pos\", \"label\"]].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3765432098765432"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df_wrong_and_target_is_verb) / len(wic_df[incorrect_preds])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This error mode accounts for over **37%** of our incorrect predictions! Let's address with _slicing_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Write slicing functions\n",
    "We write slicing functions to target specific subsets of the data that we care about—this could correspond to the examples we find underperforming in an error analysis, or specific subsets that are application critical (e.g. night-time images in a self-driving dataset). Then, we'd like to add slice-specific capacity to our model so that it pays more attention to these examples!\n",
    "\n",
    "We build our slicing functions in the same way that we write labeling functions—with a decorator: `@slicing_function()`. These slicing functions can also be passed previously defined preprocessors, resources, etc. that the slicing function depends on it—just like with labeling fucntions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.slicing.sf import slicing_function\n",
    "from snorkel.types import DataPoint\n",
    "\n",
    "@slicing_function()\n",
    "def SF_verb(x: DataPoint) -> int:\n",
    "    return x.pos == 'V'\n",
    "\n",
    "slicing_functions = [SF_verb]\n",
    "slice_names = [sf.name for sf in slicing_functions]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a _slice-aware_ model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's update our tasks to add _additional capacity_ corresponding to each slice we've specified.\n",
    "\n",
    "For each slice, the model will contain two \"task heads\" (PyTorch modules):\n",
    "- The \"indicator head\" is trained to classify whether each example is a member of that slice or not (a binary problem)\n",
    "- The \"predictor head\" is trained on the base task using only those examples that were identified as being in the slice, so it becomes an effective expert on those examples.\n",
    "\n",
    "At a high level, the helper method `convert_to_slicing_tasks()` will take an existing task and create the following:\n",
    "- Two task heads (ind + pred) for the \"base slice,\" which all examples belong to\n",
    "- Two task heads (ind + pred) for each slice you specified with a slicing function\n",
    "- A new \"master head\" that makes predictions for the main task while taking advantage of information learned by the slice-specific task heads.\n",
    "\n",
    "For each example, the indicator heads specify whether that example is in their slice or not. \n",
    "The magnitude of the predictor head output is used as a proxy for the slice-specific classifier's confidence.\n",
    "These two scores are multiplied together to make a weighted combination of the representations learned by each of the predictor heads. \n",
    "It is this reweighted representation (which accentuates those features that are most relevant to making good predictions on members of those slices) that is used by the master head to make the final prediction. \n",
    "\n",
    "Note that this plays nicely into our MTL abstraction—additional tasks are easy to pop on and off our network, and they allow us to provide \"spot\" capacity to target and improve performance on particular subsets of our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Task(name=WiC_slice:SF_verb_ind),\n",
       " Task(name=WiC_slice:base_ind),\n",
       " Task(name=WiC_slice:SF_verb_pred),\n",
       " Task(name=WiC_slice:base_pred),\n",
       " Task(name=WiC)]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from snorkel.slicing.utils import convert_to_slice_tasks\n",
    "\n",
    "slice_tasks = convert_to_slice_tasks(base_task, slice_names)\n",
    "slice_tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then update our dataloaders to include the label sets for these slices so that those heads can be trained as well in addition to the overall task head."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5428/5428 [00:00<00:00, 35696.04it/s]\n",
      "100%|██████████| 638/638 [00:00<00:00, 31381.53it/s]\n"
     ]
    }
   ],
   "source": [
    "from snorkel.slicing.apply import PandasSFApplier\n",
    "from snorkel.slicing.utils import add_slice_labels\n",
    "\n",
    "slice_dataloaders = []\n",
    "applier = PandasSFApplier(slicing_functions)\n",
    "\n",
    "for dl in dataloaders:\n",
    "    df = task_dataset_to_dataframe(dl.dataset)\n",
    "    S_matrix = applier.apply(df)\n",
    "    # updates dataloaders in place\n",
    "    add_slice_labels(dl, base_task, S_matrix, slice_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We initialize a new _slice-aware model_, and train!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "slice_model = MultitaskModel(\n",
    "    name=f\"SuperGLUE\", \n",
    "    tasks=slice_tasks, \n",
    "    dataparallel=False,\n",
    "    device=-1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we've loaded a pretrained model for you to explore on your own, but you can explore training if you'd like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# trainer = Trainer(**config)\n",
    "# trainer.train_model(slice_model, dataloaders)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2019-08-03 02:42:34--  https://www.dropbox.com/s/h6620vfeompgu9o/WiC_slice_verb.pth\n",
      "Resolving www.dropbox.com (www.dropbox.com)... 162.125.1.1, 2620:100:6016:1::a27d:101\n",
      "Connecting to www.dropbox.com (www.dropbox.com)|162.125.1.1|:443... connected.\n",
      "HTTP request sent, awaiting response... 301 Moved Permanently\n",
      "Location: /s/raw/h6620vfeompgu9o/WiC_slice_verb.pth [following]\n",
      "--2019-08-03 02:42:35--  https://www.dropbox.com/s/raw/h6620vfeompgu9o/WiC_slice_verb.pth\n",
      "Reusing existing connection to www.dropbox.com:443.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://uc961666a77e61c224a3d20242f4.dl.dropboxusercontent.com/cd/0/inline/Al7vz4cwvR11sKbF4BUlGxmTu8vQtg7g7XagSOrG-wAzgNiEhjM8_j_lazTfoIGaak6nWjNxNY7mCUfVK9EqoB43JP2PwyrRibH40Rpuog3R3Q/file# [following]\n",
      "--2019-08-03 02:42:35--  https://uc961666a77e61c224a3d20242f4.dl.dropboxusercontent.com/cd/0/inline/Al7vz4cwvR11sKbF4BUlGxmTu8vQtg7g7XagSOrG-wAzgNiEhjM8_j_lazTfoIGaak6nWjNxNY7mCUfVK9EqoB43JP2PwyrRibH40Rpuog3R3Q/file\n",
      "Resolving uc961666a77e61c224a3d20242f4.dl.dropboxusercontent.com (uc961666a77e61c224a3d20242f4.dl.dropboxusercontent.com)... 162.125.1.6, 2620:100:6016:6::a27d:106\n",
      "Connecting to uc961666a77e61c224a3d20242f4.dl.dropboxusercontent.com (uc961666a77e61c224a3d20242f4.dl.dropboxusercontent.com)|162.125.1.6|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 FOUND\n",
      "Location: /cd/0/inline2/Al6ESJLKXJ9eHJZxmSypkMHBsK8cVZ_k3ttNkwxnOnzs3t6V9pR9vwBbmNJBA29RQ1Mu--GcFxYD7Ydr6E69v9xTwPf7zP1sGxJRdBEZjIIcl69i1n3C5dPKpZcUejbbIOx3Hs4ayXTzq-WVAuJgcFLJzF4zsVdNplmQsBbu55TG52c28Zyxc-7qkKmpgYf5cP02lc5krrNO8rDFWx_WhyWzz5pHZY2RyeRWYa2qDLV3aDG_oV2CoCpKIZiNNNrdBUz-QnTCEDLYe8SDnj1zyX_lsN6FNNCmPpsJzBT0L0Hi8trFbSXtXBpGKCa8itRlKVPy25GcKPBpEJzXXuviNayU/file [following]\n",
      "--2019-08-03 02:42:36--  https://uc961666a77e61c224a3d20242f4.dl.dropboxusercontent.com/cd/0/inline2/Al6ESJLKXJ9eHJZxmSypkMHBsK8cVZ_k3ttNkwxnOnzs3t6V9pR9vwBbmNJBA29RQ1Mu--GcFxYD7Ydr6E69v9xTwPf7zP1sGxJRdBEZjIIcl69i1n3C5dPKpZcUejbbIOx3Hs4ayXTzq-WVAuJgcFLJzF4zsVdNplmQsBbu55TG52c28Zyxc-7qkKmpgYf5cP02lc5krrNO8rDFWx_WhyWzz5pHZY2RyeRWYa2qDLV3aDG_oV2CoCpKIZiNNNrdBUz-QnTCEDLYe8SDnj1zyX_lsN6FNNCmPpsJzBT0L0Hi8trFbSXtXBpGKCa8itRlKVPy25GcKPBpEJzXXuviNayU/file\n",
      "Reusing existing connection to uc961666a77e61c224a3d20242f4.dl.dropboxusercontent.com:443.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1410039344 (1.3G) [application/octet-stream]\n",
      "Saving to: ‘WiC_slice_verb.pth’\n",
      "\n",
      "WiC_slice_verb.pth  100%[===================>]   1.31G  35.3MB/s    in 39s     \n",
      "\n",
      "2019-08-03 02:43:15 (34.4 MB/s) - ‘WiC_slice_verb.pth’ saved [1410039344/1410039344]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# If you're missing the model, uncomment this line:\n",
    "# ! wget -nc https://www.dropbox.com/s/h6620vfeompgu9o/WiC_slice_verb.pth\n",
    "\n",
    "# slice_wic_path = \"WiC_slice_verb.pth\"\n",
    "# slice_model.load(slice_wic_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate _slice-aware_ model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/snorkel-superglue/.env/lib/python3.6/site-packages/snorkel/slicing/modules/slice_combiner.py:40: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  for slice_ind_name in slice_ind_op_names\n",
      "/home/ubuntu/snorkel-superglue/.env/lib/python3.6/site-packages/snorkel/slicing/modules/slice_combiner.py:47: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
      "  for slice_pred_name in slice_pred_op_names\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5min 22s, sys: 1.43 s, total: 5min 24s\n",
      "Wall time: 2min 50s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'WiC/SuperGLUE/valid/accuracy': 0.7554858934169278,\n",
       " 'WiC_slice:SF_verb_ind/SuperGLUE/valid/f1': 0.5687022900763358,\n",
       " 'WiC_slice:SF_verb_pred/SuperGLUE/valid/accuracy': 0.448559670781893,\n",
       " 'WiC_slice:base_ind/SuperGLUE/valid/f1': 1.0,\n",
       " 'WiC_slice:base_pred/SuperGLUE/valid/accuracy': 0.7570532915360502}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time \n",
    "slice_model.score(dataloaders[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With some simple error analysis and an interface to specifying which _slice_ of the data we care about, we've improved our model **0.94 accuracy points** over a previous state-of-the-art model!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}