examples/3.0_hardest_examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "22de5205",
   "metadata": {},
   "source": [
    "# Finding Errors in the Conll 2003 NER dataset\n",
    "\n",
    "In this notebook, we'll walk through identifying potential errors in the Conll 2003 NER dataset.\n",
    "We'll load up the Conll data into Recon using the huggingface [datasets](https://huggingface.co/docs/datasets/index) library and walk through a standard debugging process with Recon.\n",
    "\n",
    "We'll discuss:\n",
    "\n",
    "1. Stats\n",
    "2. Entity Coverage\n",
    "3. Label Disparities\n",
    "4. Prediction Errors\n",
    "5. Hardest Examples\n",
    "\n",
    "First, lets load up the Corpus using the `load_dataset` function from `datasets` and the `Dataset.from_hf_dataset` loader in Recon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bf8338e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from recon import Corpus, Dataset\n",
    "from recon.stats import get_ner_stats, get_entity_coverage\n",
    "from recon.insights import get_label_disparities, top_label_disparities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4542648f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from datasets.load import load_dataset\n",
    "from datasets.arrow_dataset import Dataset as HFDataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7af6c21c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading builder script: 100%|████████████████████████████████████████████████████| 9.57k/9.57k [00:00<00:00, 11.5MB/s]\n",
      "Downloading metadata: 100%|██████████████████████████████████████████████████████████| 3.73k/3.73k [00:00<00:00, 10.3MB/s]\n",
      "Downloading readme: 100%|████████████████████████████████████████████████████████████| 12.3k/12.3k [00:00<00:00, 18.3MB/s]\n",
      "Generating train split: 100%|█████████████████████████████████████████████| 14041/14041 [00:01<00:00, 12022.47 examples/s]\n",
      "Generating validation split: 100%|██████████████████████████████████████████| 3250/3250 [00:00<00:00, 11959.00 examples/s]\n",
      "Generating test split: 100%|████████████████████████████████████████████████| 3453/3453 [00:00<00:00, 13750.50 examples/s]\n"
     ]
    }
   ],
   "source": [
    "conll2003 = load_dataset(\"conll2003\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ab3cffc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "conll_labels = [\"O\", \"B-PER\", \"I-PER\", \"B-ORG\", \"I-ORG\", \"B-LOC\", \"I-LOC\", \"B-MISC\", \"I-MISC\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a23bc562",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ds = Dataset(\"train\").from_hf_dataset(conll2003[\"train\"], labels=conll_labels)\n",
    "dev_ds = Dataset(\"dev\").from_hf_dataset(conll2003[\"validation\"], labels=conll_labels)\n",
    "test_ds = Dataset(\"test\").from_hf_dataset(conll2003[\"test\"], labels=conll_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b1311c23",
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = Corpus(\"conll2003\", train_ds, dev_ds, test_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e687d54",
   "metadata": {},
   "source": [
    "## 1. Stats\n",
    "\n",
    "First we'll look at the basic stats of the each split in the Corpus using `Corpus.print_summary`. \n",
    "Under the hood, `summary` runs the `recon.stats.get_ner_stats` function across each Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "cef80ea6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset\n",
      "Name: train\n",
      "Stats: {\n",
      "    \"n_examples\": 14041,\n",
      "    \"n_examples_no_entities\": 2909,\n",
      "    \"n_annotations\": 23499,\n",
      "    \"n_annotations_per_type\": {\n",
      "        \"LOC\": 7140,\n",
      "        \"PER\": 6600,\n",
      "        \"ORG\": 6321,\n",
      "        \"MISC\": 3438\n",
      "    }\n",
      "}\n",
      "Dataset\n",
      "Name: dev\n",
      "Stats: {\n",
      "    \"n_examples\": 3250,\n",
      "    \"n_examples_no_entities\": 645,\n",
      "    \"n_annotations\": 5942,\n",
      "    \"n_annotations_per_type\": {\n",
      "        \"PER\": 1842,\n",
      "        \"LOC\": 1837,\n",
      "        \"ORG\": 1341,\n",
      "        \"MISC\": 922\n",
      "    }\n",
      "}\n",
      "Dataset\n",
      "Name: test\n",
      "Stats: {\n",
      "    \"n_examples\": 3453,\n",
      "    \"n_examples_no_entities\": 697,\n",
      "    \"n_annotations\": 5648,\n",
      "    \"n_annotations_per_type\": {\n",
      "        \"LOC\": 1668,\n",
      "        \"ORG\": 1661,\n",
      "        \"PER\": 1617,\n",
      "        \"MISC\": 702\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "corpus.print_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b0ad3fb",
   "metadata": {},
   "source": [
    "Just based on the stats above, we'd probably expect our model to do better on the `LOC` label than the `MISC` label since we have almost twice the labels for `LOC`.\n",
    "\n",
    "Let's dive into dataset statistics a bit more granularly and look at Entity Coverage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4efc3ead",
   "metadata": {},
   "source": [
    "## 2. Entity Coverage\n",
    "\n",
    "Entity Coverage looks at individual annotation spans and aggregates and sorts them. Looking at the extremes (start and end) for the Entity Coverage can tell you a lot about your Dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2883d203",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ecs = get_entity_coverage(corpus.train, case_sensitive=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b17c9f4",
   "metadata": {},
   "source": [
    "**Most** covered entity spans\n",
    "\n",
    "It's no surprise that common locations are our most covered entity spans. If we train a model, we'd expect these entities to be easy to identify."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0aa28e53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[EntityCoverage(text='U.S.', label='LOC', count=303),\n",
       " EntityCoverage(text='Germany', label='LOC', count=141),\n",
       " EntityCoverage(text='Britain', label='LOC', count=133),\n",
       " EntityCoverage(text='Australia', label='LOC', count=130),\n",
       " EntityCoverage(text='England', label='LOC', count=123),\n",
       " EntityCoverage(text='France', label='LOC', count=122),\n",
       " EntityCoverage(text='Spain', label='LOC', count=110),\n",
       " EntityCoverage(text='Italy', label='LOC', count=98),\n",
       " EntityCoverage(text='LONDON', label='LOC', count=93),\n",
       " EntityCoverage(text='Russian', label='MISC', count=92)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_ecs[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ba6ac9c",
   "metadata": {},
   "source": [
    "**Least** covered entity spans\n",
    "It's also no surprise that Person names are in our least covered entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b92c395e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[EntityCoverage(text='Luca Cadalora', label='PER', count=1),\n",
       " EntityCoverage(text='Alex Criville', label='PER', count=1),\n",
       " EntityCoverage(text='Scott Russell', label='PER', count=1),\n",
       " EntityCoverage(text='Tadayuki Okada', label='PER', count=1),\n",
       " EntityCoverage(text='Carlos Checa', label='PER', count=1),\n",
       " EntityCoverage(text='Alexandre Barros', label='PER', count=1),\n",
       " EntityCoverage(text='Shinichi Itoh', label='PER', count=1),\n",
       " EntityCoverage(text='Swe', label='LOC', count=1),\n",
       " EntityCoverage(text='Bob May', label='PER', count=1),\n",
       " EntityCoverage(text='Bradley Hughes', label='PER', count=1)]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_ecs[-10:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14156b82-c5bb-4e39-a0cd-6d1cc3254fb6",
   "metadata": {},
   "source": [
    "## Label Disparities (sometimes 1 label, sometimes another)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "e89ff9b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "lds = top_label_disparities(corpus.train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "6756f91e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[LabelDisparity(label1='ORG', label2='LOC', count=93, examples=[]),\n",
       " LabelDisparity(label1='LOC', label2='ORG', count=93, examples=[]),\n",
       " LabelDisparity(label1='ORG', label2='PER', count=16, examples=[]),\n",
       " LabelDisparity(label1='PER', label2='ORG', count=16, examples=[]),\n",
       " LabelDisparity(label1='PER', label2='LOC', count=12, examples=[]),\n",
       " LabelDisparity(label1='LOC', label2='PER', count=12, examples=[]),\n",
       " LabelDisparity(label1='MISC', label2='LOC', count=7, examples=[]),\n",
       " LabelDisparity(label1='LOC', label2='MISC', count=7, examples=[]),\n",
       " LabelDisparity(label1='ORG', label2='MISC', count=6, examples=[]),\n",
       " LabelDisparity(label1='MISC', label2='ORG', count=6, examples=[]),\n",
       " LabelDisparity(label1='MISC', label2='PER', count=1, examples=[]),\n",
       " LabelDisparity(label1='PER', label2='MISC', count=1, examples=[])]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "21d47a7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List, Tuple\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "from recon.types import LabelDisparity\n",
    "\n",
    "def plot_cm(cm: np.array, labels: List[str]):\n",
    "    labels = np.asarray(list(labels))\n",
    "    print(cm, labels)\n",
    "    return sns.heatmap(cm, annot=True, xticklabels=labels, yticklabels=labels, cmap=\"Blues\")\n",
    "\n",
    "def get_lds_cm(label_disparities: List[LabelDisparity]) -> Tuple[List[str], np.array]:\n",
    "    labels = set()\n",
    "    for ld in lds:\n",
    "        labels.add(ld.label1)\n",
    "        labels.add(ld.label2)\n",
    "    labels_map = {l:i for i, l in enumerate(labels)}\n",
    "    K = len(labels)\n",
    "    result = np.zeros((K, K))\n",
    "    for ld in lds:\n",
    "        result[labels_map[ld.label1]][labels_map[ld.label2]] = ld.count\n",
    "    return labels, result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "973e5668",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.  7. 12. 93.]\n",
      " [ 7.  0.  1.  6.]\n",
      " [12.  1.  0. 16.]\n",
      " [93.  6. 16.  0.]] ['LOC' 'MISC' 'PER' 'ORG']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Axes: >"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf8AAAGdCAYAAAAczXrvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/SrBM8AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA+LElEQVR4nO3dfVyN9/8H8Nc56U7llEqJkLkJiQwVZhthZiOau9Xc5Oc2hrahhjBkZpi532juzTZ88R2+Y3NfyN1MVESFSiU11OnmnN8f2852TplOTl2n63o9Pa7HY+dz3b2vzup93p/rc32OTK1Wq0FERESSIRc6ACIiIqpaTP5EREQSw+RPREQkMUz+REREEsPkT0REJDFM/kRERBLD5E9ERCQxTP5EREQSw+RPREQkMTWEDuAvBcVCR0B/yX1aJHQI9KcnyhKhQ6A/ter5odAh0D/kX1pZqce39JposGNVdqwVYTTJn4iIyGjIxN0xLu6rIyIiolJY+RMREemSyYSOoFIx+RMREekSebc/kz8REZEukVf+4v5oQ0RERKWw8iciItLFbn8iIiKJYbc/ERERiQkrfyIiIl3s9iciIpIYdvsTERGRmLDyJyIi0sVufyIiIolhtz8RERGJCSt/IiIiXez2JyIikhiRd/sz+RMREekSeeUv7qsjIiKiUlj5ExER6RJ55c/kT0REpEsu7nv+4v5oQ0RERKWw8iciItLFbn8iIiKJEfmjfuL+aENERESlsPInIiLSxW5/IiIiiWG3PxEREYkJK38iIiJd7PYnIiKSGJF3+zP5ExER6RJ55S/uqyMiIqJSWPkTERHpYrc/ERGRxLDbX1teXh5UKlWp9pKSEuTl5RkkKCIiIqo8eiX/PXv2oH379igoKCi1rqCgAB06dMD+/fsNFhwREZEgZDLDLUZIr+S/Zs0aTJs2DTVr1iy1zsrKCtOnT8fKlSsNFhwREZEgZHLDLUZIr6h+++03vPbaa89c37VrV1y9evVFYyIiIqJKpNeAv5ycHBQXFz9zfVFREXJycl44KCIiIkEZacVuKHol/0aNGiE2Nhbu7u5lro+NjUXDhg0NElh1s3P7NmyK2oCsrEw0a+6OGeGz0NrTU+iwJGVQ355IT7tfqt3/nSEInT5TgIik4+rlC/h++ze4GX8dD7MzMWvhMnTq2g0AUFxchE3rVyI25hTS7t+FlZUNvNp7Y+T4ybB3qCNw5OJkXdMcERPeQt9ubeBoZ40r8Xfx4eLvcSEuBQDw8dg3MbBXO9R3tkNhUQkuXU/BnJX7cf63ZIEjNyJGeq/eUPT6aDNgwAB8/PHHyMjIKLUuPT0dM2fOREBAgMGCqy4OHfwRSxZHYuyEEOz8bg+aN3fH+LGjkJ2dLXRokrJ+007sOXhMsyxd+RUA4HW/ngJHJn4F+flo3KQ5JoSGlVqnLCjArYQbGDp8DFZu/BYzFyzF3ZQ7mDt9sgCRSsOa2e+im487gmduQvtBC3Ek+gb+u3YSXBwVAICbyQ8w9dPv0H7gQnQfuRTJ9x9i/+qJcLCzFjhyqioytVqtLu/Gv//+O3x9fZGSkoKgoCA0b94cAHDjxg1s27YNrq6uiImJgY2Njd6BFDz7boLRCxwyEK08WiN85mwAgEqlQs/ur2Lou+9h1OgxAkenv9ynRUKHYBArPl+E6FPHsX33j5BV00/xT5QlQoegt95d2mhV/mWJv/4bpowOxKbvD6GOc90qjK7iWvX8UOgQysXC3BSZp5Zg4NT1OHTqmqb99LZp+N/pOMxdfaDUPjZWFnhwagl6j12BY+cSqjLcCsu/VLmDyy37rTPYsfL/M9ZgxzIUvbr9bWxscPr0aYSFheHbb7/V3N+3tbVFUFAQFixYUKHEX50VFRbietw1jBr995srl8vh49MJv165JGBk0lZUVISfDh7AoMBh1Tbxi9nTx48hk8lgJbG/F1WhhokcNWqYoKBQ+0N8gbIInbxeKrW9aQ0TjBrQGY9+f4qrCfeqKkzjJ/K/G3rP8KdQKLB69WqsWrUKWVlZUKvVcHR0lOwf2JxHOSgpKYG9vb1Wu729PW7fThIoKjp57CgeP/4dvd/yFzoU0lGoVGLjmuV41a83rKzYzWxoj58qEXMlCWGjeyP+dgYysvMw6I328PZ0w63UTM12vV/xwOZFI1HTwhTpWXl4a9xKZD96ImDkRoYD/sp29epVJCT80T3UvHlztG7dutz7KpVKKJVKrTa1iTnMzc0rGg6Rlv/u2w1v3y5wcOSAMmNSXFyEhbM/ghpqTPzwY6HDEa3gmZuxbk4gkv63AMXFJbh8IxW7DsXCq0UDzTbHzyfAe0gkHGytMXJAJ2xdHIyu7y1BZs5jASOnqqL3R5tz586hdevW8PLywqBBgzBo0CC0bdsWnp6eOH/+fLmOERkZCYVCobV89mmk3sEbAztbO5iYmJQa3JednQ0HBweBopK29LT7uHAuBn38pTf41JgVFxdh4ayP8CA9DQuXrWPVX4lu381Cz//7Ava+oWjaexZeeW8JTGuY4Pa9LM02TwsKkZSahXNX72D83O0oLlFheP9OAkZtZDjD39/i4uLQvXt3WFpaYuvWrbh48SIuXryILVu2wNzcHN27d0dcXNxzjxMWFobc3Fyt5aPppUcJVwemZmZo0bIVzsZEa9pUKhXOno2GZxsvASOTrh/374GtXW34du4qdCj0p78S//27KVi4fB1qKWyFDkkSnhYUIj0rD7Y2lvDr1AIHjj17Eja5TAZzU37X219kMpnBFn2UlJRg1qxZcHNzg6WlJV566SV88skn+OfYfLVajdmzZ6Nu3bqwtLSEn58fEhMT9TqPXu/0nDlz0KNHD/zwww9aF9S2bVsMHToUAwYMwJw5c7Br165/PY65eeku/uo82v+94SMxK3w6WrXygEdrT2zdsgn5+fnw7z9A6NAkR6VS4eD+vXijTz/UqME/ZFUl/+lT3L+XonmdkXYPtxJvwMZGgdoODlgw80PcTLiOuZ9+CZVKhYfZf1SgNrUUMDU1FSps0fLzbQGZDEi48wAvuTpi4VR/JNzOwOZ90ahpYYbp/9cL/z1+FelZubC3tcbYQV3hUscWu3+6KHTokvfpp59izZo12LRpE1q1aoXY2FiMHDkSCoUC77//PgBg8eLFWLFiBTZt2gQ3NzfMmjULvXr1QlxcHCwsLMp1Hr3+Ov7yyy84ePBgmZ9kZDIZwsPD8eabb+pzSFF4o/ebyHn4EKtXrkBWViaau7fA6nVfw57d/lUu9lw0MtLT0Kdvf6FDkZTEG9cw/f3/07xe/+USAIBf774ICh6HmFPHAAAhIwdp7ffpiq/h2a5DlcUpFQprC8yb1Bf1nGzxMPcp/nP0MiJW7UdxsQomchWaN3JC0NvesLe1wsPcp4i9lgy/4GW4npQudOhGQ6hB7GfOnEG/fv3Qp08fAH9Mrrdjxw6cO3cOwB9V//LlyzFz5kz069cPALB582Y4OTlh7969GDJkSLnOo9dz/hYWFkhMTISrq2uZ61NTU9G0adMyv/Xveapz5S82YnnOXwyq43P+YlVdnvOXisp+zt9qYJTBjvVw67ulBrmX1QMOAAsXLsT69evxv//9D82aNcOVK1fQs2dPLF26FIGBgUhKSsJLL72ES5cuoW3btpr9Xn31VbRt2xZffPFFuWLS655/w4YNNZ8+ynL27FnJTu9LRERUlrIGuUdGlj3IfcaMGRgyZAjc3d1hamoKLy8vTJkyBYGBgQD+mE0XAJycnLT2c3Jy0qwrD726/YcMGYLQ0FA0b94cHh4eWuuuXr2KDz/8EMOGDdPnkEREREbHkN3+YWFhCA0N1Wp71qPtu3btwrZt27B9+3a0atUKly9fxpQpU+Di4oLhw4cbLCa9kn9YWBiOHDmCtm3bokePHmjRogXUajWuX7+OI0eOoGPHjggPDzdYcEREREIwZPJ/Vhd/WT766CNN9Q8ArVu3RnJyMiIjIzF8+HA4OzsDADIyMlC37t9TY2dkZGjdBngevbr9LSws8Msvv2DBggVIS0vD2rVrsW7dOqSnp2P+/PnYtWuXZjQiERER6efp06eQy7VTs4mJCVQqFQDAzc0Nzs7OOHr0qGZ9Xl4ezp49C19f33KfR+9noczMzDB9+nRMnz691LorV65gw4YNWL9+vb6HJSIiMhpCjfZ/++23sWDBAjRo0ACtWrXCpUuXsHTpUgQHB2vimjJlCubPn4+mTZtqHvVzcXGBv79/uc/DB6GJiIh0CJX8v/zyS8yaNQsTJkzAgwcP4OLigrFjx2L27NmabaZNm4YnT55gzJgxePToEbp06YJDhw6V+xl/QM9H/Z7nypUraNeuHUpK9H88iY/6GQ8+6mc8+Kif8eCjfsalsh/1U7y7xWDHyt3+nsGOZSji/toiIiIiKkWvbv8BA/59utpHjx69SCxERERGQexfU69X8lcoFM9dz+f8iYioumPy/4eoKMNNd0hERETC4Gh/IiIiHaz8iYiIJEbsyZ+j/YmIiCSGlT8REZEucRf+TP5ERES62O1PREREosLKn4iISIfYK38mfyIiIh1M/kRERFIj7tzPe/5ERERSw8qfiIhIB7v9iYiIJEbsyZ/d/kRERBLDyp+IiEiH2Ct/Jn8iIiIdYk/+7PYnIiKSGFb+REREusRd+DP5ExER6WK3PxEREYkKK38iIiIdYq/8mfyJiIh0MPkTERFJjbhzP+/5ExERSQ0rfyIiIh3s9iciIpIYsSd/dvsTERFJDCt/IiIiHWKv/Jn8iYiIdIg9+bPbn4iISGJY+RMREekSd+FvPMk/92mR0CHQnxQ1TYUOgf5kY2k0v6KSl3JyudAhUBVitz8RERGJCssKIiIiHWKv/Jn8iYiIdIg89zP5ExER6RJ75c97/kRERBLDyp+IiEiHyAt/Jn8iIiJd7PYnIiIiUWHlT0REpEPkhT+TPxERkS65XNzZn93+REREEsPKn4iISAe7/YmIiCSGo/2JiIhIVFj5ExER6RB54c/kT0REpEvs3f5M/kRERDrEnvx5z5+IiEhiWPkTERHpEHnhz+RPRESki93+REREJCqs/ImIiHSIvPBn8iciItLFbn8iIiISFVb+REREOkRe+DP5ExER6WK3PxEREYkKK38iIiIdIi/8mfyJiIh0ib3bn8mfiIhIh8hzP+/5ExERSQ0rfyIiIh1i7/bXq/JPTEzE0KFDkZeXV2pdbm4u3n33XSQlJRksOCIiIiHIZIZbjJFeyf+zzz6Dq6sratWqVWqdQqGAq6srPvvsM4MFR0RERIanV/I/fvw4Bg4c+Mz1gwYNws8///zCQREREQlJJpMZbNHXvXv3EBQUBHt7e1haWqJ169aIjY3VrFer1Zg9ezbq1q0LS0tL+Pn5ITExUa9z6JX8U1JSUKdOnWeud3BwQGpqql4BEBERGRuhuv1zcnLQuXNnmJqa4uDBg4iLi8Pnn38OOzs7zTaLFy/GihUrsHbtWpw9exZWVlbo1asXCgoKyn0evQb8KRQK3Lp1Cw0bNixz/c2bN8u8JUBERETP9+mnn8LV1RVRUVGaNjc3N81/q9VqLF++HDNnzkS/fv0AAJs3b4aTkxP27t2LIUOGlOs8elX+Xbt2xZdffvnM9StWrMArr7yizyGJiIiMjiG7/ZVKJfLy8rQWpVJZ5nn37duH9u3bY+DAgahTpw68vLzw1Vdfadbfvn0b6enp8PPz07QpFAp4e3sjOjq63NenV/IPCwvDwYMH8c477+DcuXPIzc1Fbm4uzp49i4CAABw+fBhhYWH6HJKIiMjoGDL5R0ZGQqFQaC2RkZFlnjcpKQlr1qxB06ZNcfjwYYwfPx7vv/8+Nm3aBABIT08HADg5OWnt5+TkpFlXHnp1+3t5eeH7779HcHAw9uzZo7XO3t4eu3btQrt27fQ5JBERkaiFhYUhNDRUq83c3LzMbVUqFdq3b4+FCxcC+CPv/vbbb1i7di2GDx9usJj0nuTnrbfeQnJyMg4dOoSbN29CrVajWbNm6NmzJ2rWrGmwwKqLQX17Ij3tfql2/3eGIHT6TAEiop3bt2FT1AZkZWWiWXN3zAifhdaenkKHJTkXYs9jc9QGxMVdQ1ZmJpZ+sRKvd/d7/o5kcJkPMrBmxVLEnDmJgoIC1K/fAOFz5sO9pYfQoRktQz6fb25u/sxkr6tu3bpo2bKlVluLFi3www8/AACcnZ0BABkZGahbt65mm4yMDLRt27bcMVVohj9LS0v079+/IruKzvpNO1FSotK8vn0rEaETR+N1v54CRiVdhw7+iCWLIzEzYi5at26DbVs2YfzYUfjPgUOwt7cXOjxJyc/PR7Pm7ujXPwAfTJkkdDiSlZeXi/HBQWjXviOWrFgLW7vauJuSDBsbDs7+N0LN8Ne5c2fEx8drtSUkJGgG2ru5ucHZ2RlHjx7VJPu8vDycPXsW48ePL/d59LrnHx0djQMHDmi1bd68GW5ubqhTpw7GjBnzzEEMYmVrVxv2Dg6a5cyp46hX3xVt23UQOjRJ2rIpCgPeGQT//gF4qUkTzIyYCwsLC+zd/YPQoUlOl1e6IuT9Kejm10PoUCRt2zcbUMfJGeFzFqClhydc6tVHR9/OqOfaQOjQjJpQj/pNnToVMTExWLhwIW7evInt27dj/fr1CAkJ+TMuGaZMmYL58+dj3759uHr1KoYNGwYXFxf4+/uX+zx6Jf958+bh2rVrmtdXr17FqFGj4OfnhxkzZmD//v3PHMQgBUVFRfjp4AG82be/6OeFNkZFhYW4HncNPr6dNG1yuRw+Pp3w65VLAkZGJJzTJ36Be8tWmDltKt7yewUj3w3Avt3fCR0WPUOHDh2wZ88e7NixAx4eHvjkk0+wfPlyBAYGaraZNm0aJk2ahDFjxqBDhw54/PgxDh06BAsLi3KfR69u/8uXL+OTTz7RvN65cye8vb01jyG4uroiIiICc+bM+dfjKJXKUj0ESqW83PdEjNXJY0fx+PHv6P2Wv9ChSFLOoxyUlJSU6t63t7fH7dv8zgmSpvv37mLv999icOBwDAseg+txV7F8SSRMTU3R+21/ocMzWkIWcG+99RbeeuutZ66XyWSYN28e5s2bV+Fz6FX55+TkaD1ecPz4cfTu3VvzukOHDuWa4a+sxx5WLP1Un1CM0n/37Ya3bxc4OD57FkQioqqkUqnQzL0lxk6cgmbuLdBvwCD09X8He3/YJXRoRo1f7PMPTk5OuH37NgCgsLAQFy9ehI+Pj2b977//DlNT0+ceJywsTDNHwF/L+6HT9QzduKSn3ceFczHo4x8gdCiSZWdrBxMTE2RnZ2u1Z2dnw8HBQaCoiIRl7+CIRm4vabU1dGuMjPQ0gSIiY6BX8n/zzTcxY8YMnDx5EmFhYahZs6bWjH6//vorXnrppX85wh/Mzc1Rq1YtraW6d/n/uH8PbO1qw7dzV6FDkSxTMzO0aNkKZ2P+nuVKpVLh7NloeLbxEjAyIuG0buOFlOTbWm2pKXfgXNdFoIiqB7lMZrDFGOmV/D/55BPUqFEDr776Kr766it89dVXMDMz06zfuHEjevaU3iNuKpUKB/fvxRt9+qFGjQo9PUkG8t7wkdj9/S7s27sHSbduYf68OcjPz4d//wFChyY5T58+QfyN64i/cR0AcO/eXcTfuI60MubFoMozOHAYrl39FZs3rsfd1GT87+AB7Nv9PQYMHCp0aEZN7N3+MrVardZ3p9zcXFhbW8PExESr/eHDh7C2ttb6QFBeGXlFeu9jLM7FnMaHk8Zi2/cH4NqwkdDhvDBFzeffujFmO7Zt1Uzy09y9BaaHz4SnZxuhw6oQlf6/nkYj9txZjA4uPSPZ2/38MW/BIgEiejFPlCVCh1Bhp08cw7qVy3E3NRl1XepjcOAw9B3w7K9nrw4crSu30Oq5KsZgx/pfiM/zN6piFUr+laE6J3+xqe7JX0yqc/IXm+qc/MWospN/r9VnDXaswxO8DXYsQ9HrpzdgQPm6Tnfv3l2hYIiIiIyB3Ei76w1Fr+SvUCgqKw4iIiKjIfaJ2vRK/lFRUZUVBxEREVURvZJ/cHDwc7eRyWTYsGFDhQMiIiISmsgLf/2S/zfffIOGDRvCy8sLRjJOkIiIyOBkEHf21yv5jx8/Hjt27MDt27cxcuRIBAUFoXbt2pUVGxEREVUCvSb5WbVqFdLS0jBt2jTs378frq6uGDRoEA4fPsyeACIiEg25zHCLMdIr+QN/TM07dOhQ/PTTT4iLi0OrVq0wYcIENGrUCI8fP66MGImIiKqUTCYz2GKM9E7+WjvL5ZDJZFCr1Sgp4QQYRERE1YHeyV+pVGLHjh3o0aMHmjVrhqtXr2LlypVISUmBtbV1ZcRIRERUpcQ+t79eA/4mTJiAnTt3wtXVFcHBwdixYwe/KpWIiETHWL+Nz1D0mttfLpejQYMG8PLy+tf7GBWZ3pdz+xsPzu1vPDi3v/Hg3P7GpbLn9h+w4YLBjrV71MsGO5ah6PXTGzZsmNEOXiAiIjIUsac6vSf5ISIiEjuxF7qV229CRERUDYk897/Yo35ERERU/bDyJyIi0iH20f5M/kRERDrEnfrZ7U9ERCQ5rPyJiIh0cLQ/ERGRxBjrt/EZCrv9iYiIJIaVPxERkQ52+xMREUmMyHM/u/2JiIikhpU/ERGRDnb7ExERSYzYR/sz+RMREekQe+XPe/5EREQSw8qfiIhIh7jrfiZ/IiKiUsT+rX7s9iciIpIYVv5EREQ6RF74M/kTERHp4mh/IiIiEhVW/kRERDpEXvgz+RMREeniaH8iIiISFVb+REREOkRe+DP5ExER6RL7aH+jSf5PlCVCh0B/srE0mv8tJE/s9x2rk8w8pdAh0D84Wlfu3ymx3xMX+/URERGRDpZ4REREOtjtT0REJDFyced+dvsTERFJDSt/IiIiHWKv/Jn8iYiIdIj9nj+7/YmIiCSGlT8REZEOdvsTERFJjMh7/dntT0REJDWs/ImIiHSIfWptJn8iIiIdYu8WZ/InIiLSIfLCX/QfboiIiEgHK38iIiIdvOdPREQkMSLP/ez2JyIikhpW/kRERDo4wx8REZHEiP2eP7v9iYiIJIaVPxERkQ6RF/6s/ImIiHTJZYZbKmrRokWQyWSYMmWKpq2goAAhISGwt7eHtbU1AgICkJGRof/1VTwsIiIiqgznz5/HunXr4OnpqdU+depU7N+/H9999x2OHz+O+/fvY8CAAXofn8mfiIhIh8yA//T1+PFjBAYG4quvvoKdnZ2mPTc3Fxs2bMDSpUvRrVs3vPzyy4iKisKZM2cQExOj1zmY/ImIiHQYsttfqVQiLy9Pa1Eqlc88d0hICPr06QM/Pz+t9gsXLqCoqEir3d3dHQ0aNEB0dLR+16ffj4OIiEj8DJn8IyMjoVAotJbIyMgyz7tz505cvHixzPXp6ekwMzODra2tVruTkxPS09P1uj6O9iciIqpEYWFhCA0N1WozNzcvtV1qaiomT56Mn376CRYWFpUaE5M/ERGRDpkBn/UzNzcvM9nrunDhAh48eIB27dpp2kpKSnDixAmsXLkShw8fRmFhIR49eqRV/WdkZMDZ2VmvmJj8iYiIdAgxvW/37t1x9epVrbaRI0fC3d0d06dPh6urK0xNTXH06FEEBAQAAOLj45GSkgJfX1+9zsXkT0REZARsbGzg4eGh1WZlZQV7e3tN+6hRoxAaGoratWujVq1amDRpEnx9feHj46PXuZj8iYiIdBjrDH/Lli2DXC5HQEAAlEolevXqhdWrV+t9HJlarVZXQnx6S8osEDoE+pOz7fPvTVHVEPuXi1QnSQ+eCB0C/UNLF6tKPf7yk7cNdqwpr7gZ7FiGwkf9iIiIJIbd/kRERDqEGPBXlZj8iYiIdIj9jpvBu/3v3btn6EMSERGRARks+aenp2PSpElo2rSpoQ5JREQkCDlkBluMkV7JPycnB0OHDoWDgwNcXFywYsUKqFQqzJ49G40bN8b58+cRFRVVWbESERFVCZnMcIsx0uue/4wZM3DmzBmMGDEChw8fxtSpU3Ho0CHI5XL8/PPPek8yQEREZIzEPuBPr8r/4MGDiIqKwpIlS7B//36o1Wq0bdsWBw4cYOInIiKqJvSq/O/fv48WLVoAABo1agQLCwsEBQVVSmDG6urlC/h++ze4GX8dD7MzMWvhMnTq2g0AUFxchE3rVyI25hTS7t+FlZUNvNp7Y+T4ybB3qCNw5NJwIfY8NkdtQFzcNWRlZmLpFyvxene/5+9IlWLn9m3YFLUBWVmZaNbcHTPCZ6G1p6fQYYnetSsXsPfbzbiVcB052VmY8cnn8O7yutY2qclJ2LJ+Ba5duYiSkmK4NmyMaXM/g6NTXYGiNi5in2BLr8pfrVajRo2/Py+YmJjA0tLS4EEZs4L8fDRu0hwTQsNKrVMWFOBWwg0MHT4GKzd+i5kLluJuyh3MnT5ZgEilKT8/H82auyPs49lChyJ5hw7+iCWLIzF2Qgh2frcHzZu7Y/zYUcjOzhY6NNErKChAo5eaYczkGWWuT7uXivD3R6GeayN8smw9ln39LQa+NxqmZpzd8y+85/8ParUa3bt313wAyM/Px9tvvw0zMzOt7S5evGi4CI1MB98u6ODbpcx1VtY2WLh8nVbb+NAwTBkdiAfpaajjzE/Ula3LK13R5ZWuQodBALZsisKAdwbBv/8f3z42M2IuTpw4hr27f8Co0WMEjk7cXvbujJe9Oz9z/fYNq/Cyd2cMHzdF01a3nmsVREbGQq/kHxERofW6X79+Bg1GjJ4+fgyZTAYrGxuhQyGqMkWFhbgedw2jRo/VtMnlcvj4dMKvVy4JGBmpVCrExpxC/yHDMfejCUi6GQ8n53oICBxZ6taAlIm92/+Fkj/9u0KlEhvXLMerfr1hZWUtdDhEVSbnUQ5KSkpgb2+v1W5vb4/bt5MEiooAIPfRQxTkP8XuHVF4N3gCho2djIvnzuDT2R9i3tL18Gj7stAhGgWR5379kv+DBw9Qp86zB64VFxfj4sWL6Nix478eR6lUQqlU6rSpYW4unvtNxcVFWDj7I6ihxsQPPxY6HCIiAIBa9ccXuXbs9Br6DvxjwLZbk+aIv3YFh/d/z+QvEXoN+Ktbty4ePHiged26dWukpqZqXmdnZ8PX1/e5x4mMjIRCodBa1n7xmT6hGLXi4iIsnPURHqSnYeGydaz6SXLsbO1gYmJSanBfdnY2HBwcBIqKAMBGYQsTkxpwbdRYq71+AzdkZaQLFJXxkRtwMUZ6j/b/pzt37qCoqOhftylLWFgYcnNztZZxkz/SJxSj9Vfiv383BQuXr0Mtha3QIRFVOVMzM7Ro2QpnY6I1bSqVCmfPRsOzjZeAkZGpqSmauLfEvdQ7Wu3376bwMb9/kMlkBluMkcG/1a88F2publ6qiz9LWWDoUCpF/tOnuH8vRfM6I+0ebiXegI2NArUdHLBg5oe4mXAdcz/9EiqVCg+zswAANrUUMDU1FSpsyXj69AlSU/5+f+7du4v4G9dRS6FA3bouAkYmPe8NH4lZ4dPRqpUHPFp7YuuWTcjPz4d//wFChyZ6+flPkX7v717ZjLR7uH0zHtY2teDoVBf+g4fh83kz0NKzHVp7tcelc2dw/swJfLJ8vYBRU1WSqctTqv9JLpcjPT1dc9/fxsYGV65cQePGf3QfZWRkwMXFBSUlJXoHkpRZPZL/rxfPY/r7/1eq3a93XwQFj8OIgW+Wud+nK76GZ7sOlR2eQTjbVt+xF7HnzmJ08PBS7W/388e8BYsEiOjFVPcRxzu2bdVM8tPcvQWmh8+Ep2cbocOqkKQHT4QOodx+uxyLWVNLP075eq+38f6MuQCAIz/uxe7tUcjOfAAX14YYMmIcvLu8VsWRVlxLF6tKPf7m2NTnb1ROw9ob32OUeiV/ExMTJCQkwNHREWq1Gq6urjh16hQaNWoE4I/k7+7uLurkLwXVOfmLTXVP/mJSnZK/FFR28t964a7BjhX0cn2DHctQ9J7kp1mzZlqvvby8tF4b6/0NIiKi8hJ7JtMr+f/yyy+VFQcRERFVEb2Sf5cuXbBkyRLs27cPhYWF6N69OyIiIiQ3vz8REYmb2Dux9XrUb+HChQgPD4e1tTXq1auHL774AiEhIZUVGxERkSDE/qifXsl/8+bNWL16NQ4fPoy9e/di//792LZtG1QqVWXFR0RERAamV/JPSUnBm2/+/Sibn58fZDIZ7t+/b/DAiIiIhCL2Gf70uudfXFwMCwsLrTZTU9NSs/wRERFVZ8baXW8oej/qN2LECK3Z+QoKCjBu3DhYWf39zOXu3bsNFyEREREZlF7Jf/jw0jOnBQUFGSwYIiIiYyDuul/P5B8VFVVZcRARERkNsXf7G+tYBCIiIqokBv9WPyIioupO7JUxkz8REZEOsXf7M/kTERHpEHfqF3/PBhEREelg5U9ERKRD5L3+TP5ERES65CLv+Ge3PxERkcSw8iciItLBbn8iIiKJkbHbn4iIiMSElT8REZEOdvsTERFJDEf7ExERkaiw8iciItLBbn8iIiKJYfInIiKSGD7qR0RERKLCyp+IiEiHXNyFP5M/ERGRLnb7ExERkaiw8iciItLB0f5EREQSw25/IiIiEhVW/kRERDo42p+IiEhi2O1PREREosLKn4iISAdH+xMREUmMyHM/kz8REZEuuchLf97zJyIikhijqfxb9fxQ6BDoTyknlwsdAv0pM08pdAj0p8Z1rIQOgaqQuOt+I0r+RERERkPk2Z/d/kRERBLDyp+IiEiH2Cf5YfInIiLSIfLB/uz2JyIikhpW/kRERDpEXvgz+RMREZUi8uzPbn8iIiKJYfInIiLSITPgP31ERkaiQ4cOsLGxQZ06deDv74/4+HitbQoKChASEgJ7e3tYW1sjICAAGRkZep2HyZ+IiEiHTGa4RR/Hjx9HSEgIYmJi8NNPP6GoqAg9e/bEkydPNNtMnToV+/fvx3fffYfjx4/j/v37GDBggH7Xp1ar1fqFVjksvSYKHQL9idP7Gg9O72s8OL2vcbGo5BFrF+/kGexY7RrVqvC+mZmZqFOnDo4fP46uXbsiNzcXjo6O2L59O9555x0AwI0bN9CiRQtER0fDx8enXMdl5U9ERFSJlEol8vLytBalsnwf7HNzcwEAtWvXBgBcuHABRUVF8PPz02zj7u6OBg0aIDo6utwxMfkTERHpkhluiYyMhEKh0FoiIyOfG4JKpcKUKVPQuXNneHh4AADS09NhZmYGW1tbrW2dnJyQnp5e7svjo35EREQ6DDm9b1hYGEJDQ7XazM3Nn7tfSEgIfvvtN5w6dcpgsfyFyZ+IiKgSmZublyvZ/9PEiRNx4MABnDhxAvXr19e0Ozs7o7CwEI8ePdKq/jMyMuDs7Fzu47Pbn4iISIdQo/3VajUmTpyIPXv24Oeff4abm5vW+pdffhmmpqY4evSopi0+Ph4pKSnw9fUt93lY+RMREekQaoK/kJAQbN++Hf/5z39gY2OjuY+vUChgaWkJhUKBUaNGITQ0FLVr10atWrUwadIk+Pr6lnukP8DkT0REZDTWrFkDAHjttde02qOiojBixAgAwLJlyyCXyxEQEAClUolevXph9erVep2HyZ+IiEiXQKV/eabesbCwwKpVq7Bq1aoKn4fJn4iISIchR/sbIw74IyIikhhW/kRERDr0HaVf3TD5ExER6RB57mfyJyIiKkXk2Z/3/ImIiCSGlT8REZEOsY/2Z/InIiLSIfYBf+z2JyIikhhW/kRERDpEXvgz+RMREZUi8uzPbn8iIiKJYeVPRESkg6P9/8Xvv/+u9Q1Ecrkc1tbWLxwUERGRkDja/x8uX76MN998U/PaxcUFdnZ2msXW1hbnz583eJBERERkOHpV/l9++SW6dOmi1bZlyxbUq1cParUaGzduxIoVK7BlyxaDBklERFSVRF7465f8z5w5g4kTJ2q1+fj4oHHjxgAAS0tLDBo0yHDRERERCUHk2V+v5J+cnAxHR0fN63nz5sHBwUHzum7dusjIyDBcdERERAIQ+4A/ve75W1hYIDk5WfN66tSpqFWrluZ1amoqatasabjoiIiIyOD0Sv5eXl7Yu3fvM9fv3r0bXl5eLxoTERGRoGQywy3GSK9u/wkTJmDIkCFo1KgRxo8fD7n8j88OJSUlWL16Nb788kts3769UgI1FtY1zREx4S307dYGjnbWuBJ/Fx8u/h4X4lIAAB+PfRMDe7VDfWc7FBaV4NL1FMxZuR/nf0t+zpHJEDIfZGDNiqWIOXMSBQUFqF+/AcLnzId7Sw+hQxO1a1cuYO+3m3Er4TpysrMw45PP4d3lda1tUpOTsGX9Cly7chElJcVwbdgY0+Z+BkenugJFLS07t2/DpqgNyMrKRLPm7pgRPgutPT2FDstoGWnONhi9kn9AQABCQ0MxadIkhIeHawb6JSUl4fHjxwgNDcU777xTKYEaizWz30XLJi4InrkJaZm5GPpmR/x37SS0C5iP+5m5uJn8AFM//Q6372bB0twUk4K6Yf/qifDoNxdZOY+FDl/U8vJyMT44CO3ad8SSFWtha1cbd1OSYWNT6/k70wspKChAo5eaoXvvfvh09oel1qfdS0X4+6Pg17sfhowYB8uaVki9kwRTM3MBopWeQwd/xJLFkZgZMRetW7fBti2bMH7sKPznwCHY29sLHR4JQKb+5yw95RQTE4MdO3YgMTERANC0aVMMHToUPj4+FQ7E0mvi8zcSmIW5KTJPLcHAqetx6NQ1TfvpbdPwv9NxmLv6QKl9bKws8ODUEvQeuwLHziVUZbgVlnJyudAhVMiaFUtx9colrN4gnkdNM/OUQoegt/6vtytV+X8+bwZMatTAlPD5Akb2YhrXsRI6hAoLHDIQrTxaI3zmbACASqVCz+6vYui772HU6DECR1cxFpU8P+2tzHyDHeslR0uDHctQKvTj8/HxeaFEX13VMJGjRg0TFBQWabUXKIvQyeulUtub1jDBqAGd8ej3p7iacK+qwpSs0yd+QUffzpg5bSouX4yFY5066P/OEPQdMFDo0CRNpVIhNuYU+g8ZjrkfTUDSzXg4OddDQODIUrcGyPCKCgtxPe4aRo0eq2mTy+Xw8emEX69cEjAy48bR/nrYvXs3PEV8D+nxUyViriQhbHRv1HVUQC6XYcibHeDt6QZnh7+7lnu/4oHM05/j0dllmBT0Ot4atxLZj54IGLk03L93F3u//xauDRpi6cr18H9nMJYvicTB/XuFDk3Sch89REH+U+zeEQWvjp0w57PV8H7ldXw6+0P8dvmC0OGJXs6jHJSUlJTq3re3t0dWVpZAUZHQ9K78161bh59++glmZmaYPHkyvL298fPPP+ODDz5AQkIChg0b9txjKJVKKJXa3ZlqVQlkchN9w6lywTM3Y92cQCT9bwGKi0tw+UYqdh2KhVeLBpptjp9PgPeQSDjYWmPkgE7YujgYXd9bgkze869UKpUK7i09MHbiFABAM/cWuH3zJvb+sAu93/YXNDYpU6v+uLPYsdNr6DswCADg1qQ54q9dweH938Oj7ctChkdUJmMdpW8oelX+ixYtwqRJk3Dnzh3s27cP3bp1w8KFCxEYGIjBgwfj7t27WLNmzXOPExkZCYVCobUUZ1SPCuD23Sz0/L8vYO8biqa9Z+GV95bAtIYJbt/7+xP004JCJKVm4dzVOxg/dzuKS1QY3r+TgFFLg72DIxq5ad9+aejWGBnpaQJFRABgo7CFiUkNuDZqrNVev4EbsjLSBYpKOuxs7WBiYoLs7Gyt9uzsbK1J2kibzICLMdIr+UdFReGrr75CbGwsDh48iPz8fJw5cwY3b97EjBkzYGdnV67jhIWFITc3V2up4VS9Pv0/LShEelYebG0s4depBQ4cu/rMbeUyGcxN+e3Jla11Gy+kJN/WaktNuQPnui4CRUQAYGpqiibuLXEv9Y5W+/27KXzMrwqYmpmhRctWOBsTrWlTqVQ4ezYanm04L4tU6ZWRUlJS0K1bNwDAK6+8AlNTU8ydOxdWVvqNgjU3N4e5ufYjPtWhyx8A/HxbQCYDEu48wEuujlg41R8JtzOweV80alqYYfr/9cJ/j19FelYu7G2tMXZQV7jUscXuny4KHbroDQ4chnEjg7B543p069ELcb9dxb7d32Pax3OEDk308vOfIv1equZ1Rto93L4ZD2ubWnB0qgv/wcPw+bwZaOnZDq292uPSuTM4f+YEPlm+XsCopeO94SMxK3w6WrXygEdrT2zdsgn5+fnw7z9A6NCMl7GW7Aai16N+crkcGRkZmvn9bWxs8Ouvv8LNze2FA6kOj/oBQEAPL8yb1Bf1nGzxMPcp/nP0MiJW7Ufe4wKYm9XApoUj0KF1I9jbWuFh7lPEXkvGp18d0kwCVB1U10f9AOD0iWNYt3I57qYmo65LfQwOHFatR/tXl0f9frsci1lTSz8y9nqvt/H+jLkAgCM/7sXu7VHIznwAF9eGGDJiHLy7vFbFkVZcdX7UDwB2bNuqmeSnuXsLTA+fCU/PNkKHVWGV/ahfcrbhfvca2hvffBZ6J/8xY8Zo5u9ftWoVgoKCoFAotLZbunSp3oFUl+QvBdU5+YtNdUn+UlDdk7/YVHbyT3louN+9BrWNL/nr9ePr2rUr4uPjNa87deqEpKQkrW1kYh8iSUREVM3plfyPHTum9fqvZ0Q5YpSIiMRE7GWs3pP8PHr0CCEhIXBwcICTkxOcnJzg4OCAiRMn4tGjR5UQIhERUdXit/r9w8OHD+Hr64t79+4hMDAQLVq0AADExcXhm2++wdGjR3HmzJlyP/JHREREVU+v5D9v3jyYmZnh1q1bcHJyKrWuZ8+emDdvHpYtW2bQIImIiKqWkZbsBqJXt//evXuxZMmSUokfAJydnbF48WLs2bPHYMEREREJQezd/nol/7S0NLRq1eqZ6z08PJCezuk6iYiIjJleyd/BwQF37tx55vrbt2+jdu3aLxoTERGRoDi3/z/06tULH3/8MQoLC0utUyqVmDVrFt544w2DBUdERCQEsXf76z3gr3379mjatClCQkLg7u4OtVqN69evY/Xq1VAqldiyZUtlxUpEREQGoFfyr1+/PqKjozFhwgSEhYXhr5mBZTIZevTogZUrV8LV1bVSAiUiIqoqMqPtsDcMvWdHdnNzw8GDB5GTk4PExEQAQJMmTXivn4iIxEPcuV//5P8XOzs7dOzY0ZCxEBERGQWR5379p/clIiKi6q2SvxSRiIio+jHWUfqGwuRPRESkQ+wD/tjtT0REJDGs/ImIiHSJu/Bn8iciItIl8tzPbn8iIiKpYeVPRESkg6P9iYiIJIaj/YmIiEhUWPkTERHpEHu3Pyt/IiIiiWHlT0REpIOVPxEREYkKK38iIiIdYh/tz+RPRESkg93+REREJCqs/ImIiHSIvPBn8iciIipF5Nmf3f5EREQSw8qfiIhIB0f7ExERSQxH+xMREZGosPInIiLSIfLCn5U/ERFRKTIDLnpatWoVGjVqBAsLC3h7e+PcuXMvejWlMPkTERHpkBnwnz6+/fZbhIaGIiIiAhcvXkSbNm3Qq1cvPHjwwKDXx+RPRERkJJYuXYrRo0dj5MiRaNmyJdauXYuaNWti48aNBj0P7/kTERHpMORof6VSCaVSqdVmbm4Oc3NzrbbCwkJcuHABYWFhmja5XA4/Pz9ER0cbLiAYUfLPv7RS6BBeiFKpRGRkJMLCwkq9oVT1xPJ+OFobza9ohYnlvRADvhflZ2HAX7058yMxd+5crbaIiAjMmTNHqy0rKwslJSVwcnLSandycsKNGzcMFxAAmVqtVhv0iBKVl5cHhUKB3Nxc1KpVS+hwJI/vh/Hge2E8+F4Io7yV//3791GvXj2cOXMGvr6+mvZp06bh+PHjOHv2rMFiqv5lBRERkRErK9GXxcHBASYmJsjIyNBqz8jIgLOzs0Fj4oA/IiIiI2BmZoaXX34ZR48e1bSpVCocPXpUqyfAEFj5ExERGYnQ0FAMHz4c7du3R8eOHbF8+XI8efIEI0eONOh5mPwNxNzcHBERERxEYyT4fhgPvhfGg++F8Rs8eDAyMzMxe/ZspKeno23btjh06FCpQYAvigP+iIiIJIb3/ImIiCSGyZ+IiEhimPyJiIgkhsmfiIhIYpj8/8WIESPg7+9f5rr8/HxERESgWbNmMDc3h4ODAwYOHIhr166V2jYvLw8ff/wx3N3dYWFhAWdnZ/j5+WH37t3geMtnGzFiBGQyGcaNG1dqXUhICGQyGUaMGKHZ9p/vVWZmJsaPH48GDRrA3Nwczs7O6NWrF06fPq11nEuXLmHgwIFwcnKChYUFmjZtitGjRyMhIaEyL63a++u9kclkMDMzQ5MmTTBv3jwUFxfj2LFjmnW6S3p6OgBgzpw5mjYTExO4urpizJgxePjwocBXVr2kpqYiODgYLi4uMDMzQ8OGDTF58mRkZ2drtnnttdc0P2sLCws0a9YMkZGRZf7t+eGHH9CtWzfY2dnB0tISzZs3R3BwMC5dulSVl0VVgMm/ApRKJfz8/LBx40bMnz8fCQkJ+PHHH1FcXAxvb2/ExMRotn306BE6deqEzZs3IywsDBcvXsSJEycwePBgTJs2Dbm5uQJeifFzdXXFzp07kZ+fr2krKCjA9u3b0aBBg2fuFxAQgEuXLmHTpk1ISEjAvn378Nprr2n9UTxw4AB8fHygVCqxbds2XL9+HVu3boVCocCsWbMq9brE4I033kBaWhoSExPxwQcfYM6cOfjss8806+Pj45GWlqa11KlTR7O+VatWSEtLQ0pKCqKionDo0CGMHz9eiEuplpKSktC+fXskJiZix44duHnzJtauXauZEOafH6RGjx6NtLQ0xMfHIywsDLNnz8batWu1jjd9+nQMHjwYbdu2xb59+xAfH4/t27ejcePGWl80QyKhpmcaPny4ul+/fqXaFy1apJbJZOrLly9rtZeUlKjbt2+vbtmypVqlUqnVarV6/PjxaisrK/W9e/dKHef3339XFxUVVUrsYvDXz9/Dw0O9detWTfu2bdvUnp6e6n79+qmHDx+uta1arVbn5OSoAaiPHTv2zGM/efJE7eDgoPb39y9zfU5OjqEuQ5TK+t3o0aOH2sfHR/3LL7+oAfzrzzAiIkLdpk0brbbQ0FC1nZ2d4YMVqTfeeENdv3599dOnT7Xa09LS1DVr1lSPGzdOrVar1a+++qp68uTJWtu0a9dO3b9/f83r6OhoNQD1F198Uea5/vp7RuLByr8Ctm/fjh49eqBNmzZa7XK5HFOnTkVcXByuXLkClUqFnTt3IjAwEC4uLqWOY21tjRo1OM/S8wQHByMqKkrzeuPGjf8625W1tTWsra2xd+/eUl+m8ZfDhw8jKysL06ZNK3O9ra3tC8UsRZaWligsLKzQvnfu3MHhw4dhZmZm4KjE6eHDhzh8+DAmTJgAS0tLrXXOzs4IDAzEt99+W6prX61W4+TJk7hx44bWz3rHjh2wtrbGhAkTyjyfzJDfb0tGgcm/AhISEtCiRYsy1/3VnpCQgKysLOTk5MDd3b0qwxOdoKAgnDp1CsnJyUhOTsbp06cRFBT0zO1r1KiBb775Bps2bYKtrS06d+6M8PBw/Prrr5ptEhMTAYDvjQGo1WocOXIEhw8fRrdu3TTt9evX13wQs7a2RqtWrbT2u3r1KqytrWFpaQk3Nzdcu3YN06dPr+rwq6XExESo1ep//TuUk5ODzMxMAMDq1athbW0Nc3NzdO3aFSqVCu+//75m+4SEBDRu3FirGFm6dKnW+8dblOLCsrOCdD9RV3Qbej5HR0f06dMH33zzDdRqNfr06QMHB4d/3ScgIAB9+vTByZMnERMTg4MHD2Lx4sX4+uuvMWLECL43BnDgwAFYW1ujqKgIKpUK7777LubMmYPz588DAE6ePAkbGxvN9qamplr7N2/eHPv27UNBQQG2bt2Ky5cvY9KkSVV6DdVdef8/DgwMxMcff4ycnBxERESgU6dO6NSp07/uExwcjL59++Ls2bMICgri74zIsPKvgGbNmuH69etlrvurvVmzZnB0dIStrS1u3LhRleGJUnBwsKaaDw4OLtc+FhYW6NGjB2bNmoUzZ85gxIgRiIiIAPDH+wOA780LeP3113H58mUkJiYiPz8fmzZtgpWVlWa9m5sbmjRpolkaNmyotf9fTwl4eHhg0aJFMDExwdy5c6v6MqqlJk2aQCaT/evfITs7Ozg6OgIAFAoFmjRpgg4dOmDXrl1YuXIljhw5otm+adOmSEpKQlFRkabN1tYWTZo0Qb169Sr3YkgQTP4VMGTIEBw5cgRXrlzRalepVFi2bBlatmyJNm3aQC6XY8iQIdi2bRvu379f6jiPHz9GcXFxVYVdrb3xxhsoLCxEUVERevXqVaFjtGzZEk+ePAEA9OzZEw4ODli8eHGZ2z569KiioUqGlZUVmjRpggYNGhhk7MrMmTOxZMmSMn9XSJu9vT169OiB1atXaz0JAwDp6enYtm0bBg8eXOa9emtra0yePBkffvihppofOnQoHj9+jNWrV1dJ/CQ8Jv/nyM3NxeXLl7WWoKAgdOzYEW+//Ta+++47pKSk4Pz58wgICMD169exYcMGzS/dggUL4OrqCm9vb2zevBlxcXFITEzExo0b4eXlhcePHwt8hdWDiYkJrl+/jri4OJiYmPzrttnZ2ejWrRu2bt2KX3/9Fbdv38Z3332HxYsXo1+/fgD+SFxff/01/vvf/6Jv3744cuQI7ty5g9jYWEybNq3MuQVIPw8ePEB6errW8s/KUpevry88PT2xcOHCKoyy+lq5ciWUSiV69eqFEydOIDU1FYcOHUKPHj1Qr149LFiw4Jn7jh07FgkJCfjhhx8A/PGz/+CDD/DBBx8gNDRUM8YmJiZG8/dMLme6EBPe83+OY8eOwcvLS6tt1KhR+Pnnn7Fw4UKEh4cjOTkZNjY2eP311xETEwMPDw/NtrVr10ZMTAwWLVqE+fPnIzk5GXZ2dmjdujU+++wzKBSKqr6kaqtWrVrl2s7a2hre3t5YtmwZbt26haKiIri6umL06NEIDw/XbNevXz+cOXMGkZGRePfdd5GXlwdXV1d069YN8+fPr6zLkIzmzZuXaouOjoaPj88z95k6dSpGjBiB6dOnw9XVtTLDq/aaNm2K2NhYREREYNCgQXj48CGcnZ3h7++PiIgI1K5d+5n71q5dG8OGDcOcOXMwYMAAyOVyLFmyBB07dsSaNWuwceNGPH36FE5OTujatSuio6PL/ftH1QO/0peIiEhi2I9DREQkMUz+REREEsPkT0REJDFM/kRERBLD5E9ERCQxTP5EREQSw+RPREQkMUz+REREEsPkT0REJDFM/kRERBLD5E9ERCQxTP5EREQS8/+ep2vGGUD1dwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "labels, cm = get_lds_cm(lds)\n",
    "plot_cm(cm, labels=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "ae70897e",
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus.to_disk(\"./data/conll2003\", overwrite=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "e9fa04af",
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus.train_ds.to_spacy(\"./data/conll2003\")\n",
    "corpus.dev_ds.to_spacy(\"./data/conll2003\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "4b7f3e4c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting en-core-web-md==3.6.0\n",
      "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_md-3.6.0/en_core_web_md-3.6.0-py3-none-any.whl (42.8 MB)\n",
      "\u001b[2K     \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.8/42.8 MB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0mm\n",
      "\u001b[?25hRequirement already satisfied: spacy<3.7.0,>=3.6.0 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from en-core-web-md==3.6.0) (3.6.1)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (3.0.12)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (1.0.4)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (1.0.9)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.0.7)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (3.0.8)\n",
      "Requirement already satisfied: thinc<8.2.0,>=8.1.8 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (8.1.12)\n",
      "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (1.1.2)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.4.7)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.0.9)\n",
      "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (0.9.0)\n",
      "Requirement already satisfied: pathy>=0.10.0 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (0.10.2)\n",
      "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (6.3.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (4.66.1)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (1.24.4)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.31.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.3.0)\n",
      "Requirement already satisfied: jinja2 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (3.1.2)\n",
      "Requirement already satisfied: setuptools in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (68.1.2)\n",
      "Requirement already satisfied: packaging>=20.0 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (23.1)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (3.3.0)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (0.5.0)\n",
      "Requirement already satisfied: pydantic-core==2.6.3 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.6.3)\n",
      "Requirement already satisfied: typing-extensions>=4.6.1 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (4.7.1)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (3.2.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.0.4)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2023.7.22)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from thinc<8.2.0,>=8.1.8->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (0.7.10)\n",
      "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from thinc<8.2.0,>=8.1.8->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (0.1.1)\n",
      "Requirement already satisfied: click<9.0.0,>=7.1.1 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from typer<0.10.0,>=0.3.0->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (8.1.7)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /Users/kabirkhan/Documents/recon/.venv/lib/python3.11/site-packages (from jinja2->spacy<3.7.0,>=3.6.0->en-core-web-md==3.6.0) (2.1.3)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.1.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('en_core_web_md')\n",
      "\u001b[38;5;4mℹ Saving to output directory: models/en_conll_md\u001b[0m\n",
      "\u001b[38;5;4mℹ Using CPU\u001b[0m\n",
      "\u001b[1m\n",
      "=========================== Initializing pipeline ===========================\u001b[0m\n",
      "\u001b[38;5;2m✔ Initialized pipeline\u001b[0m\n",
      "\u001b[1m\n",
      "============================= Training pipeline =============================\u001b[0m\n",
      "\u001b[38;5;4mℹ Pipeline: ['tok2vec', 'ner']\u001b[0m\n",
      "\u001b[38;5;4mℹ Initial learn rate: 0.001\u001b[0m\n",
      "E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE \n",
      "---  ------  ------------  --------  ------  ------  ------  ------\n",
      "  0       0          0.00     44.28    0.00    0.00    0.00    0.00\n",
      "  0     200         82.58   2838.65   52.41   60.56   46.20    0.52\n",
      "  0     400       1039.30   2465.74   63.43   64.10   62.77    0.63\n",
      "  0     600        256.40   2220.14   71.76   74.24   69.44    0.72\n",
      "  0     800        331.60   2215.43   76.00   76.09   75.90    0.76\n",
      "  0    1000        424.40   2506.67   77.33   77.49   77.16    0.77\n",
      "  1    1200        747.23   2470.54   81.29   82.58   80.04    0.81\n",
      "  1    1400        557.30   2108.81   81.58   81.55   81.61    0.82\n",
      "  1    1600        731.19   2369.50   83.59   84.28   82.92    0.84\n",
      "  2    1800        805.75   2527.25   84.53   85.11   83.96    0.85\n",
      "  2    2000        930.29   2263.20   84.68   84.35   85.01    0.85\n",
      "  3    2200       1026.41   2208.37   86.23   86.62   85.85    0.86\n",
      "  4    2400       1033.96   1906.29   86.28   87.67   84.94    0.86\n",
      "  5    2600       1214.51   1830.59   85.98   86.73   85.24    0.86\n",
      "  6    2800        916.89   1385.59   86.13   86.54   85.73    0.86\n",
      "  7    3000        990.98   1218.15   86.06   86.63   85.49    0.86\n",
      "  8    3200        937.03   1127.00   86.02   87.19   84.89    0.86\n",
      "  9    3400        922.85    972.14   86.42   87.56   85.31    0.86\n",
      "  9    3600        874.39    860.02   86.04   86.38   85.71    0.86\n",
      " 10    3800        972.10    791.65   85.80   86.33   85.27    0.86\n",
      " 11    4000       1506.55    928.18   85.51   85.83   85.19    0.86\n",
      " 12    4200        734.26    644.70   85.99   86.29   85.70    0.86\n",
      " 13    4400       1011.98    700.55   86.49   86.72   86.27    0.86\n",
      " 14    4600        898.05    669.54   86.10   86.76   85.46    0.86\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!spacy download en_core_web_md\n",
    "!spacy train config/config.cfg \\\n",
    "    --paths.train ./data/conll2003/train.spacy \\\n",
    "    --paths.dev ./data/conll2003/dev.spacy \\\n",
    "    --paths.vectors \"en_core_web_md\" \\\n",
    "    --output ./models/en_conll_md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "c4f2cb01",
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "from recon.insights import top_prediction_errors, get_hardest_examples\n",
    "from recon.recognizer import SpacyEntityRecognizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "e9edcb4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load(\"models/en_conll_md/model-best/\")\n",
    "rec = SpacyEntityRecognizer(nlp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "fa6ae642",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PredictionError(text='Pirelli', true_label='ORG', pred_label='PER', count=9),\n",
       " PredictionError(text='Suu Kyi', true_label='PER', pred_label='ORG', count=9),\n",
       " PredictionError(text=\"Boatmen 's\", true_label='ORG', pred_label='NOT_LABELED', count=7),\n",
       " PredictionError(text='Washington', true_label='PER', pred_label='LOC', count=6),\n",
       " PredictionError(text='Bernardin', true_label='PER', pred_label='NOT_LABELED', count=6),\n",
       " PredictionError(text='Jansher', true_label='PER', pred_label='LOC', count=6),\n",
       " PredictionError(text='King', true_label='PER', pred_label='NOT_LABELED', count=6),\n",
       " PredictionError(text='Boatmen', true_label='NOT_LABELED', pred_label='PER', count=6),\n",
       " PredictionError(text='Magna', true_label='ORG', pred_label='NOT_LABELED', count=5),\n",
       " PredictionError(text='Lukashenko', true_label='PER', pred_label='ORG', count=5)]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pe = top_prediction_errors(rec, corpus.dev)\n",
    "pe[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "6576040b-d2ea-4d82-ae9a-6ac2e996291c",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_hes = get_hardest_examples(rec, corpus.train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "2fe81868",
   "metadata": {},
   "outputs": [],
   "source": [
    "dev_hes = get_hardest_examples(rec, corpus.dev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "9c0ec835-e29f-4586-bb23-c51a9fc9e93b",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_hes = get_hardest_examples(rec, corpus.test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "a46af639-be40-489c-aae5-55a05b04e78a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    English\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    County Championship\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " cricket matches on Thursday : </div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    English County Championship\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " cricket matches on Thursday : </div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "idx = 0\n",
    "train_hes[idx].reference.show()\n",
    "train_hes[idx].prediction.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "43b3ac6c-051d-4965-a7a9-58aa6d7fdf37",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">The titles of his other novels translate as &quot; \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    In the Year of January\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " &quot; ( 1963 ) , &quot; \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    The Collapse\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " &quot; ( 1964 ) , &quot; \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Sleeping Bread\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " &quot; ( 1975 ) , &quot; \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    The Decaying Mansion\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " &quot; ( 1977 ) and &quot; \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    A World of Things\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " &quot; ( 1982 ) , followed by &quot; \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    The Knot\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " , &quot; &quot; \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Soul Alone\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " &quot; and , most recently , &quot; \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    A Woman\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " . &quot; </div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">The titles of his other novels translate as &quot; In the Year of January &quot; ( 1963 ) , &quot; The \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Collapse\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " &quot; ( 1964 ) , &quot; \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Sleeping Bread\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PER</span>\n",
       "</mark>\n",
       " &quot; ( 1975 ) , &quot; The \n",
       "<mark class=\"entity\" style=\"background: #ff9561; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Decaying Mansion\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">LOC</span>\n",
       "</mark>\n",
       " &quot; ( 1977 ) and &quot; A \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    World of Things\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " &quot; ( 1982 ) , followed by &quot; The \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Knot\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PER</span>\n",
       "</mark>\n",
       " , &quot; &quot; Soul Alone &quot; and , most recently , &quot; A \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Woman\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " . &quot; </div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "idx = 0\n",
    "dev_hes[idx].reference.show()\n",
    "dev_hes[idx].prediction.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "b1ccf373-d1cb-4f12-b76e-ff0ba7bc9cb5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    NCAA\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " AMERICAN \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    FOOTBALL-OHIO STATE\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " 'S \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    PACE\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PER</span>\n",
       "</mark>\n",
       " FIRST REPEAT \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    LOMBARDI AWARD\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " WINNER . </div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    NCAA AMERICAN\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " FOOTBALL-\n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    OHIO STATE 'S\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " PACE FIRST REPEAT \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    LOMBARDI\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PER</span>\n",
       "</mark>\n",
       " \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    AWARD\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PER</span>\n",
       "</mark>\n",
       " WINNER . </div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "idx=0\n",
    "test_hes[idx].reference.show()\n",
    "test_hes[idx].prediction.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "7d80de55-f516-4553-8f5e-86868be28edf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">SOCCER - \n",
       "<mark class=\"entity\" style=\"background: #ff9561; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    JAPAN\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">LOC</span>\n",
       "</mark>\n",
       " GET LUCKY WIN , \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    CHINA\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PER</span>\n",
       "</mark>\n",
       " IN SURPRISE DEFEAT . </div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">SOCCER - \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    JAPAN\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    GET LUCKY\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " WIN , \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    CHINA\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " IN \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    SURPRISE\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " DEFEAT . </div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "china_he = [he for he in test_hes if \"CHINA\" in he.reference.text][0]\n",
    "china_he.reference.show()\n",
    "china_he.prediction.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4b088567",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"spans\" style=\"line-height: 2.5; direction: ltr\">The titles of his other novels translate as &quot; \n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    In\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        MISC\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    the\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    Year\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    of\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    January\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "</span>\n",
       "&quot; ( 1963 ) , &quot; \n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    The\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        MISC\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    Collapse\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        ORG:PRED\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "</span>\n",
       "&quot; ( 1964 ) , &quot; \n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    Sleeping\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        MISC\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        PER:PRED\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    Bread\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "</span>\n",
       "&quot; ( 1975 ) , &quot; \n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    The\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        MISC\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    Decaying\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        ORG:PRED\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    Mansion\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "</span>\n",
       "&quot; ( 1977 ) and &quot; \n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    A\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        MISC\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    World\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        MISC:PRED\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    of\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    Things\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "</span>\n",
       "&quot; ( 1982 ) , followed by &quot; \n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    The\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        MISC\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    Knot\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        PER:PRED\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "</span>\n",
       ", &quot; &quot; \n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    Soul\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        MISC\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        MISC:PRED\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    Alone\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "</span>\n",
       "&quot; and , most recently , &quot; \n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    A\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        MISC\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "</span>\n",
       "\n",
       "<span style=\"font-weight: bold; display: inline-block; position: relative;\">\n",
       "    Woman\n",
       "    \n",
       "<span style=\"background: #ddd; top: 40px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "</span>\n",
       "\n",
       "    \n",
       "<span style=\"background: #ddd; top: 57px; height: 4px; border-top-left-radius: 3px; border-bottom-left-radius: 3px; left: -1px; width: calc(100% + 2px); position: absolute;\">\n",
       "    <span style=\"background: #ddd; z-index: 10; color: #000; top: -0.5em; padding: 2px 3px; position: absolute; font-size: 0.6em; font-weight: bold; line-height: 1; border-radius: 3px\">\n",
       "        ORG:PRED\n",
       "    </span>\n",
       "</span>\n",
       "\n",
       "\n",
       "</span>\n",
       ". &quot; </div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from spacy.tokens import Doc\n",
    "from spacy.util import get_words_and_spaces\n",
    "from spacy.vocab import Vocab\n",
    "from recon.types import HardestExample\n",
    "from spacy import displacy\n",
    "\n",
    "def he_show(he: HardestExample, label_suffix: str = \"PRED\"):\n",
    "    \n",
    "    combined = he.reference.copy(deep=True)\n",
    "    pred_spans = []\n",
    "    for s in he.prediction.spans:\n",
    "        span = s.copy(deep=True)\n",
    "        span.label = f\"{s.label}:{label_suffix}\"\n",
    "        pred_spans.append(span)\n",
    "\n",
    "    combined.spans = sorted(combined.spans + pred_spans, key=lambda s: s.start)\n",
    "    \n",
    "    tokens = [token.text for token in combined.tokens]\n",
    "    words, spaces = get_words_and_spaces(tokens, combined.text)\n",
    "    doc = Doc(Vocab(), words=words, spaces=spaces)\n",
    "    doc.spans[\"ref\"] = [doc.char_span(s.start, s.end, label=s.label) for s in combined.spans]\n",
    "    displacy.render(doc, style=\"span\", jupyter=True, options={\"spans_key\": \"ref\"})\n",
    "\n",
    "\n",
    "he_show(he[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "4005653b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving dataset to prodigy dataset: dev_671418edc919b3e9\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'dev_671418edc919b3e9'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.dev_ds.to_prodigy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "0a99bad0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/kabirkhan/Library/Caches/pypoetry/virtualenvs/reconner-cI2LIzZM-py3.10/lib/python3.10/site-packages/spacy/util.py:837: UserWarning: [W095] Model 'en_pipeline' (0.0.0) was trained with spaCy v3.2 and may not be 100% compatible with the current version (3.3.0). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate\n",
      "  warnings.warn(warn_msg)\n",
      "Using 4 labels from model: LOC, MISC, ORG, PER\n",
      "\n",
      "✨  Starting the web server at http://localhost:8080 ...\n",
      "Open the app in your browser and start annotating!\n",
      "\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!prodigy recon.ner_correct conll_he ./models/en_conll_sm/model-best dataset:dev_671418edc919b3e9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "360f42e3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "vscode": {
   "interpreter": {
    "hash": "8b3ae71f03514a19ab794870705b50e4422d52f261458df48e0c827ff20c44d6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}