docs/performance_ner.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Named Entity Recognition\n",
    "\n",
    "\n",
    "This page examines the performance of competing models for Danish named entity recognition over multiple datasets. Performance is not limited to \n",
    "accuracy, but also includes domain generalization, biases and robustness. This page is also a notebook, which can be opened and run to replicate the results."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## State-of-the-Art comparison\n",
    "To our knowledge there exists three datasets for Danish named entity recognition;\n",
    "\n",
    "1) DaNE {cite}`hvingelby2020dane`, which uses the simple annotation scheme of CoNLL 2003 {cite}`sang2003introduction` with the entities; *person*, *location*, *organization*, and *miscellaneus*.\n",
    "2) [DANSK](https://huggingface.co/datasets/chcaa/DANSK), which uses the extensive annotation scheme similar to that of OntoNotes 5.0 {cite}`weischedel2013ontonotes` including more that 16 entity types.\n",
    "3) and DAN+ {cite}`plank2021dan+`, which also uses the annotation scheme of CoNLL 2003, but allows for nested entities for instance *Aarhus Universitet*, where *Aarhus* is a location and *Aarhus Universitet* is an organization.\n",
    "\n",
    "In this comparison we will be examing performance on DaNE and DANSK, but as no known models have been trained on Danish nested entities, we will not be comparing performance on DAN+.\n",
    "\n",
    "\n",
    "```{admonition} Measuring Performance\n",
    "Typically when measuring performance on these benchmark it is normal to feed the model the gold standard tokens. While this allows for easier comparisons of modules and architectures, it inflates the performance metrics. Further, it does not proberly reflect what you are really interested in:\n",
    "*the performance you can expect when you apply the model to data of a similar type*. Therefore we estimate the model is given no prior knowledge of the data, and only the raw text is fed to the model. Thus the performance metrics might be slightly different compared to e.g. DaNLP.\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DaNE: Simple Named Entity Recognition\n",
    "As already stated DaNE uses an extraction from the CoNLL 2003 dataset, which is as follows {cite}`hvingelby2020dane`:\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "| Entity | Description |\n",
    "|--------------|-------------|\n",
    "| LOC          | includes locations like cities, roads and mountains, as well as both public and commercial places like specific buildings or meeting points, but also abstract places. |\n",
    "| PERSON | consists of names of people, fictional characters, and animals. The names includes aliases. |\n",
    "| ORG | can be summarized as all sorts of organizations and collections of people, ranging from companies, brands, political movements, governmental bodies and clubs. |\n",
    "| MISC | is a broad category of e.g. events, languages, titles and religions, but this tag also includes words derived from one of the four tags as well as words for which one part is from one of the three other tags. |\n",
    "\n",
    "Here is an example from the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "from spacy import displacy\n",
    "from spacy.tokens import Span\n",
    "\n",
    "text = \"\"\"To kendte russiske historikere Andronik Mirganjan og Igor Klamkin tror ikke, at Rusland kan udvikles uden en \"jernnæve\".\"\"\"\n",
    "nlp = spacy.blank(\"da\")\n",
    "doc = nlp(text)\n",
    "doc.ents = [  # type: ignore\n",
    "    Span(doc, 2, 3, label=\"MISC\"),\n",
    "    Span(doc, 4, 6, label=\"PERSON\"),\n",
    "    Span(doc, 7, 9, label=\"PERSON\"),\n",
    "    Span(doc, 13, 14, label=\"LOC\"),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">To kendte \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    russiske\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">MISC</span>\n",
       "</mark>\n",
       " historikere \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Andronik Mirganjan\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " og \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Igor Klamkin\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " tror ikke, at \n",
       "<mark class=\"entity\" style=\"background: #ff9561; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Rusland\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">LOC</span>\n",
       "</mark>\n",
       " kan udvikles uden en &quot;jernnæve&quot;.</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The table below shows the performance of Danish language processing pipelines scored on the DaNE test set. The best scores in each category are highlighted with bold and the second best is underlined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "from evaluation.models import MODELS\n",
    "from evaluation.utils import apply_models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dane (test): Loading prediction for da_dacy_large_trf-0.2.0\n",
      "dane (test): Loading prediction for da_dacy_medium_trf-0.2.0\n",
      "dane (test): Loading prediction for da_dacy_small_trf-0.2.0\n",
      "dane (test): Loading prediction for da_dacy_large_ner_fine_grained-0.1.0\n",
      "dane (test): Loading prediction for da_dacy_medium_ner_fine_grained-0.1.0\n",
      "dane (test): Loading prediction for da_dacy_small_ner_fine_grained-0.1.0\n",
      "dane (test): Loading prediction for saattrupdan/nbailab-base-ner-scandi\n",
      "dane (test): Loading prediction for alexandrainst/da-ner-base\n",
      "dane (test): Loading prediction for da_core_news_trf-3.5.0\n",
      "dane (test): Loading prediction for da_core_news_lg-3.5.0\n",
      "dane (test): Loading prediction for da_core_news_md-3.5.0\n",
      "dane (test): Loading prediction for da_core_news_sm-3.5.0\n",
      "dane (test): Loading prediction for openai/gpt-3.5-turbo (02/05/23)\n",
      "dane (test): Loading prediction for openai/gpt-4 (02/05/23)\n"
     ]
    }
   ],
   "source": [
    "dane = {}\n",
    "for mdl_name, model_getter in MODELS.items():\n",
    "    mdl_results = apply_models(mdl_name, model_getter, dataset=\"dane\", splits=[\"test\"])\n",
    "    dane[mdl_name] = mdl_results[\"test\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "# normalize labels to match the dataset\n",
    "for mdl in dane:\n",
    "    if \"openai\" not in mdl:\n",
    "        continue\n",
    "    examples = dane[mdl][\"examples\"]\n",
    "    mapping = {\n",
    "        \"PERSON\": \"PER\",\n",
    "        \"ORGANISATION\": \"ORG\",\n",
    "        \"LOCATION\": \"LOC\",\n",
    "    }\n",
    "    for e in examples:\n",
    "        ents = e.x.ents\n",
    "        for ent in ents:\n",
    "            ent.label_ = mapping[ent.label_]\n",
    "\n",
    "        e.x.ents = ents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from evaluation.utils import create_dataframe\n",
    "\n",
    "\n",
    "def highlight_max(s: pd.Series) -> list:\n",
    "    \"\"\"Highlight the maximum in a Series with bold text.\"\"\"\n",
    "    # convert to str for comparison\n",
    "    s = s.astype(str)\n",
    "    is_max = s == s.max()\n",
    "    return [\"font-weight: bold\" if v else \"\" for v in is_max]\n",
    "\n",
    "\n",
    "def underline_second_max(s: pd.Series) -> list:\n",
    "    \"\"\"Underline the second maximum in a Series.\"\"\"\n",
    "    is_second_max = s == s.sort_values(ascending=False).iloc[1]\n",
    "    return [\"text-decoration: underline\" if v else \"\" for v in is_second_max]\n",
    "\n",
    "\n",
    "def create_table(\n",
    "    df: pd.DataFrame,\n",
    "    caption=\"F1 score with 95% confidence interval calculated using bootstrapping with 100 samples.\",\n",
    "):\n",
    "    # replace index with range\n",
    "    df.index = range(len(df))  # type: ignore\n",
    "\n",
    "    col_names = [(\"\", \"Models\")] + [(\"F1\", col) for col in df.columns[1:]]\n",
    "    super_header = pd.MultiIndex.from_tuples(col_names)\n",
    "    df.columns = super_header\n",
    "\n",
    "    s = df.style.apply(highlight_max, axis=0, subset=df.columns[1:])\n",
    "    s = s.apply(underline_second_max, axis=0, subset=df.columns[1:])\n",
    "\n",
    "    # Add a caption\n",
    "    s = s.set_caption(caption)\n",
    "\n",
    "    # Center the header and left align the model names\n",
    "    s = s.set_properties(subset=df.columns[1:], **{\"text-align\": \"right\"})\n",
    "\n",
    "    super_header_style = [\n",
    "        {\"selector\": \".level0\", \"props\": [(\"text-align\", \"center\")]},\n",
    "        {\"selector\": \".col_heading\", \"props\": [(\"text-align\", \"center\")]},\n",
    "    ]\n",
    "    # Apply the CSS style to the styler\n",
    "    s = s.set_table_styles(super_header_style)  # type: ignore\n",
    "    s = s.set_properties(subset=[(\"\", \"Models\")], **{\"text-align\": \"left\"})\n",
    "    # remove the index\n",
    "    s = s.hide(axis=\"index\")\n",
    "\n",
    "    # smaller font size\n",
    "    s = s.set_table_attributes('style=\"font-size: 0.65em\"')\n",
    "    return s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "from multiprocessing import Pool\n",
    "\n",
    "with Pool(8) as p:\n",
    "    tables = p.starmap(\n",
    "        create_dataframe,\n",
    "        [\n",
    "            (dane[mdl][\"examples\"], mdl, 1, 500)\n",
    "            for mdl in dane\n",
    "            if \"fine_grained\" not in mdl\n",
    "        ],\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_98f1b .level0 {\n",
       "  text-align: center;\n",
       "}\n",
       "#T_98f1b .col_heading {\n",
       "  text-align: center;\n",
       "}\n",
       "#T_98f1b_row0_col0, #T_98f1b_row1_col0, #T_98f1b_row2_col0, #T_98f1b_row3_col0, #T_98f1b_row4_col0, #T_98f1b_row5_col0, #T_98f1b_row6_col0, #T_98f1b_row7_col0, #T_98f1b_row8_col0, #T_98f1b_row9_col0, #T_98f1b_row10_col0 {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_98f1b_row0_col1, #T_98f1b_row0_col4, #T_98f1b_row1_col3, #T_98f1b_row1_col5, #T_98f1b_row3_col2 {\n",
       "  text-decoration: underline;\n",
       "  text-align: right;\n",
       "}\n",
       "#T_98f1b_row0_col2, #T_98f1b_row0_col5, #T_98f1b_row3_col1, #T_98f1b_row3_col3, #T_98f1b_row3_col4 {\n",
       "  font-weight: bold;\n",
       "  text-align: right;\n",
       "}\n",
       "#T_98f1b_row0_col3, #T_98f1b_row1_col1, #T_98f1b_row1_col2, #T_98f1b_row1_col4, #T_98f1b_row2_col1, #T_98f1b_row2_col2, #T_98f1b_row2_col3, #T_98f1b_row2_col4, #T_98f1b_row2_col5, #T_98f1b_row3_col5, #T_98f1b_row4_col1, #T_98f1b_row4_col2, #T_98f1b_row4_col3, #T_98f1b_row4_col4, #T_98f1b_row4_col5, #T_98f1b_row5_col1, #T_98f1b_row5_col2, #T_98f1b_row5_col3, #T_98f1b_row5_col4, #T_98f1b_row5_col5, #T_98f1b_row6_col1, #T_98f1b_row6_col2, #T_98f1b_row6_col3, #T_98f1b_row6_col4, #T_98f1b_row6_col5, #T_98f1b_row7_col1, #T_98f1b_row7_col2, #T_98f1b_row7_col3, #T_98f1b_row7_col4, #T_98f1b_row7_col5, #T_98f1b_row8_col1, #T_98f1b_row8_col2, #T_98f1b_row8_col3, #T_98f1b_row8_col4, #T_98f1b_row8_col5, #T_98f1b_row9_col1, #T_98f1b_row9_col2, #T_98f1b_row9_col3, #T_98f1b_row9_col4, #T_98f1b_row9_col5, #T_98f1b_row10_col1, #T_98f1b_row10_col2, #T_98f1b_row10_col3, #T_98f1b_row10_col4, #T_98f1b_row10_col5 {\n",
       "  text-align: right;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_98f1b\" style=\"font-size: 0.65em\">\n",
       "  <caption>F1 score with 95% confidence interval calculated using bootstrapping with 500 samples.</caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th id=\"T_98f1b_level0_col0\" class=\"col_heading level0 col0\" ></th>\n",
       "      <th id=\"T_98f1b_level0_col1\" class=\"col_heading level0 col1\" colspan=\"5\">F1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_98f1b_level1_col0\" class=\"col_heading level1 col0\" >Models</th>\n",
       "      <th id=\"T_98f1b_level1_col1\" class=\"col_heading level1 col1\" >Average</th>\n",
       "      <th id=\"T_98f1b_level1_col2\" class=\"col_heading level1 col2\" >Location</th>\n",
       "      <th id=\"T_98f1b_level1_col3\" class=\"col_heading level1 col3\" >Person</th>\n",
       "      <th id=\"T_98f1b_level1_col4\" class=\"col_heading level1 col4\" >Organization</th>\n",
       "      <th id=\"T_98f1b_level1_col5\" class=\"col_heading level1 col5\" >Misc.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td id=\"T_98f1b_row0_col0\" class=\"data row0 col0\" >da_dacy_large_trf-0.2.0</td>\n",
       "      <td id=\"T_98f1b_row0_col1\" class=\"data row0 col1\" >85.4 (81.2, 88.9)</td>\n",
       "      <td id=\"T_98f1b_row0_col2\" class=\"data row0 col2\" >89.5 (84.0, 94.7)</td>\n",
       "      <td id=\"T_98f1b_row0_col3\" class=\"data row0 col3\" >92.6 (89.0, 95.4)</td>\n",
       "      <td id=\"T_98f1b_row0_col4\" class=\"data row0 col4\" >79.0 (72.5, 84.6)</td>\n",
       "      <td id=\"T_98f1b_row0_col5\" class=\"data row0 col5\" >79.0 (70.8, 86.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_98f1b_row1_col0\" class=\"data row1 col0\" >da_dacy_medium_trf-0.2.0</td>\n",
       "      <td id=\"T_98f1b_row1_col1\" class=\"data row1 col1\" >84.9 (81.0, 88.5)</td>\n",
       "      <td id=\"T_98f1b_row1_col2\" class=\"data row1 col2\" >86.8 (81.2, 92.3)</td>\n",
       "      <td id=\"T_98f1b_row1_col3\" class=\"data row1 col3\" >92.7 (89.2, 95.6)</td>\n",
       "      <td id=\"T_98f1b_row1_col4\" class=\"data row1 col4\" >78.7 (71.8, 85.0)</td>\n",
       "      <td id=\"T_98f1b_row1_col5\" class=\"data row1 col5\" >78.7 (70.6, 86.1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_98f1b_row2_col0\" class=\"data row2 col0\" >da_dacy_small_trf-0.2.0</td>\n",
       "      <td id=\"T_98f1b_row2_col1\" class=\"data row2 col1\" >82.7 (79.3, 85.9)</td>\n",
       "      <td id=\"T_98f1b_row2_col2\" class=\"data row2 col2\" >84.2 (78.3, 89.8)</td>\n",
       "      <td id=\"T_98f1b_row2_col3\" class=\"data row2 col3\" >92.2 (88.5, 95.1)</td>\n",
       "      <td id=\"T_98f1b_row2_col4\" class=\"data row2 col4\" >75.9 (69.3, 81.7)</td>\n",
       "      <td id=\"T_98f1b_row2_col5\" class=\"data row2 col5\" >75.7 (68.8, 81.8)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_98f1b_row3_col0\" class=\"data row3 col0\" >saattrupdan/nbailab-base-ner-scandi</td>\n",
       "      <td id=\"T_98f1b_row3_col1\" class=\"data row3 col1\" >86.3 (82.4, 89.7)</td>\n",
       "      <td id=\"T_98f1b_row3_col2\" class=\"data row3 col2\" >88.6 (83.0, 93.3)</td>\n",
       "      <td id=\"T_98f1b_row3_col3\" class=\"data row3 col3\" >95.1 (92.4, 97.8)</td>\n",
       "      <td id=\"T_98f1b_row3_col4\" class=\"data row3 col4\" >80.3 (73.6, 85.8)</td>\n",
       "      <td id=\"T_98f1b_row3_col5\" class=\"data row3 col5\" >78.6 (69.4, 86.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_98f1b_row4_col0\" class=\"data row4 col0\" >alexandrainst/da-ner-base</td>\n",
       "      <td id=\"T_98f1b_row4_col1\" class=\"data row4 col1\" >70.7 (66.2, 75.2)</td>\n",
       "      <td id=\"T_98f1b_row4_col2\" class=\"data row4 col2\" >84.8 (77.8, 91.0)</td>\n",
       "      <td id=\"T_98f1b_row4_col3\" class=\"data row4 col3\" >90.3 (86.3, 93.9)</td>\n",
       "      <td id=\"T_98f1b_row4_col4\" class=\"data row4 col4\" >64.7 (57.0, 71.3)</td>\n",
       "      <td id=\"T_98f1b_row4_col5\" class=\"data row4 col5\" > </td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_98f1b_row5_col0\" class=\"data row5 col0\" >da_core_news_trf-3.5.0</td>\n",
       "      <td id=\"T_98f1b_row5_col1\" class=\"data row5 col1\" >79.0 (75.1, 82.3)</td>\n",
       "      <td id=\"T_98f1b_row5_col2\" class=\"data row5 col2\" >82.1 (75.5, 88.5)</td>\n",
       "      <td id=\"T_98f1b_row5_col3\" class=\"data row5 col3\" >91.6 (88.2, 94.5)</td>\n",
       "      <td id=\"T_98f1b_row5_col4\" class=\"data row5 col4\" >68.0 (61.0, 75.2)</td>\n",
       "      <td id=\"T_98f1b_row5_col5\" class=\"data row5 col5\" >69.0 (61.1, 77.3)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_98f1b_row6_col0\" class=\"data row6 col0\" >da_core_news_lg-3.5.0</td>\n",
       "      <td id=\"T_98f1b_row6_col1\" class=\"data row6 col1\" >74.6 (70.8, 78.1)</td>\n",
       "      <td id=\"T_98f1b_row6_col2\" class=\"data row6 col2\" >81.6 (75.3, 88.2)</td>\n",
       "      <td id=\"T_98f1b_row6_col3\" class=\"data row6 col3\" >85.5 (81.1, 89.9)</td>\n",
       "      <td id=\"T_98f1b_row6_col4\" class=\"data row6 col4\" >62.7 (54.8, 70.3)</td>\n",
       "      <td id=\"T_98f1b_row6_col5\" class=\"data row6 col5\" >64.4 (55.9, 72.8)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_98f1b_row7_col0\" class=\"data row7 col0\" >da_core_news_md-3.5.0</td>\n",
       "      <td id=\"T_98f1b_row7_col1\" class=\"data row7 col1\" >71.2 (66.9, 75.2)</td>\n",
       "      <td id=\"T_98f1b_row7_col2\" class=\"data row7 col2\" >76.8 (69.9, 83.6)</td>\n",
       "      <td id=\"T_98f1b_row7_col3\" class=\"data row7 col3\" >82.6 (77.8, 87.0)</td>\n",
       "      <td id=\"T_98f1b_row7_col4\" class=\"data row7 col4\" >58.2 (49.6, 66.7)</td>\n",
       "      <td id=\"T_98f1b_row7_col5\" class=\"data row7 col5\" >61.8 (52.6, 70.6)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_98f1b_row8_col0\" class=\"data row8 col0\" >da_core_news_sm-3.5.0</td>\n",
       "      <td id=\"T_98f1b_row8_col1\" class=\"data row8 col1\" >64.4 (59.7, 68.5)</td>\n",
       "      <td id=\"T_98f1b_row8_col2\" class=\"data row8 col2\" >61.6 (52.2, 69.9)</td>\n",
       "      <td id=\"T_98f1b_row8_col3\" class=\"data row8 col3\" >80.1 (74.9, 85.1)</td>\n",
       "      <td id=\"T_98f1b_row8_col4\" class=\"data row8 col4\" >49.0 (39.0, 57.5)</td>\n",
       "      <td id=\"T_98f1b_row8_col5\" class=\"data row8 col5\" >58.4 (49.8, 67.1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_98f1b_row9_col0\" class=\"data row9 col0\" >openai/gpt-3.5-turbo (02/05/23)</td>\n",
       "      <td id=\"T_98f1b_row9_col1\" class=\"data row9 col1\" >57.5 (52.3, 62.2)</td>\n",
       "      <td id=\"T_98f1b_row9_col2\" class=\"data row9 col2\" >50.7 (41.9, 59.2)</td>\n",
       "      <td id=\"T_98f1b_row9_col3\" class=\"data row9 col3\" >81.9 (76.8, 86.5)</td>\n",
       "      <td id=\"T_98f1b_row9_col4\" class=\"data row9 col4\" >55.7 (47.1, 63.7)</td>\n",
       "      <td id=\"T_98f1b_row9_col5\" class=\"data row9 col5\" > </td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_98f1b_row10_col0\" class=\"data row10 col0\" >openai/gpt-4 (02/05/23)</td>\n",
       "      <td id=\"T_98f1b_row10_col1\" class=\"data row10 col1\" >70.1 (66.0, 74.3)</td>\n",
       "      <td id=\"T_98f1b_row10_col2\" class=\"data row10 col2\" >78.9 (71.5, 85.7)</td>\n",
       "      <td id=\"T_98f1b_row10_col3\" class=\"data row10 col3\" >85.3 (80.4, 89.5)</td>\n",
       "      <td id=\"T_98f1b_row10_col4\" class=\"data row10 col4\" >72.0 (65.4, 78.5)</td>\n",
       "      <td id=\"T_98f1b_row10_col5\" class=\"data row10 col5\" > </td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x176781ba0>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.concat(tables)\n",
    "# sort columns\n",
    "df = df[[\"Models\", \"Average\", \"Location\", \"Person\", \"Organization\", \"Misc.\"]]\n",
    "df_average = df[\"Average\"]\n",
    "create_table(\n",
    "    df,\n",
    "    \"F1 score with 95% confidence interval calculated using bootstrapping with 500 samples.\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is worth mentioning that while the `da_dacy_large_trf-0.2.0` and `saattrupdan/nbailab-base-ner-scandi` performs similarly they have their independent strength and weaknesses. The large DaCy model is a multi-task model performing named-entity recognition as only one of its many tasks and thus if you wish to use one of those we would recommend that model. On the other hand the `nbailab-base-ner-scandi` is trained on multiple Scandinavian languages and thus might be ideal if your dataset might contain these languages as well. `saattrupdan/nbailab-base-ner-scandi` is available in DaCy using `nlp.add_pipe(\"dacy/ner\")`.\n",
    "\n",
    "```{admonition} You are missing a model\n",
    ":class: note\n",
    "\n",
    "These tables are continually updated and thus we try to limit the number of models to only the most relevant Danish models. Therefore models like Polyglot with strict requirements and consistently worse performance are excluded. If you want to see a specific model, please open an issue on GitHub.\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DANSK: Fine-grained Named Entity Recognition\n",
    "\n",
    "DANSK is annotated from the Danish Gigaword Corpus {cite}`derczynski2021danish` and a wide variety of domains including conversational, legal, news, social media, web content,  wiki's and Books. Dansk follows includes the following labels:\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "|  Entity        |             Description                                         |\n",
    "| -------- | ---------------------------------------------------- |\n",
    "| PERSON   | People, including fictional                          |\n",
    "| NORP     | Nationalities or religious or political groups       |\n",
    "| FACILITY | Building, airports, highways, bridges, etc.          |\n",
    "| ORGANIZATION | Companies, agencies, institutions, etc.              |\n",
    "| GPE      | Countries, cities, states.                           |\n",
    "| LOCATION | Non-GPE locations, mountain ranges, bodies of water  |\n",
    "| PRODUCT  | Vehicles, weapons, foods, etc. (not services)        |\n",
    "| EVENT    | Named hurricanes, battles, wars, sports events, etc. |\n",
    "| WORK OF ART | Titles of books, songs, etc.                         |\n",
    "| LAW      | Named documents made into laws                       |\n",
    "| LANGUAGE | Any named language                                   |\n",
    "\n",
    "As well as annotation for the following concepts:\n",
    "\n",
    "|   Entity       |   Description                                         |\n",
    "| -------- | ------------------------------------------- |\n",
    "| DATE     | Absolute or relative dates or periods       |\n",
    "| TIME     | Times smaller than a day                    |\n",
    "| PERCENT  | Percentage (including \"*\"%)                |\n",
    "| MONEY    | Monetary values, including unit             |\n",
    "| QUANTITY | Measurements, as of weight or distance      |\n",
    "| ORDINAL  | \"first\", \"second\"                           |\n",
    "| CARDINAL | Numerals that do no fall under another type |\n",
    "\n",
    "\n",
    "We have here opted to create an interactive chart over a table as with the number of labels it quickly becomes unruly. The chart is interactive and you can select the label you want to compare the models on. You can also hover over the dots the see the exact values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "functools.partial(<function openai_model_loader_fine_ner at 0x169e02b00>, model='gpt-4')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from functools import partial\n",
    "\n",
    "from evaluation.models import openai_model_loader_fine_ner\n",
    "\n",
    "MODELS_ = MODELS.copy()\n",
    "MODELS_[\"openai/gpt-3.5-turbo (02/05/23)\"] = partial(\n",
    "    openai_model_loader_fine_ner, model=\"gpt-3.5-turbo\"\n",
    ")\n",
    "MODELS_[\"openai/gpt-4 (02/05/23)\"] = partial(\n",
    "    openai_model_loader_fine_ner, model=\"gpt-4\"\n",
    ")\n",
    "\n",
    "# don't test openai models on DANSK\n",
    "MODELS_.pop(\"openai/gpt-3.5-turbo (02/05/23)\")\n",
    "MODELS_.pop(\"openai/gpt-4 (02/05/23)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dansk (train): Loading prediction for da_dacy_large_trf-0.2.0\n",
      "dansk (dev): Loading prediction for da_dacy_large_trf-0.2.0\n",
      "dansk (test): Loading prediction for da_dacy_large_trf-0.2.0\n",
      "dansk (train): Loading prediction for da_dacy_medium_trf-0.2.0\n",
      "dansk (dev): Loading prediction for da_dacy_medium_trf-0.2.0\n",
      "dansk (test): Loading prediction for da_dacy_medium_trf-0.2.0\n",
      "dansk (train): Loading prediction for da_dacy_small_trf-0.2.0\n",
      "dansk (dev): Loading prediction for da_dacy_small_trf-0.2.0\n",
      "dansk (test): Loading prediction for da_dacy_small_trf-0.2.0\n",
      "dansk (train): Loading prediction for da_dacy_large_ner_fine_grained-0.1.0\n",
      "dansk (dev): Loading prediction for da_dacy_large_ner_fine_grained-0.1.0\n",
      "dansk (test): Loading prediction for da_dacy_large_ner_fine_grained-0.1.0\n",
      "dansk (train): Loading prediction for da_dacy_medium_ner_fine_grained-0.1.0\n",
      "dansk (dev): Loading prediction for da_dacy_medium_ner_fine_grained-0.1.0\n",
      "dansk (test): Loading prediction for da_dacy_medium_ner_fine_grained-0.1.0\n",
      "dansk (train): Loading prediction for da_dacy_small_ner_fine_grained-0.1.0\n",
      "dansk (dev): Loading prediction for da_dacy_small_ner_fine_grained-0.1.0\n",
      "dansk (test): Loading prediction for da_dacy_small_ner_fine_grained-0.1.0\n",
      "dansk (train): Loading prediction for saattrupdan/nbailab-base-ner-scandi\n",
      "dansk (dev): Loading prediction for saattrupdan/nbailab-base-ner-scandi\n",
      "dansk (test): Loading prediction for saattrupdan/nbailab-base-ner-scandi\n",
      "dansk (train): Loading prediction for alexandrainst/da-ner-base\n",
      "dansk (dev): Loading prediction for alexandrainst/da-ner-base\n",
      "dansk (test): Loading prediction for alexandrainst/da-ner-base\n",
      "dansk (train): Loading prediction for da_core_news_trf-3.5.0\n",
      "dansk (dev): Loading prediction for da_core_news_trf-3.5.0\n",
      "dansk (test): Loading prediction for da_core_news_trf-3.5.0\n",
      "dansk (train): Loading prediction for da_core_news_lg-3.5.0\n",
      "dansk (dev): Loading prediction for da_core_news_lg-3.5.0\n",
      "dansk (test): Loading prediction for da_core_news_lg-3.5.0\n",
      "dansk (train): Loading prediction for da_core_news_md-3.5.0\n",
      "dansk (dev): Loading prediction for da_core_news_md-3.5.0\n",
      "dansk (test): Loading prediction for da_core_news_md-3.5.0\n",
      "dansk (train): Loading prediction for da_core_news_sm-3.5.0\n",
      "dansk (dev): Loading prediction for da_core_news_sm-3.5.0\n",
      "dansk (test): Loading prediction for da_core_news_sm-3.5.0\n"
     ]
    }
   ],
   "source": [
    "dansk = {}\n",
    "for mdl_name, model_getter in MODELS_.items():\n",
    "    if \"openai\" in mdl_name:\n",
    "        splits = [\"test\"]\n",
    "    else:\n",
    "        splits = [\"train\", \"dev\", \"test\"]\n",
    "    mdl_results = apply_models(mdl_name, model_getter, dataset=\"dansk\", splits=splits)\n",
    "    dansk[mdl_name] = mdl_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "with Pool(8) as p:\n",
    "    tables = p.starmap(\n",
    "        create_dataframe,\n",
    "        [\n",
    "            (dansk[mdl][\"test\"][\"examples\"], mdl, 1, 100, 2000)\n",
    "            for mdl in dansk\n",
    "            if \"fine_grained\" in mdl\n",
    "        ],\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "import altair as alt\n",
    "\n",
    "\n",
    "def create_dansk_viz(df: pd.DataFrame):\n",
    "    plot_df = df.melt(\n",
    "        id_vars=[\"Models\"],\n",
    "        var_name=\"Label\",\n",
    "        value_name=\"F1 string\",\n",
    "    )\n",
    "\n",
    "    # Convert the score value to a float\n",
    "    plot_df[\"F1\"] = plot_df[\"F1 string\"].apply(\n",
    "        lambda x: float(x.split()[0]) if not isinstance(x, float) else x\n",
    "    )\n",
    "    plot_df[\"CI Lower\"] = plot_df[\"F1 string\"].apply(\n",
    "        lambda x: float(x.split(\"(\")[1].split(\",\")[0])\n",
    "    )\n",
    "    plot_df[\"CI Upper\"] = plot_df[\"F1 string\"].apply(\n",
    "        lambda x: float(x.split(\",\")[1].split(\")\")[0])\n",
    "    )\n",
    "\n",
    "    selection = alt.selection_point(\n",
    "        fields=[\"Label\"],\n",
    "        bind=\"legend\",\n",
    "        value=[{\"Label\": \"Average\"}],\n",
    "    )\n",
    "\n",
    "    base = (\n",
    "        alt.Chart(plot_df)\n",
    "        .mark_point(filled=True, size=100)\n",
    "        .encode(\n",
    "            x=alt.X(\"F1\", title=\"F1\"),\n",
    "            y=\"Models\",\n",
    "            color=\"Label\",\n",
    "            tooltip=[\n",
    "                \"Models\",\n",
    "                \"Label\",\n",
    "                alt.Tooltip(\"F1 string\", title=\"F1\"),\n",
    "            ],\n",
    "            opacity=alt.condition(selection, alt.value(1), alt.value(0.0)),\n",
    "            # only show the tooltip when when the label is selected\n",
    "        )\n",
    "    )\n",
    "    error_bars = (\n",
    "        alt.Chart(plot_df)\n",
    "        .mark_errorbar(ticks=False)\n",
    "        .encode(\n",
    "            x=alt.X(\"CI Lower\", title=\"F1\"),\n",
    "            x2=\"CI Upper\",\n",
    "            y=\"Models\",\n",
    "            color=\"Label\",\n",
    "            opacity=alt.condition(selection, alt.value(1), alt.value(0.0)),\n",
    "        )\n",
    "    )\n",
    "\n",
    "    chart = base + error_bars\n",
    "\n",
    "    return chart.add_params(selection).properties(width=400, height=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "  #altair-viz-93ea216721f84b06bfa5848ac3e2ee19.vega-embed {\n",
       "    width: 100%;\n",
       "    display: flex;\n",
       "  }\n",
       "\n",
       "  #altair-viz-93ea216721f84b06bfa5848ac3e2ee19.vega-embed details,\n",
       "  #altair-viz-93ea216721f84b06bfa5848ac3e2ee19.vega-embed details summary {\n",
       "    position: relative;\n",
       "  }\n",
       "</style>\n",
       "<div id=\"altair-viz-93ea216721f84b06bfa5848ac3e2ee19\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-93ea216721f84b06bfa5848ac3e2ee19\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-93ea216721f84b06bfa5848ac3e2ee19\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.8.0?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function maybeLoadScript(lib, version) {\n",
       "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
       "      return (VEGA_DEBUG[key] == version) ?\n",
       "        Promise.resolve(paths[lib]) :\n",
       "        new Promise(function(resolve, reject) {\n",
       "          var s = document.createElement('script');\n",
       "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "          s.async = true;\n",
       "          s.onload = () => {\n",
       "            VEGA_DEBUG[key] = version;\n",
       "            return resolve(paths[lib]);\n",
       "          };\n",
       "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "          s.src = paths[lib];\n",
       "        });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else {\n",
       "      maybeLoadScript(\"vega\", \"5\")\n",
       "        .then(() => maybeLoadScript(\"vega-lite\", \"5.8.0\"))\n",
       "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"layer\": [{\"mark\": {\"type\": \"point\", \"filled\": true, \"size\": 100}, \"encoding\": {\"color\": {\"field\": \"Label\", \"type\": \"nominal\"}, \"opacity\": {\"condition\": {\"param\": \"param_1\", \"value\": 1}, \"value\": 0.0}, \"tooltip\": [{\"field\": \"Models\", \"type\": \"nominal\"}, {\"field\": \"Label\", \"type\": \"nominal\"}, {\"field\": \"F1 string\", \"title\": \"F1\", \"type\": \"nominal\"}], \"x\": {\"field\": \"F1\", \"title\": \"F1\", \"type\": \"quantitative\"}, \"y\": {\"field\": \"Models\", \"type\": \"nominal\"}}, \"name\": \"view_1\"}, {\"mark\": {\"type\": \"errorbar\", \"ticks\": false}, \"encoding\": {\"color\": {\"field\": \"Label\", \"type\": \"nominal\"}, \"opacity\": {\"condition\": {\"param\": \"param_1\", \"value\": 1}, \"value\": 0.0}, \"x\": {\"field\": \"CI Lower\", \"title\": \"F1\", \"type\": \"quantitative\"}, \"x2\": {\"field\": \"CI Upper\"}, \"y\": {\"field\": \"Models\", \"type\": \"nominal\"}}}], \"data\": {\"name\": \"data-db2e689a8dbaa1f239c072df7e662711\"}, \"height\": 300, \"params\": [{\"name\": \"param_1\", \"select\": {\"type\": \"point\", \"fields\": [\"Label\"]}, \"bind\": \"legend\", \"value\": [{\"Label\": \"Average\"}], \"views\": [\"view_1\"]}], \"width\": 400, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.8.0.json\", \"datasets\": {\"data-db2e689a8dbaa1f239c072df7e662711\": [{\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Average\", \"F1 string\": \"80.1 (78.2, 81.9)\", \"F1\": 80.1, \"CI Lower\": 78.2, \"CI Upper\": 81.9}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Average\", \"F1 string\": \"79.7 (77.7, 81.5)\", \"F1\": 79.7, \"CI Lower\": 77.7, \"CI Upper\": 81.5}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Average\", \"F1 string\": \"78.4 (76.3, 80.4)\", \"F1\": 78.4, \"CI Lower\": 76.3, \"CI Upper\": 80.4}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Language\", \"F1 string\": \"74.5 (60.0, 83.3)\", \"F1\": 74.5, \"CI Lower\": 60.0, \"CI Upper\": 83.3}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Language\", \"F1 string\": \"51.9 (23.3, 100.0)\", \"F1\": 51.9, \"CI Lower\": 23.3, \"CI Upper\": 100.0}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Language\", \"F1 string\": \"45.9 (13.3, 93.3)\", \"F1\": 45.9, \"CI Lower\": 13.3, \"CI Upper\": 93.3}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Organization\", \"F1 string\": \"79.5 (74.9, 83.1)\", \"F1\": 79.5, \"CI Lower\": 74.9, \"CI Upper\": 83.1}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Organization\", \"F1 string\": \"80.5 (78.1, 84.2)\", \"F1\": 80.5, \"CI Lower\": 78.1, \"CI Upper\": 84.2}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Organization\", \"F1 string\": \"79.1 (75.7, 82.3)\", \"F1\": 79.1, \"CI Lower\": 75.7, \"CI Upper\": 82.3}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Percent\", \"F1 string\": \"100.0 (100.0, 100.0)\", \"F1\": 100.0, \"CI Lower\": 100.0, \"CI Upper\": 100.0}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Percent\", \"F1 string\": \"100.0 (100.0, 100.0)\", \"F1\": 100.0, \"CI Lower\": 100.0, \"CI Upper\": 100.0}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Percent\", \"F1 string\": \"100.0 (100.0, 100.0)\", \"F1\": 100.0, \"CI Lower\": 100.0, \"CI Upper\": 100.0}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Law\", \"F1 string\": \"54.2 (38.1, 72.5)\", \"F1\": 54.2, \"CI Lower\": 38.1, \"CI Upper\": 72.5}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Law\", \"F1 string\": \"59.3 (37.4, 77.3)\", \"F1\": 59.3, \"CI Lower\": 37.4, \"CI Upper\": 77.3}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Law\", \"F1 string\": \"57.6 (39.6, 75.1)\", \"F1\": 57.6, \"CI Lower\": 39.6, \"CI Upper\": 75.1}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"GPE\", \"F1 string\": \"90.6 (87.2, 93.1)\", \"F1\": 90.6, \"CI Lower\": 87.2, \"CI Upper\": 93.1}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"GPE\", \"F1 string\": \"88.0 (82.7, 92.1)\", \"F1\": 88.0, \"CI Lower\": 82.7, \"CI Upper\": 92.1}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"GPE\", \"F1 string\": \"79.6 (73.0, 84.6)\", \"F1\": 79.6, \"CI Lower\": 73.0, \"CI Upper\": 84.6}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Facility\", \"F1 string\": \"69.8 (54.3, 84.4)\", \"F1\": 69.8, \"CI Lower\": 54.3, \"CI Upper\": 84.4}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Facility\", \"F1 string\": \"72.3 (56.2, 84.6)\", \"F1\": 72.3, \"CI Lower\": 56.2, \"CI Upper\": 84.6}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Facility\", \"F1 string\": \"55.5 (36.2, 70.5)\", \"F1\": 55.5, \"CI Lower\": 36.2, \"CI Upper\": 70.5}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Product\", \"F1 string\": \"62.4 (53.9, 72.0)\", \"F1\": 62.4, \"CI Lower\": 53.9, \"CI Upper\": 72.0}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Product\", \"F1 string\": \"62.6 (53.9, 71.6)\", \"F1\": 62.6, \"CI Lower\": 53.9, \"CI Upper\": 71.6}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Product\", \"F1 string\": \"59.5 (48.9, 67.9)\", \"F1\": 59.5, \"CI Lower\": 48.9, \"CI Upper\": 67.9}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Ordinal\", \"F1 string\": \"37.8 (22.5, 51.2)\", \"F1\": 37.8, \"CI Lower\": 22.5, \"CI Upper\": 51.2}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Ordinal\", \"F1 string\": \"68.7 (49.1, 82.6)\", \"F1\": 68.7, \"CI Lower\": 49.1, \"CI Upper\": 82.6}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Ordinal\", \"F1 string\": \"68.5 (47.6, 83.1)\", \"F1\": 68.5, \"CI Lower\": 47.6, \"CI Upper\": 83.1}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Time\", \"F1 string\": \"90.9 (83.8, 96.7)\", \"F1\": 90.9, \"CI Lower\": 83.8, \"CI Upper\": 96.7}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Time\", \"F1 string\": \"85.1 (74.0, 93.7)\", \"F1\": 85.1, \"CI Lower\": 74.0, \"CI Upper\": 93.7}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Time\", \"F1 string\": \"83.4 (68.0, 95.6)\", \"F1\": 83.4, \"CI Lower\": 68.0, \"CI Upper\": 95.6}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Money\", \"F1 string\": \"99.3 (97.9, 100.0)\", \"F1\": 99.3, \"CI Lower\": 97.9, \"CI Upper\": 100.0}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Money\", \"F1 string\": \"98.6 (97.2, 100.0)\", \"F1\": 98.6, \"CI Lower\": 97.2, \"CI Upper\": 100.0}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Money\", \"F1 string\": \"95.2 (90.0, 98.2)\", \"F1\": 95.2, \"CI Lower\": 90.0, \"CI Upper\": 98.2}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Event\", \"F1 string\": \"43.5 (27.0, 56.0)\", \"F1\": 43.5, \"CI Lower\": 27.0, \"CI Upper\": 56.0}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Event\", \"F1 string\": \"64.2 (50.0, 79.4)\", \"F1\": 64.2, \"CI Lower\": 50.0, \"CI Upper\": 79.4}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Event\", \"F1 string\": \"46.1 (27.8, 62.4)\", \"F1\": 46.1, \"CI Lower\": 27.8, \"CI Upper\": 62.4}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Person\", \"F1 string\": \"85.9 (82.7, 88.8)\", \"F1\": 85.9, \"CI Lower\": 82.7, \"CI Upper\": 88.8}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Person\", \"F1 string\": \"84.8 (80.6, 88.2)\", \"F1\": 84.8, \"CI Lower\": 80.6, \"CI Upper\": 88.2}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Person\", \"F1 string\": \"86.8 (83.2, 90.1)\", \"F1\": 86.8, \"CI Lower\": 83.2, \"CI Upper\": 90.1}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Work of Art\", \"F1 string\": \"39.3 (25.5, 50.3)\", \"F1\": 39.3, \"CI Lower\": 25.5, \"CI Upper\": 50.3}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Work of Art\", \"F1 string\": \"58.4 (48.7, 69.1)\", \"F1\": 58.4, \"CI Lower\": 48.7, \"CI Upper\": 69.1}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Work of Art\", \"F1 string\": \"46.6 (36.2, 56.9)\", \"F1\": 46.6, \"CI Lower\": 36.2, \"CI Upper\": 56.9}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Location\", \"F1 string\": \"75.3 (66.9, 83.8)\", \"F1\": 75.3, \"CI Lower\": 66.9, \"CI Upper\": 83.8}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Location\", \"F1 string\": \"72.5 (62.1, 80.8)\", \"F1\": 72.5, \"CI Lower\": 62.1, \"CI Upper\": 80.8}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Location\", \"F1 string\": \"65.6 (55.4, 74.1)\", \"F1\": 65.6, \"CI Lower\": 55.4, \"CI Upper\": 74.1}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"NORP\", \"F1 string\": \"84.8 (76.9, 90.8)\", \"F1\": 84.8, \"CI Lower\": 76.9, \"CI Upper\": 90.8}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"NORP\", \"F1 string\": \"78.2 (68.6, 85.8)\", \"F1\": 78.2, \"CI Lower\": 68.6, \"CI Upper\": 85.8}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"NORP\", \"F1 string\": \"73.3 (62.9, 81.5)\", \"F1\": 73.3, \"CI Lower\": 62.9, \"CI Upper\": 81.5}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Cardinal\", \"F1 string\": \"87.0 (82.8, 90.3)\", \"F1\": 87.0, \"CI Lower\": 82.8, \"CI Upper\": 90.3}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Cardinal\", \"F1 string\": \"80.5 (77.0, 84.4)\", \"F1\": 80.5, \"CI Lower\": 77.0, \"CI Upper\": 84.4}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Cardinal\", \"F1 string\": \"89.2 (86.0, 91.7)\", \"F1\": 89.2, \"CI Lower\": 86.0, \"CI Upper\": 91.7}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Quantity\", \"F1 string\": \"78.6 (59.8, 93.8)\", \"F1\": 78.6, \"CI Lower\": 59.8, \"CI Upper\": 93.8}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Quantity\", \"F1 string\": \"76.9 (63.9, 89.9)\", \"F1\": 76.9, \"CI Lower\": 63.9, \"CI Upper\": 89.9}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Quantity\", \"F1 string\": \"71.3 (50.0, 91.1)\", \"F1\": 71.3, \"CI Lower\": 50.0, \"CI Upper\": 91.1}, {\"Models\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Label\": \"Date\", \"F1 string\": \"77.3 (71.6, 81.8)\", \"F1\": 77.3, \"CI Lower\": 71.6, \"CI Upper\": 81.8}, {\"Models\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Label\": \"Date\", \"F1 string\": \"77.6 (72.8, 82.2)\", \"F1\": 77.6, \"CI Lower\": 72.8, \"CI Upper\": 82.2}, {\"Models\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Label\": \"Date\", \"F1 string\": \"78.8 (73.9, 83.4)\", \"F1\": 78.8, \"CI Lower\": 73.9, \"CI Upper\": 83.4}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.LayerChart(...)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dansk_df = pd.concat(tables)\n",
    "create_dansk_viz(dansk_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "_df = dansk_df\n",
    "_df = _df.set_index(\"Models\")\n",
    "ent_columns = sorted(\n",
    "    [\n",
    "        \"Event\",\n",
    "        \"Organization\",\n",
    "        \"Language\",\n",
    "        \"Person\",\n",
    "        \"Ordinal\",\n",
    "        \"NORP\",\n",
    "        \"Work of Art\",\n",
    "        \"Facility\",\n",
    "        \"Law\",\n",
    "        \"Location\",\n",
    "        \"Product\",\n",
    "        \"GPE\",\n",
    "    ]\n",
    ")\n",
    "non_ent_columns = sorted([\"Cardinal\", \"Date\", \"Money\", \"Percent\", \"Quantity\", \"Time\"])\n",
    "columns_to_keep = ent_columns + non_ent_columns + [\"Average\"]\n",
    "\n",
    "_df = _df[columns_to_keep]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "table = _df.T\n",
    "iidx = pd.MultiIndex.from_arrays(\n",
    "    [\n",
    "        [\"Entities\"] * len(ent_columns)\n",
    "        + [\"Non-Entities\"] * len(non_ent_columns)\n",
    "        + [\"Average\"],\n",
    "        ent_columns + non_ent_columns + [\"Average\"],\n",
    "    ]\n",
    ")\n",
    "table.index = iidx\n",
    "\n",
    "mdl_names = [\"Large 0.1.0\", \"Medium 0.1.0\", \"Small 0.1.0\"]\n",
    "header = pd.MultiIndex.from_arrays(\n",
    "    [[\"Fine-grained Models\"] * len(mdl_names), mdl_names]\n",
    ")\n",
    "table.columns = header"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_edc40 .level0 {\n",
       "  text-align: center;\n",
       "}\n",
       "#T_edc40 .col_heading {\n",
       "  text-align: center;\n",
       "}\n",
       "#T_edc40_row0_col1, #T_edc40_row1_col1, #T_edc40_row2_col0, #T_edc40_row3_col0, #T_edc40_row4_col1, #T_edc40_row5_col0, #T_edc40_row6_col0, #T_edc40_row7_col1, #T_edc40_row8_col1, #T_edc40_row9_col2, #T_edc40_row10_col1, #T_edc40_row11_col1, #T_edc40_row12_col2, #T_edc40_row13_col2, #T_edc40_row14_col0, #T_edc40_row16_col0, #T_edc40_row17_col0, #T_edc40_row18_col0 {\n",
       "  font-weight: bold;\n",
       "}\n",
       "#T_edc40_row0_col2, #T_edc40_row1_col0, #T_edc40_row2_col1, #T_edc40_row3_col1, #T_edc40_row4_col2, #T_edc40_row5_col1, #T_edc40_row6_col1, #T_edc40_row7_col2, #T_edc40_row8_col0, #T_edc40_row9_col0, #T_edc40_row10_col0, #T_edc40_row11_col2, #T_edc40_row12_col0, #T_edc40_row13_col1, #T_edc40_row14_col1, #T_edc40_row16_col1, #T_edc40_row17_col1, #T_edc40_row18_col1 {\n",
       "  font-style: italic;\n",
       "}\n",
       "#T_edc40_row15_col0, #T_edc40_row15_col1, #T_edc40_row15_col2 {\n",
       "  font-weight: bold;\n",
       "  font-style: normal;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_edc40\" style=\"font-size: 0.8em\">\n",
       "  <caption>F1 score with 95% confidence interval calculated using bootstrapping with 100 samples.</caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank\" >&nbsp;</th>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_edc40_level0_col0\" class=\"col_heading level0 col0\" colspan=\"3\">Fine-grained Models</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"blank\" >&nbsp;</th>\n",
       "      <th class=\"blank level1\" >&nbsp;</th>\n",
       "      <th id=\"T_edc40_level1_col0\" class=\"col_heading level1 col0\" >Large 0.1.0</th>\n",
       "      <th id=\"T_edc40_level1_col1\" class=\"col_heading level1 col1\" >Medium 0.1.0</th>\n",
       "      <th id=\"T_edc40_level1_col2\" class=\"col_heading level1 col2\" >Small 0.1.0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level0_row0\" class=\"row_heading level0 row0\" rowspan=\"12\">Entities</th>\n",
       "      <th id=\"T_edc40_level1_row0\" class=\"row_heading level1 row0\" >Event</th>\n",
       "      <td id=\"T_edc40_row0_col0\" class=\"data row0 col0\" >43.5 (27.0, 56.0)</td>\n",
       "      <td id=\"T_edc40_row0_col1\" class=\"data row0 col1\" >64.2 (50.0, 79.4)</td>\n",
       "      <td id=\"T_edc40_row0_col2\" class=\"data row0 col2\" >46.1 (27.8, 62.4)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row1\" class=\"row_heading level1 row1\" >Facility</th>\n",
       "      <td id=\"T_edc40_row1_col0\" class=\"data row1 col0\" >69.8 (54.3, 84.4)</td>\n",
       "      <td id=\"T_edc40_row1_col1\" class=\"data row1 col1\" >72.3 (56.2, 84.6)</td>\n",
       "      <td id=\"T_edc40_row1_col2\" class=\"data row1 col2\" >55.5 (36.2, 70.5)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row2\" class=\"row_heading level1 row2\" >GPE</th>\n",
       "      <td id=\"T_edc40_row2_col0\" class=\"data row2 col0\" >90.6 (87.2, 93.1)</td>\n",
       "      <td id=\"T_edc40_row2_col1\" class=\"data row2 col1\" >88.0 (82.7, 92.1)</td>\n",
       "      <td id=\"T_edc40_row2_col2\" class=\"data row2 col2\" >79.6 (73.0, 84.6)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row3\" class=\"row_heading level1 row3\" >Language</th>\n",
       "      <td id=\"T_edc40_row3_col0\" class=\"data row3 col0\" >74.5 (60.0, 83.3)</td>\n",
       "      <td id=\"T_edc40_row3_col1\" class=\"data row3 col1\" >51.9 (23.3, 100.0)</td>\n",
       "      <td id=\"T_edc40_row3_col2\" class=\"data row3 col2\" >45.9 (13.3, 93.3)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row4\" class=\"row_heading level1 row4\" >Law</th>\n",
       "      <td id=\"T_edc40_row4_col0\" class=\"data row4 col0\" >54.2 (38.1, 72.5)</td>\n",
       "      <td id=\"T_edc40_row4_col1\" class=\"data row4 col1\" >59.3 (37.4, 77.3)</td>\n",
       "      <td id=\"T_edc40_row4_col2\" class=\"data row4 col2\" >57.6 (39.6, 75.1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row5\" class=\"row_heading level1 row5\" >Location</th>\n",
       "      <td id=\"T_edc40_row5_col0\" class=\"data row5 col0\" >75.3 (66.9, 83.8)</td>\n",
       "      <td id=\"T_edc40_row5_col1\" class=\"data row5 col1\" >72.5 (62.1, 80.8)</td>\n",
       "      <td id=\"T_edc40_row5_col2\" class=\"data row5 col2\" >65.6 (55.4, 74.1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row6\" class=\"row_heading level1 row6\" >NORP</th>\n",
       "      <td id=\"T_edc40_row6_col0\" class=\"data row6 col0\" >84.8 (76.9, 90.8)</td>\n",
       "      <td id=\"T_edc40_row6_col1\" class=\"data row6 col1\" >78.2 (68.6, 85.8)</td>\n",
       "      <td id=\"T_edc40_row6_col2\" class=\"data row6 col2\" >73.3 (62.9, 81.5)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row7\" class=\"row_heading level1 row7\" >Ordinal</th>\n",
       "      <td id=\"T_edc40_row7_col0\" class=\"data row7 col0\" >37.8 (22.5, 51.2)</td>\n",
       "      <td id=\"T_edc40_row7_col1\" class=\"data row7 col1\" >68.7 (49.1, 82.6)</td>\n",
       "      <td id=\"T_edc40_row7_col2\" class=\"data row7 col2\" >68.5 (47.6, 83.1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row8\" class=\"row_heading level1 row8\" >Organization</th>\n",
       "      <td id=\"T_edc40_row8_col0\" class=\"data row8 col0\" >79.5 (74.9, 83.1)</td>\n",
       "      <td id=\"T_edc40_row8_col1\" class=\"data row8 col1\" >80.5 (78.1, 84.2)</td>\n",
       "      <td id=\"T_edc40_row8_col2\" class=\"data row8 col2\" >79.1 (75.7, 82.3)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row9\" class=\"row_heading level1 row9\" >Person</th>\n",
       "      <td id=\"T_edc40_row9_col0\" class=\"data row9 col0\" >85.9 (82.7, 88.8)</td>\n",
       "      <td id=\"T_edc40_row9_col1\" class=\"data row9 col1\" >84.8 (80.6, 88.2)</td>\n",
       "      <td id=\"T_edc40_row9_col2\" class=\"data row9 col2\" >86.8 (83.2, 90.1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row10\" class=\"row_heading level1 row10\" >Product</th>\n",
       "      <td id=\"T_edc40_row10_col0\" class=\"data row10 col0\" >62.4 (53.9, 72.0)</td>\n",
       "      <td id=\"T_edc40_row10_col1\" class=\"data row10 col1\" >62.6 (53.9, 71.6)</td>\n",
       "      <td id=\"T_edc40_row10_col2\" class=\"data row10 col2\" >59.5 (48.9, 67.9)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row11\" class=\"row_heading level1 row11\" >Work of Art</th>\n",
       "      <td id=\"T_edc40_row11_col0\" class=\"data row11 col0\" >39.3 (25.5, 50.3)</td>\n",
       "      <td id=\"T_edc40_row11_col1\" class=\"data row11 col1\" >58.4 (48.7, 69.1)</td>\n",
       "      <td id=\"T_edc40_row11_col2\" class=\"data row11 col2\" >46.6 (36.2, 56.9)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level0_row12\" class=\"row_heading level0 row12\" rowspan=\"6\">Non-Entities</th>\n",
       "      <th id=\"T_edc40_level1_row12\" class=\"row_heading level1 row12\" >Cardinal</th>\n",
       "      <td id=\"T_edc40_row12_col0\" class=\"data row12 col0\" >87.0 (82.8, 90.3)</td>\n",
       "      <td id=\"T_edc40_row12_col1\" class=\"data row12 col1\" >80.5 (77.0, 84.4)</td>\n",
       "      <td id=\"T_edc40_row12_col2\" class=\"data row12 col2\" >89.2 (86.0, 91.7)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row13\" class=\"row_heading level1 row13\" >Date</th>\n",
       "      <td id=\"T_edc40_row13_col0\" class=\"data row13 col0\" >77.3 (71.6, 81.8)</td>\n",
       "      <td id=\"T_edc40_row13_col1\" class=\"data row13 col1\" >77.6 (72.8, 82.2)</td>\n",
       "      <td id=\"T_edc40_row13_col2\" class=\"data row13 col2\" >78.8 (73.9, 83.4)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row14\" class=\"row_heading level1 row14\" >Money</th>\n",
       "      <td id=\"T_edc40_row14_col0\" class=\"data row14 col0\" >99.3 (97.9, 100.0)</td>\n",
       "      <td id=\"T_edc40_row14_col1\" class=\"data row14 col1\" >98.6 (97.2, 100.0)</td>\n",
       "      <td id=\"T_edc40_row14_col2\" class=\"data row14 col2\" >95.2 (90.0, 98.2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row15\" class=\"row_heading level1 row15\" >Percent</th>\n",
       "      <td id=\"T_edc40_row15_col0\" class=\"data row15 col0\" >100.0 (100.0, 100.0)</td>\n",
       "      <td id=\"T_edc40_row15_col1\" class=\"data row15 col1\" >100.0 (100.0, 100.0)</td>\n",
       "      <td id=\"T_edc40_row15_col2\" class=\"data row15 col2\" >100.0 (100.0, 100.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row16\" class=\"row_heading level1 row16\" >Quantity</th>\n",
       "      <td id=\"T_edc40_row16_col0\" class=\"data row16 col0\" >78.6 (59.8, 93.8)</td>\n",
       "      <td id=\"T_edc40_row16_col1\" class=\"data row16 col1\" >76.9 (63.9, 89.9)</td>\n",
       "      <td id=\"T_edc40_row16_col2\" class=\"data row16 col2\" >71.3 (50.0, 91.1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level1_row17\" class=\"row_heading level1 row17\" >Time</th>\n",
       "      <td id=\"T_edc40_row17_col0\" class=\"data row17 col0\" >90.9 (83.8, 96.7)</td>\n",
       "      <td id=\"T_edc40_row17_col1\" class=\"data row17 col1\" >85.1 (74.0, 93.7)</td>\n",
       "      <td id=\"T_edc40_row17_col2\" class=\"data row17 col2\" >83.4 (68.0, 95.6)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_edc40_level0_row18\" class=\"row_heading level0 row18\" >Average</th>\n",
       "      <th id=\"T_edc40_level1_row18\" class=\"row_heading level1 row18\" >Average</th>\n",
       "      <td id=\"T_edc40_row18_col0\" class=\"data row18 col0\" >80.1 (78.2, 81.9)</td>\n",
       "      <td id=\"T_edc40_row18_col1\" class=\"data row18 col1\" >79.7 (77.7, 81.5)</td>\n",
       "      <td id=\"T_edc40_row18_col2\" class=\"data row18 col2\" >78.4 (76.3, 80.4)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x3ceec3f10>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert to latex using styler\n",
    "style = table.style.format_index(escape=\"latex\", axis=1).format_index(\n",
    "    escape=\"latex\", axis=0\n",
    ")\n",
    "\n",
    "\n",
    "# highlight the maximum\n",
    "def italicize_second_max(s: pd.Series) -> list:\n",
    "    \"\"\"Italicize the second maximum in a Series.\"\"\"\n",
    "    is_second_max = s == s.sort_values(ascending=False).iloc[1]\n",
    "    # check if the second maximum is the same as the maximum\n",
    "    same_as_max = s == s.max()\n",
    "\n",
    "    if same_as_max.sum() > 1:\n",
    "        # if there are more than one maximum, don't italicize\n",
    "        return [\"font-style: normal\" for v in is_second_max]\n",
    "    return [\"font-style: italic\" if v else \"\" for v in is_second_max]\n",
    "\n",
    "\n",
    "style = style.apply(highlight_max, axis=1)\n",
    "# style = style.apply(underline_second_max, axis=1)\n",
    "style = style.apply(italicize_second_max, axis=1)\n",
    "\n",
    "# apply the CSS style\n",
    "super_header_style = [\n",
    "    {\"selector\": \".level0\", \"props\": [(\"text-align\", \"center\")]},\n",
    "    {\"selector\": \".col_heading\", \"props\": [(\"text-align\", \"center\")]},\n",
    "]\n",
    "style = style.set_table_styles(super_header_style)  # type: ignore\n",
    "\n",
    "\n",
    "# add caption\n",
    "caption = \"F1 score with 95% confidence interval calculated using bootstrapping with 100 samples.\"\n",
    "style = style.set_caption(caption)\n",
    "# font size\n",
    "style = style.set_table_attributes('style=\"font-size: 0.8em\"')\n",
    "style\n",
    "\n",
    "# latex = style.to_latex(\n",
    "#         hrules=True,\n",
    "#         convert_css=True,\n",
    "#     )\n",
    "\n",
    "# print(latex)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Domain Generalization\n",
    "For the domains generalization benchmark we utilize the [DANSK](https://huggingface.co/datasets/chcaa/DANSK) dataset. This dataset is annotated across many different domains including fiction, web content, social media, wikis, news, legal and conversational data.\n",
    "As some models are trained on DANSK (`da_dacy_{size}_ner_fine_grained-{version}`) these models are tested on the test set using all of the\n",
    "labels. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "from evaluation.utils import evaluate_generalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "tables = []\n",
    "for mdl_name in dansk:\n",
    "    if \"fine_grained\" not in mdl_name:\n",
    "        continue\n",
    "\n",
    "    table = evaluate_generalization(\n",
    "        examples=dansk[mdl_name][\"test\"][\"examples\"],\n",
    "        mdl_name=mdl_name,\n",
    "        n_rep=100,\n",
    "        n_samples=1000,\n",
    "    )\n",
    "    tables.append(table)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "df = pd.concat(tables)\n",
    "df = df[df[\"Domain\"] != \"dannet\"]\n",
    "df = df[df[\"Domain\"].notnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "  #altair-viz-fe456292bec24900bf9b81c521c84de2.vega-embed {\n",
       "    width: 100%;\n",
       "    display: flex;\n",
       "  }\n",
       "\n",
       "  #altair-viz-fe456292bec24900bf9b81c521c84de2.vega-embed details,\n",
       "  #altair-viz-fe456292bec24900bf9b81c521c84de2.vega-embed details summary {\n",
       "    position: relative;\n",
       "  }\n",
       "</style>\n",
       "<div id=\"altair-viz-fe456292bec24900bf9b81c521c84de2\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-fe456292bec24900bf9b81c521c84de2\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-fe456292bec24900bf9b81c521c84de2\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.8.0?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function maybeLoadScript(lib, version) {\n",
       "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
       "      return (VEGA_DEBUG[key] == version) ?\n",
       "        Promise.resolve(paths[lib]) :\n",
       "        new Promise(function(resolve, reject) {\n",
       "          var s = document.createElement('script');\n",
       "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "          s.async = true;\n",
       "          s.onload = () => {\n",
       "            VEGA_DEBUG[key] = version;\n",
       "            return resolve(paths[lib]);\n",
       "          };\n",
       "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "          s.src = paths[lib];\n",
       "        });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else {\n",
       "      maybeLoadScript(\"vega\", \"5\")\n",
       "        .then(() => maybeLoadScript(\"vega-lite\", \"5.8.0\"))\n",
       "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"layer\": [{\"mark\": {\"type\": \"point\", \"filled\": true}, \"encoding\": {\"color\": {\"field\": \"Domain\", \"type\": \"nominal\"}, \"opacity\": {\"condition\": {\"param\": \"param_2\", \"value\": 1}, \"value\": 0.0}, \"size\": {\"condition\": {\"param\": \"param_3\", \"field\": \"Number of docs\", \"legend\": null}, \"value\": 100}, \"tooltip\": [{\"field\": \"Model\", \"type\": \"nominal\"}, {\"field\": \"Domain\", \"type\": \"nominal\"}, {\"field\": \"Average F1\", \"type\": \"nominal\"}], \"x\": {\"field\": \"Average\", \"scale\": {\"domain\": [0.0, 1.0]}, \"title\": \"F1\", \"type\": \"quantitative\"}, \"y\": {\"field\": \"Model\", \"sort\": [\"da_dacy_large_trf-0.2.0\", \"da_dacy_medium_trf-0.2.0\", \"da_dacy_small_trf-0.2.0\", \"da_dacy_large_ner_fine_grained-0.1.0\", \"da_dacy_medium_ner_fine_grained-0.1.0\", \"da_dacy_small_ner_fine_grained-0.1.0\", \"saattrupdan/nbailab-base-ner-scandi\", \"alexandrainst/da-ner-base\", \"da_core_news_trf-3.5.0\", \"da_core_news_lg-3.5.0\", \"da_core_news_md-3.5.0\", \"da_core_news_sm-3.5.0\"], \"type\": \"nominal\"}}, \"name\": \"view_2\"}, {\"mark\": {\"type\": \"errorbar\", \"ticks\": false}, \"encoding\": {\"color\": {\"field\": \"Domain\", \"type\": \"nominal\"}, \"opacity\": {\"condition\": {\"param\": \"param_2\", \"value\": 1}, \"value\": 0.0}, \"x\": {\"field\": \"Average Lower CI\", \"title\": \"F1\", \"type\": \"quantitative\"}, \"x2\": {\"field\": \"Average Upper CI\"}, \"y\": {\"field\": \"Model\", \"sort\": [\"da_dacy_large_trf-0.2.0\", \"da_dacy_medium_trf-0.2.0\", \"da_dacy_small_trf-0.2.0\", \"da_dacy_large_ner_fine_grained-0.1.0\", \"da_dacy_medium_ner_fine_grained-0.1.0\", \"da_dacy_small_ner_fine_grained-0.1.0\", \"saattrupdan/nbailab-base-ner-scandi\", \"alexandrainst/da-ner-base\", \"da_core_news_trf-3.5.0\", \"da_core_news_lg-3.5.0\", \"da_core_news_md-3.5.0\", \"da_core_news_sm-3.5.0\"], \"type\": \"nominal\"}}}], \"data\": {\"name\": \"data-f41e944fb77a7a56152b17a7560d7b9e\"}, \"height\": 300, \"params\": [{\"name\": \"param_2\", \"select\": {\"type\": \"point\", \"fields\": [\"Domain\"]}, \"bind\": \"legend\", \"value\": [{\"Domain\": \"All\"}], \"views\": [\"view_2\"]}, {\"name\": \"param_3\", \"bind\": {\"input\": \"checkbox\", \"name\": \"Scale point size by number of documents: \"}}], \"title\": \"DANSK test set performance\", \"width\": 400, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.8.0.json\", \"datasets\": {\"data-f41e944fb77a7a56152b17a7560d7b9e\": [{\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"Web\", \"Average\": 0.8101067602127482, \"Average Lower CI\": 0.7841651487553125, \"Average Upper CI\": 0.830900900900901, \"Average F1\": \"0.81 (0.78, 0.83)\", \"Number of docs\": 783}, {\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"Legal\", \"Average\": 0.8663320350050492, \"Average Lower CI\": 0.8419256402801457, \"Average Upper CI\": 0.8887819496820972, \"Average F1\": \"0.87 (0.84, 0.89)\", \"Number of docs\": 239}, {\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"Conversation\", \"Average\": 0.7641179934708223, \"Average Lower CI\": 0.7337162907078971, \"Average Upper CI\": 0.7902515430981282, \"Average F1\": \"0.76 (0.73, 0.79)\", \"Number of docs\": 168}, {\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"Wiki & Books\", \"Average\": 0.7220179014224088, \"Average Lower CI\": 0.6878643801132788, \"Average Upper CI\": 0.762446517440944, \"Average F1\": \"0.72 (0.69, 0.76)\", \"Number of docs\": 182}, {\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"News\", \"Average\": 0.823329124708956, \"Average Lower CI\": 0.7975750707209395, \"Average Upper CI\": 0.8467087702231627, \"Average F1\": \"0.82 (0.80, 0.85)\", \"Number of docs\": 39}, {\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"Social Media\", \"Average\": 0.5842981151577813, \"Average Lower CI\": 0.5411082722470623, \"Average Upper CI\": 0.6276502609352638, \"Average F1\": \"0.58 (0.54, 0.63)\", \"Number of docs\": 64}, {\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"All\", \"Average\": 0.8012382615583585, \"Average Lower CI\": 0.7773314556164734, \"Average Upper CI\": 0.8290060904096623, \"Average F1\": \"0.80 (0.78, 0.83)\", \"Number of docs\": 1500}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"Web\", \"Average\": 0.8069059577346168, \"Average Lower CI\": 0.7792190204698447, \"Average Upper CI\": 0.8320044217003898, \"Average F1\": \"0.81 (0.78, 0.83)\", \"Number of docs\": 783}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"Legal\", \"Average\": 0.8792798287165162, \"Average Lower CI\": 0.860279516303508, \"Average Upper CI\": 0.8961524325753569, \"Average F1\": \"0.88 (0.86, 0.90)\", \"Number of docs\": 239}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"Conversation\", \"Average\": 0.6636923028744094, \"Average Lower CI\": 0.6349232630707509, \"Average Upper CI\": 0.6883207901715027, \"Average F1\": \"0.66 (0.63, 0.69)\", \"Number of docs\": 168}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"Wiki & Books\", \"Average\": 0.7937488810814491, \"Average Lower CI\": 0.7653571428571427, \"Average Upper CI\": 0.8218744746115128, \"Average F1\": \"0.79 (0.77, 0.82)\", \"Number of docs\": 182}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"News\", \"Average\": 0.6966281148220598, \"Average Lower CI\": 0.6733559377459511, \"Average Upper CI\": 0.7196226874856663, \"Average F1\": \"0.70 (0.67, 0.72)\", \"Number of docs\": 39}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"Social Media\", \"Average\": 0.6944594484094899, \"Average Lower CI\": 0.6478237984608682, \"Average Upper CI\": 0.7331416617609958, \"Average F1\": \"0.69 (0.65, 0.73)\", \"Number of docs\": 64}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"All\", \"Average\": 0.7954568921751537, \"Average Lower CI\": 0.7682324162204714, \"Average Upper CI\": 0.825265478923979, \"Average F1\": \"0.80 (0.77, 0.83)\", \"Number of docs\": 1500}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"Web\", \"Average\": 0.7781528937111613, \"Average Lower CI\": 0.7453085067542898, \"Average Upper CI\": 0.7999058473736373, \"Average F1\": \"0.78 (0.75, 0.80)\", \"Number of docs\": 783}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"Legal\", \"Average\": 0.9090202269126079, \"Average Lower CI\": 0.8905576809510362, \"Average Upper CI\": 0.9278187376371155, \"Average F1\": \"0.91 (0.89, 0.93)\", \"Number of docs\": 239}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"Conversation\", \"Average\": 0.7369381979372508, \"Average Lower CI\": 0.7024462244622447, \"Average Upper CI\": 0.7668525088402879, \"Average F1\": \"0.74 (0.70, 0.77)\", \"Number of docs\": 168}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"Wiki & Books\", \"Average\": 0.7414870979112445, \"Average Lower CI\": 0.711301648812756, \"Average Upper CI\": 0.774739456390258, \"Average F1\": \"0.74 (0.71, 0.77)\", \"Number of docs\": 182}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"News\", \"Average\": 0.7978561822955089, \"Average Lower CI\": 0.7712527545212747, \"Average Upper CI\": 0.8215780291306777, \"Average F1\": \"0.80 (0.77, 0.82)\", \"Number of docs\": 39}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"Social Media\", \"Average\": 0.707274819499705, \"Average Lower CI\": 0.6689111635220126, \"Average Upper CI\": 0.7518443992428864, \"Average F1\": \"0.71 (0.67, 0.75)\", \"Number of docs\": 64}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"All\", \"Average\": 0.7842946502544421, \"Average Lower CI\": 0.7563924902895867, \"Average Upper CI\": 0.8127158179241442, \"Average F1\": \"0.78 (0.76, 0.81)\", \"Number of docs\": 1500}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.LayerChart(...)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create altair viz\n",
    "selection = alt.selection_point(\n",
    "    fields=[\"Domain\"],\n",
    "    bind=\"legend\",\n",
    "    value=[{\"Domain\": \"All\"}],\n",
    ")\n",
    "bind_checkbox = alt.binding_checkbox(\n",
    "    name=\"Scale point size by number of documents: \",\n",
    ")\n",
    "param_checkbox = alt.param(bind=bind_checkbox)\n",
    "\n",
    "sort_order = list(dansk.keys())\n",
    "\n",
    "base = (\n",
    "    alt.Chart(df)\n",
    "    .mark_point(filled=True)\n",
    "    .encode(\n",
    "        x=alt.X(\"Average\", title=\"F1\", scale=alt.Scale(domain=[0.0, 1.0])),\n",
    "        y=alt.Y(\"Model\", sort=sort_order),\n",
    "        color=\"Domain\",\n",
    "        size=alt.condition(\n",
    "            param_checkbox, \"Number of docs\", alt.value(100), legend=None\n",
    "        ),\n",
    "        tooltip=[\n",
    "            \"Model\",\n",
    "            \"Domain\",\n",
    "            \"Average F1\",\n",
    "        ],\n",
    "        opacity=alt.condition(selection, alt.value(1), alt.value(0.0)),\n",
    "    )\n",
    ")\n",
    "error_bars = (\n",
    "    alt.Chart(df)\n",
    "    .mark_errorbar(ticks=False)\n",
    "    .encode(\n",
    "        x=alt.X(\"Average Lower CI\", title=\"F1\"),\n",
    "        x2=\"Average Upper CI\",\n",
    "        y=alt.Y(\"Model\", sort=sort_order),\n",
    "        color=\"Domain\",\n",
    "        opacity=alt.condition(selection, alt.value(1), alt.value(0.0)),\n",
    "    )\n",
    ")\n",
    "\n",
    "chart = base + error_bars\n",
    "\n",
    "chart = chart.add_params(selection, param_checkbox).properties(\n",
    "    title=\"DANSK test set performance\",\n",
    ")\n",
    "\n",
    "\n",
    "chart.properties(width=400, height=300)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Domain generalization using CoNLL-2003 format\n",
    "To test the generalization we here convert the annotations to the CoNLL-2003 format using the labels `Person`, `Location`, `Organization`. As CoNLL-2003, `Location` includes cities, roads, mountains, abstract places, specific buildings, and meeting points. Thus the `GPE` (geo-political entity) were converted to `Location`. The `MISC` category in CoNLL-2003 is a diverse category meant to denote all names not in other categories (encapsulating both e.g. events and adjectives such as ”2004 World Cup” and ”Italian”), and is therefore not included."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "from evaluation.utils import convert_to_conll_2003, create_row_conll2003"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "tables = []\n",
    "for mdl_name in dansk:\n",
    "    # if \"fine_grained\" in mdl_name:\n",
    "    #     continue\n",
    "    examples = dansk[mdl_name][\"test\"][\"examples\"]\n",
    "    # examples += dansk[mdl_name][\"dev\"][\"examples\"]\n",
    "    # examples += dansk[mdl_name][\"train\"][\"examples\"]\n",
    "\n",
    "    examples = convert_to_conll_2003(examples)\n",
    "    table = evaluate_generalization(\n",
    "        mdl_name,\n",
    "        examples,\n",
    "        n_rep=100,\n",
    "        n_samples=1000,\n",
    "        create_row_fn=create_row_conll2003,\n",
    "    )\n",
    "    tables.append(table)\n",
    "\n",
    "tables = pd.concat(tables, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "df = tables\n",
    "df = df[df[\"Domain\"] != \"dannet\"]  # type: ignore\n",
    "df = df[df[\"Domain\"].notnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "  #altair-viz-af178b9951294daab69185da8a796b24.vega-embed {\n",
       "    width: 100%;\n",
       "    display: flex;\n",
       "  }\n",
       "\n",
       "  #altair-viz-af178b9951294daab69185da8a796b24.vega-embed details,\n",
       "  #altair-viz-af178b9951294daab69185da8a796b24.vega-embed details summary {\n",
       "    position: relative;\n",
       "  }\n",
       "</style>\n",
       "<div id=\"altair-viz-af178b9951294daab69185da8a796b24\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "  var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
       "  (function(spec, embedOpt){\n",
       "    let outputDiv = document.currentScript.previousElementSibling;\n",
       "    if (outputDiv.id !== \"altair-viz-af178b9951294daab69185da8a796b24\") {\n",
       "      outputDiv = document.getElementById(\"altair-viz-af178b9951294daab69185da8a796b24\");\n",
       "    }\n",
       "    const paths = {\n",
       "      \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
       "      \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
       "      \"vega-lite\": \"https://cdn.jsdelivr.net/npm/vega-lite@5.8.0?noext\",\n",
       "      \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
       "    };\n",
       "\n",
       "    function maybeLoadScript(lib, version) {\n",
       "      var key = `${lib.replace(\"-\", \"\")}_version`;\n",
       "      return (VEGA_DEBUG[key] == version) ?\n",
       "        Promise.resolve(paths[lib]) :\n",
       "        new Promise(function(resolve, reject) {\n",
       "          var s = document.createElement('script');\n",
       "          document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "          s.async = true;\n",
       "          s.onload = () => {\n",
       "            VEGA_DEBUG[key] = version;\n",
       "            return resolve(paths[lib]);\n",
       "          };\n",
       "          s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
       "          s.src = paths[lib];\n",
       "        });\n",
       "    }\n",
       "\n",
       "    function showError(err) {\n",
       "      outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
       "      throw err;\n",
       "    }\n",
       "\n",
       "    function displayChart(vegaEmbed) {\n",
       "      vegaEmbed(outputDiv, spec, embedOpt)\n",
       "        .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
       "    }\n",
       "\n",
       "    if(typeof define === \"function\" && define.amd) {\n",
       "      requirejs.config({paths});\n",
       "      require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
       "    } else {\n",
       "      maybeLoadScript(\"vega\", \"5\")\n",
       "        .then(() => maybeLoadScript(\"vega-lite\", \"5.8.0\"))\n",
       "        .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
       "        .catch(showError)\n",
       "        .then(() => displayChart(vegaEmbed));\n",
       "    }\n",
       "  })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"layer\": [{\"mark\": {\"type\": \"point\", \"filled\": true}, \"encoding\": {\"color\": {\"field\": \"Domain\", \"type\": \"nominal\"}, \"opacity\": {\"condition\": {\"param\": \"param_4\", \"value\": 1}, \"value\": 0.0}, \"size\": {\"condition\": {\"param\": \"param_5\", \"field\": \"Number of docs\", \"legend\": null}, \"value\": 100}, \"tooltip\": [{\"field\": \"Model\", \"type\": \"nominal\"}, {\"field\": \"Domain\", \"type\": \"nominal\"}, {\"field\": \"Average F1\", \"type\": \"nominal\"}, {\"field\": \"Person F1\", \"type\": \"nominal\"}, {\"field\": \"Location F1\", \"type\": \"nominal\"}, {\"field\": \"Organization F1\", \"type\": \"nominal\"}], \"x\": {\"field\": \"Average\", \"scale\": {\"domain\": [0.0, 1.0]}, \"title\": \"F1\", \"type\": \"quantitative\"}, \"y\": {\"field\": \"Model\", \"sort\": [\"da_dacy_large_trf-0.2.0\", \"da_dacy_medium_trf-0.2.0\", \"da_dacy_small_trf-0.2.0\", \"da_dacy_large_ner_fine_grained-0.1.0\", \"da_dacy_medium_ner_fine_grained-0.1.0\", \"da_dacy_small_ner_fine_grained-0.1.0\", \"saattrupdan/nbailab-base-ner-scandi\", \"alexandrainst/da-ner-base\", \"da_core_news_trf-3.5.0\", \"da_core_news_lg-3.5.0\", \"da_core_news_md-3.5.0\", \"da_core_news_sm-3.5.0\"], \"type\": \"nominal\"}}, \"name\": \"view_3\"}, {\"mark\": {\"type\": \"errorbar\", \"ticks\": false}, \"encoding\": {\"color\": {\"field\": \"Domain\", \"type\": \"nominal\"}, \"opacity\": {\"condition\": {\"param\": \"param_4\", \"value\": 1}, \"value\": 0.0}, \"x\": {\"field\": \"Average Lower CI\", \"title\": \"F1\", \"type\": \"quantitative\"}, \"x2\": {\"field\": \"Average Upper CI\"}, \"y\": {\"field\": \"Model\", \"sort\": [\"da_dacy_large_trf-0.2.0\", \"da_dacy_medium_trf-0.2.0\", \"da_dacy_small_trf-0.2.0\", \"da_dacy_large_ner_fine_grained-0.1.0\", \"da_dacy_medium_ner_fine_grained-0.1.0\", \"da_dacy_small_ner_fine_grained-0.1.0\", \"saattrupdan/nbailab-base-ner-scandi\", \"alexandrainst/da-ner-base\", \"da_core_news_trf-3.5.0\", \"da_core_news_lg-3.5.0\", \"da_core_news_md-3.5.0\", \"da_core_news_sm-3.5.0\"], \"type\": \"nominal\"}}}], \"data\": {\"name\": \"data-d6e3e3cc205ffd0b72f085fec1aedba8\"}, \"height\": 300, \"params\": [{\"name\": \"param_4\", \"select\": {\"type\": \"point\", \"fields\": [\"Domain\"]}, \"bind\": \"legend\", \"value\": [{\"Domain\": \"All\"}], \"views\": [\"view_3\"]}, {\"name\": \"param_5\", \"bind\": {\"input\": \"checkbox\", \"name\": \"Scale point size by number of documents: \"}}], \"title\": \"Generalization to Unseen Domains\", \"width\": 400, \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.8.0.json\", \"datasets\": {\"data-d6e3e3cc205ffd0b72f085fec1aedba8\": [{\"Model\": \"da_dacy_large_trf-0.2.0\", \"Domain\": \"Web\", \"Average\": 0.667528985236362, \"Average Lower CI\": 0.6285297837859204, \"Average Upper CI\": 0.7115764966229456, \"Average F1\": \"0.67 (0.63, 0.71)\", \"Person F1\": \"0.76 (0.69, 0.82)\", \"Organization F1\": \"0.50 (0.44, 0.57)\", \"Location F1\": \"0.79 (0.74, 0.85)\", \"Number of docs\": 783}, {\"Model\": \"da_dacy_large_trf-0.2.0\", \"Domain\": \"Legal\", \"Average\": 0.6968613827360955, \"Average Lower CI\": 0.6479282839802326, \"Average Upper CI\": 0.7412643365519742, \"Average F1\": \"0.70 (0.65, 0.74)\", \"Person F1\": \"0.69 (0.62, 0.77)\", \"Organization F1\": \"0.70 (0.64, 0.76)\", \"Location F1\": \"0.66 (0.49, 0.83)\", \"Number of docs\": 239}, {\"Model\": \"da_dacy_large_trf-0.2.0\", \"Domain\": \"Conversation\", \"Average\": 0.6528786650509781, \"Average Lower CI\": 0.6040030164278661, \"Average Upper CI\": 0.6989262384874426, \"Average F1\": \"0.65 (0.60, 0.70)\", \"Person F1\": \"0.48 (0.38, 0.57)\", \"Organization F1\": \"0.32 (0.21, 0.45)\", \"Location F1\": \"0.94 (0.92, 0.97)\", \"Number of docs\": 168}, {\"Model\": \"da_dacy_large_trf-0.2.0\", \"Domain\": \"Wiki & Books\", \"Average\": 0.5765523040063475, \"Average Lower CI\": 0.5331836914924933, \"Average Upper CI\": 0.6298384082527658, \"Average F1\": \"0.58 (0.53, 0.63)\", \"Person F1\": \"0.53 (0.45, 0.61)\", \"Organization F1\": \"0.35 (0.25, 0.44)\", \"Location F1\": \"0.75 (0.69, 0.79)\", \"Number of docs\": 182}, {\"Model\": \"da_dacy_large_trf-0.2.0\", \"Domain\": \"News\", \"Average\": 0.8067395073942872, \"Average Lower CI\": 0.7685981818675921, \"Average Upper CI\": 0.8357818313352625, \"Average F1\": \"0.81 (0.77, 0.84)\", \"Person F1\": \"0.71 (0.66, 0.76)\", \"Organization F1\": \"0.67 (0.58, 0.72)\", \"Location F1\": \"0.95 (0.93, 0.97)\", \"Number of docs\": 39}, {\"Model\": \"da_dacy_large_trf-0.2.0\", \"Domain\": \"Social Media\", \"Average\": 0.6659289149665456, \"Average Lower CI\": 0.6125936329588015, \"Average Upper CI\": 0.7080083849594789, \"Average F1\": \"0.67 (0.61, 0.71)\", \"Person F1\": \"0.75 (0.69, 0.81)\", \"Organization F1\": \"0.50 (0.43, 0.57)\", \"Location F1\": \"0.80 (0.69, 0.89)\", \"Number of docs\": 64}, {\"Model\": \"da_dacy_large_trf-0.2.0\", \"Domain\": \"All\", \"Average\": 0.6664886410109309, \"Average Lower CI\": 0.6254746646499224, \"Average Upper CI\": 0.7091693893271708, \"Average F1\": \"0.67 (0.63, 0.71)\", \"Person F1\": \"0.69 (0.62, 0.76)\", \"Organization F1\": \"0.54 (0.47, 0.60)\", \"Location F1\": \"0.81 (0.76, 0.86)\", \"Number of docs\": 1500}, {\"Model\": \"da_dacy_medium_trf-0.2.0\", \"Domain\": \"Web\", \"Average\": 0.560980315279223, \"Average Lower CI\": 0.5193585634026125, \"Average Upper CI\": 0.6079729504090832, \"Average F1\": \"0.56 (0.52, 0.61)\", \"Person F1\": \"0.68 (0.62, 0.75)\", \"Organization F1\": \"0.39 (0.33, 0.47)\", \"Location F1\": \"0.65 (0.59, 0.73)\", \"Number of docs\": 783}, {\"Model\": \"da_dacy_medium_trf-0.2.0\", \"Domain\": \"Legal\", \"Average\": 0.6142971438502267, \"Average Lower CI\": 0.5566547785932218, \"Average Upper CI\": 0.6644733355448209, \"Average F1\": \"0.61 (0.56, 0.66)\", \"Person F1\": \"0.68 (0.59, 0.76)\", \"Organization F1\": \"0.55 (0.49, 0.60)\", \"Location F1\": \"0.79 (0.65, 0.88)\", \"Number of docs\": 239}, {\"Model\": \"da_dacy_medium_trf-0.2.0\", \"Domain\": \"Conversation\", \"Average\": 0.5841716925183772, \"Average Lower CI\": 0.5273059944237919, \"Average Upper CI\": 0.6335597406606979, \"Average F1\": \"0.58 (0.53, 0.63)\", \"Person F1\": \"0.27 (0.19, 0.35)\", \"Organization F1\": \"0.33 (0.21, 0.48)\", \"Location F1\": \"0.97 (0.95, 0.99)\", \"Number of docs\": 168}, {\"Model\": \"da_dacy_medium_trf-0.2.0\", \"Domain\": \"Wiki & Books\", \"Average\": 0.6093583129257678, \"Average Lower CI\": 0.5549533604933324, \"Average Upper CI\": 0.6755938881058109, \"Average F1\": \"0.61 (0.55, 0.68)\", \"Person F1\": \"0.60 (0.50, 0.69)\", \"Organization F1\": \"0.32 (0.22, 0.41)\", \"Location F1\": \"0.77 (0.71, 0.82)\", \"Number of docs\": 182}, {\"Model\": \"da_dacy_medium_trf-0.2.0\", \"Domain\": \"News\", \"Average\": 0.5369590803281531, \"Average Lower CI\": 0.5053508771929824, \"Average Upper CI\": 0.5729970657943418, \"Average F1\": \"0.54 (0.51, 0.57)\", \"Person F1\": \"0.50 (0.42, 0.57)\", \"Organization F1\": \"0.40 (0.32, 0.48)\", \"Location F1\": \"0.63 (0.59, 0.67)\", \"Number of docs\": 39}, {\"Model\": \"da_dacy_medium_trf-0.2.0\", \"Domain\": \"Social Media\", \"Average\": 0.514405014305934, \"Average Lower CI\": 0.453710852521392, \"Average Upper CI\": 0.5639178617992177, \"Average F1\": \"0.51 (0.45, 0.56)\", \"Person F1\": \"0.46 (0.39, 0.54)\", \"Organization F1\": \"0.53 (0.46, 0.60)\", \"Location F1\": \"0.66 (0.48, 0.82)\", \"Number of docs\": 64}, {\"Model\": \"da_dacy_medium_trf-0.2.0\", \"Domain\": \"All\", \"Average\": 0.5604888568076893, \"Average Lower CI\": 0.5077672413793103, \"Average Upper CI\": 0.6092426309009623, \"Average F1\": \"0.56 (0.51, 0.61)\", \"Person F1\": \"0.60 (0.52, 0.67)\", \"Organization F1\": \"0.42 (0.34, 0.52)\", \"Location F1\": \"0.70 (0.63, 0.76)\", \"Number of docs\": 1500}, {\"Model\": \"da_dacy_small_trf-0.2.0\", \"Domain\": \"Web\", \"Average\": 0.5861013092620353, \"Average Lower CI\": 0.5469861835863377, \"Average Upper CI\": 0.6220607908715325, \"Average F1\": \"0.59 (0.55, 0.62)\", \"Person F1\": \"0.76 (0.69, 0.82)\", \"Organization F1\": \"0.38 (0.30, 0.44)\", \"Location F1\": \"0.72 (0.65, 0.78)\", \"Number of docs\": 783}, {\"Model\": \"da_dacy_small_trf-0.2.0\", \"Domain\": \"Legal\", \"Average\": 0.6031883979717402, \"Average Lower CI\": 0.5434979369093571, \"Average Upper CI\": 0.6511363175744145, \"Average F1\": \"0.60 (0.54, 0.65)\", \"Person F1\": \"0.60 (0.48, 0.70)\", \"Organization F1\": \"0.61 (0.56, 0.65)\", \"Location F1\": \"0.57 (0.41, 0.69)\", \"Number of docs\": 239}, {\"Model\": \"da_dacy_small_trf-0.2.0\", \"Domain\": \"Conversation\", \"Average\": 0.5917388201389233, \"Average Lower CI\": 0.5411907114624507, \"Average Upper CI\": 0.6359840854149879, \"Average F1\": \"0.59 (0.54, 0.64)\", \"Person F1\": \"0.32 (0.23, 0.41)\", \"Organization F1\": \"0.27 (0.19, 0.36)\", \"Location F1\": \"0.97 (0.95, 0.99)\", \"Number of docs\": 168}, {\"Model\": \"da_dacy_small_trf-0.2.0\", \"Domain\": \"Wiki & Books\", \"Average\": 0.4403569955595427, \"Average Lower CI\": 0.4010310606479816, \"Average Upper CI\": 0.4939674287025001, \"Average F1\": \"0.44 (0.40, 0.49)\", \"Person F1\": \"0.55 (0.47, 0.64)\", \"Organization F1\": \"0.21 (0.14, 0.28)\", \"Location F1\": \"0.50 (0.43, 0.58)\", \"Number of docs\": 182}, {\"Model\": \"da_dacy_small_trf-0.2.0\", \"Domain\": \"News\", \"Average\": 0.8149559916249054, \"Average Lower CI\": 0.785616108996193, \"Average Upper CI\": 0.8356736335825664, \"Average F1\": \"0.81 (0.79, 0.84)\", \"Person F1\": \"0.94 (0.93, 0.96)\", \"Organization F1\": \"0.54 (0.49, 0.60)\", \"Location F1\": \"0.85 (0.81, 0.89)\", \"Number of docs\": 39}, {\"Model\": \"da_dacy_small_trf-0.2.0\", \"Domain\": \"Social Media\", \"Average\": 0.4971664867414185, \"Average Lower CI\": 0.44813006632277086, \"Average Upper CI\": 0.5539955490931692, \"Average F1\": \"0.50 (0.45, 0.55)\", \"Person F1\": \"0.71 (0.66, 0.77)\", \"Organization F1\": \"0.44 (0.35, 0.53)\", \"Location F1\": \"0.00 (0.00, 0.00)\", \"Number of docs\": 64}, {\"Model\": \"da_dacy_small_trf-0.2.0\", \"Domain\": \"All\", \"Average\": 0.5723364354480662, \"Average Lower CI\": 0.5205699646051561, \"Average Upper CI\": 0.6184630650966705, \"Average F1\": \"0.57 (0.52, 0.62)\", \"Person F1\": \"0.66 (0.57, 0.74)\", \"Organization F1\": \"0.41 (0.35, 0.48)\", \"Location F1\": \"0.70 (0.65, 0.75)\", \"Number of docs\": 1500}, {\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"Web\", \"Average\": 0.8479359476866852, \"Average Lower CI\": 0.815980341313657, \"Average Upper CI\": 0.876074782111192, \"Average F1\": \"0.85 (0.82, 0.88)\", \"Person F1\": \"0.88 (0.85, 0.92)\", \"Organization F1\": \"0.77 (0.71, 0.83)\", \"Location F1\": \"0.93 (0.90, 0.96)\", \"Number of docs\": 783}, {\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"Legal\", \"Average\": 0.9323961905417734, \"Average Lower CI\": 0.9095775312066574, \"Average Upper CI\": 0.9538810989194607, \"Average F1\": \"0.93 (0.91, 0.95)\", \"Person F1\": \"0.93 (0.90, 0.96)\", \"Organization F1\": \"0.94 (0.92, 0.96)\", \"Location F1\": \"0.80 (0.61, 0.95)\", \"Number of docs\": 239}, {\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"Conversation\", \"Average\": 0.8591304681896426, \"Average Lower CI\": 0.8296825361438397, \"Average Upper CI\": 0.8847152775751919, \"Average F1\": \"0.86 (0.83, 0.88)\", \"Person F1\": \"0.75 (0.69, 0.80)\", \"Organization F1\": \"0.81 (0.75, 0.87)\", \"Location F1\": \"1.00 (1.00, 1.00)\", \"Number of docs\": 168}, {\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"Wiki & Books\", \"Average\": 0.7875348540037215, \"Average Lower CI\": 0.7445908183632733, \"Average Upper CI\": 0.8366649937927573, \"Average F1\": \"0.79 (0.74, 0.84)\", \"Person F1\": \"0.78 (0.69, 0.86)\", \"Organization F1\": \"0.55 (0.44, 0.66)\", \"Location F1\": \"0.90 (0.85, 0.93)\", \"Number of docs\": 182}, {\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"News\", \"Average\": 0.8500446954275824, \"Average Lower CI\": 0.8161730032419687, \"Average Upper CI\": 0.8810427070328756, \"Average F1\": \"0.85 (0.82, 0.88)\", \"Person F1\": \"0.87 (0.82, 0.92)\", \"Organization F1\": \"0.80 (0.74, 0.85)\", \"Location F1\": \"0.85 (0.81, 0.89)\", \"Number of docs\": 39}, {\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"Social Media\", \"Average\": 0.5909784946289866, \"Average Lower CI\": 0.5274659030969784, \"Average Upper CI\": 0.6600959892254662, \"Average F1\": \"0.59 (0.53, 0.66)\", \"Person F1\": \"0.66 (0.59, 0.75)\", \"Organization F1\": \"0.40 (0.31, 0.49)\", \"Location F1\": \"0.66 (0.57, 0.77)\", \"Number of docs\": 64}, {\"Model\": \"da_dacy_large_ner_fine_grained-0.1.0\", \"Domain\": \"All\", \"Average\": 0.8518845269547171, \"Average Lower CI\": 0.814045689694639, \"Average Upper CI\": 0.8786310223266744, \"Average F1\": \"0.85 (0.81, 0.88)\", \"Person F1\": \"0.86 (0.80, 0.90)\", \"Organization F1\": \"0.79 (0.73, 0.85)\", \"Location F1\": \"0.93 (0.89, 0.96)\", \"Number of docs\": 1500}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"Web\", \"Average\": 0.8552022198021011, \"Average Lower CI\": 0.8190691929361429, \"Average Upper CI\": 0.8796596467868855, \"Average F1\": \"0.86 (0.82, 0.88)\", \"Person F1\": \"0.90 (0.86, 0.95)\", \"Organization F1\": \"0.77 (0.72, 0.81)\", \"Location F1\": \"0.92 (0.86, 0.97)\", \"Number of docs\": 783}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"Legal\", \"Average\": 0.8940226089404762, \"Average Lower CI\": 0.8673171181131379, \"Average Upper CI\": 0.920923950234295, \"Average F1\": \"0.89 (0.87, 0.92)\", \"Person F1\": \"0.88 (0.81, 0.93)\", \"Organization F1\": \"0.92 (0.89, 0.94)\", \"Location F1\": \"0.73 (0.56, 0.86)\", \"Number of docs\": 239}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"Conversation\", \"Average\": 0.8127690875409891, \"Average Lower CI\": 0.7722243765403375, \"Average Upper CI\": 0.8532208363903153, \"Average F1\": \"0.81 (0.77, 0.85)\", \"Person F1\": \"0.64 (0.56, 0.71)\", \"Organization F1\": \"0.95 (0.90, 0.98)\", \"Location F1\": \"0.94 (0.91, 0.98)\", \"Number of docs\": 168}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"Wiki & Books\", \"Average\": 0.8422759141178982, \"Average Lower CI\": 0.8056981273920175, \"Average Upper CI\": 0.8823107129709167, \"Average F1\": \"0.84 (0.81, 0.88)\", \"Person F1\": \"0.81 (0.73, 0.88)\", \"Organization F1\": \"0.66 (0.57, 0.76)\", \"Location F1\": \"0.93 (0.88, 0.96)\", \"Number of docs\": 182}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"News\", \"Average\": 0.7106600021241404, \"Average Lower CI\": 0.6810680688735207, \"Average Upper CI\": 0.7343385734988342, \"Average F1\": \"0.71 (0.68, 0.73)\", \"Person F1\": \"0.57 (0.51, 0.63)\", \"Organization F1\": \"0.57 (0.50, 0.63)\", \"Location F1\": \"0.83 (0.81, 0.86)\", \"Number of docs\": 39}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"Social Media\", \"Average\": 0.7540892243218367, \"Average Lower CI\": 0.7042482243890695, \"Average Upper CI\": 0.7994123340707965, \"Average F1\": \"0.75 (0.70, 0.80)\", \"Person F1\": \"0.80 (0.74, 0.86)\", \"Organization F1\": \"0.61 (0.53, 0.70)\", \"Location F1\": \"1.00 (1.00, 1.00)\", \"Number of docs\": 64}, {\"Model\": \"da_dacy_medium_ner_fine_grained-0.1.0\", \"Domain\": \"All\", \"Average\": 0.8497648811435619, \"Average Lower CI\": 0.8141689896266509, \"Average Upper CI\": 0.8784788241704733, \"Average F1\": \"0.85 (0.81, 0.88)\", \"Person F1\": \"0.85 (0.79, 0.90)\", \"Organization F1\": \"0.80 (0.76, 0.85)\", \"Location F1\": \"0.91 (0.86, 0.96)\", \"Number of docs\": 1500}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"Web\", \"Average\": 0.8193685336482619, \"Average Lower CI\": 0.7841398986925701, \"Average Upper CI\": 0.8474517107325313, \"Average F1\": \"0.82 (0.78, 0.85)\", \"Person F1\": \"0.88 (0.82, 0.92)\", \"Organization F1\": \"0.76 (0.70, 0.80)\", \"Location F1\": \"0.85 (0.79, 0.92)\", \"Number of docs\": 783}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"Legal\", \"Average\": 0.9427008846663705, \"Average Lower CI\": 0.9202775032938078, \"Average Upper CI\": 0.9608768656716418, \"Average F1\": \"0.94 (0.92, 0.96)\", \"Person F1\": \"0.93 (0.89, 0.95)\", \"Organization F1\": \"0.96 (0.94, 0.98)\", \"Location F1\": \"0.79 (0.64, 0.92)\", \"Number of docs\": 239}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"Conversation\", \"Average\": 0.7649388627018243, \"Average Lower CI\": 0.7207038002469476, \"Average Upper CI\": 0.8055727594260237, \"Average F1\": \"0.76 (0.72, 0.81)\", \"Person F1\": \"0.59 (0.51, 0.68)\", \"Organization F1\": \"0.78 (0.63, 0.87)\", \"Location F1\": \"0.95 (0.92, 0.97)\", \"Number of docs\": 168}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"Wiki & Books\", \"Average\": 0.8216292630976697, \"Average Lower CI\": 0.7807349131540755, \"Average Upper CI\": 0.8766204170925924, \"Average F1\": \"0.82 (0.78, 0.88)\", \"Person F1\": \"0.86 (0.80, 0.91)\", \"Organization F1\": \"0.53 (0.39, 0.68)\", \"Location F1\": \"0.89 (0.84, 0.94)\", \"Number of docs\": 182}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"News\", \"Average\": 0.8671303590558316, \"Average Lower CI\": 0.8393492316927406, \"Average Upper CI\": 0.8886426079967054, \"Average F1\": \"0.87 (0.84, 0.89)\", \"Person F1\": \"1.00 (1.00, 1.00)\", \"Organization F1\": \"0.49 (0.42, 0.55)\", \"Location F1\": \"0.91 (0.88, 0.93)\", \"Number of docs\": 39}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"Social Media\", \"Average\": 0.8086330903888892, \"Average Lower CI\": 0.7568825910931174, \"Average Upper CI\": 0.8574352548036759, \"Average F1\": \"0.81 (0.76, 0.86)\", \"Person F1\": \"1.00 (1.00, 1.00)\", \"Organization F1\": \"0.80 (0.74, 0.85)\", \"Location F1\": \"0.32 (0.18, 0.47)\", \"Number of docs\": 64}, {\"Model\": \"da_dacy_small_ner_fine_grained-0.1.0\", \"Domain\": \"All\", \"Average\": 0.8330711878200586, \"Average Lower CI\": 0.7951427836377669, \"Average Upper CI\": 0.8642824695513596, \"Average F1\": \"0.83 (0.80, 0.86)\", \"Person F1\": \"0.87 (0.82, 0.92)\", \"Organization F1\": \"0.79 (0.74, 0.83)\", \"Location F1\": \"0.85 (0.78, 0.92)\", \"Number of docs\": 1500}, {\"Model\": \"saattrupdan/nbailab-base-ner-scandi\", \"Domain\": \"Web\", \"Average\": 0.6524810903923257, \"Average Lower CI\": 0.6132546942561155, \"Average Upper CI\": 0.6905463578038948, \"Average F1\": \"0.65 (0.61, 0.69)\", \"Person F1\": \"0.77 (0.70, 0.84)\", \"Organization F1\": \"0.50 (0.43, 0.56)\", \"Location F1\": \"0.73 (0.66, 0.78)\", \"Number of docs\": 783}, {\"Model\": \"saattrupdan/nbailab-base-ner-scandi\", \"Domain\": \"Legal\", \"Average\": 0.5976467528188156, \"Average Lower CI\": 0.5316067972907595, \"Average Upper CI\": 0.6539271918293053, \"Average F1\": \"0.60 (0.53, 0.65)\", \"Person F1\": \"0.54 (0.45, 0.65)\", \"Organization F1\": \"0.62 (0.54, 0.70)\", \"Location F1\": \"0.73 (0.56, 0.88)\", \"Number of docs\": 239}, {\"Model\": \"saattrupdan/nbailab-base-ner-scandi\", \"Domain\": \"Conversation\", \"Average\": 0.5505961721940514, \"Average Lower CI\": 0.4930601080237976, \"Average Upper CI\": 0.600929203539823, \"Average F1\": \"0.55 (0.49, 0.60)\", \"Person F1\": \"0.25 (0.17, 0.34)\", \"Organization F1\": \"0.30 (0.17, 0.46)\", \"Location F1\": \"0.89 (0.85, 0.92)\", \"Number of docs\": 168}, {\"Model\": \"saattrupdan/nbailab-base-ner-scandi\", \"Domain\": \"Wiki & Books\", \"Average\": 0.7177764961181036, \"Average Lower CI\": 0.6609971847892879, \"Average Upper CI\": 0.7679188004082333, \"Average F1\": \"0.72 (0.66, 0.77)\", \"Person F1\": \"0.68 (0.60, 0.76)\", \"Organization F1\": \"0.66 (0.53, 0.76)\", \"Location F1\": \"0.76 (0.69, 0.81)\", \"Number of docs\": 182}, {\"Model\": \"saattrupdan/nbailab-base-ner-scandi\", \"Domain\": \"News\", \"Average\": 0.851357711465195, \"Average Lower CI\": 0.8335065109144074, \"Average Upper CI\": 0.8668864284886384, \"Average F1\": \"0.85 (0.83, 0.87)\", \"Person F1\": \"0.93 (0.91, 0.96)\", \"Organization F1\": \"0.67 (0.61, 0.72)\", \"Location F1\": \"0.87 (0.84, 0.89)\", \"Number of docs\": 39}, {\"Model\": \"saattrupdan/nbailab-base-ner-scandi\", \"Domain\": \"Social Media\", \"Average\": 0.4788494920088626, \"Average Lower CI\": 0.4215434857821456, \"Average Upper CI\": 0.5433242253573487, \"Average F1\": \"0.48 (0.42, 0.54)\", \"Person F1\": \"0.66 (0.60, 0.73)\", \"Organization F1\": \"0.24 (0.15, 0.33)\", \"Location F1\": \"0.33 (0.20, 0.44)\", \"Number of docs\": 64}, {\"Model\": \"saattrupdan/nbailab-base-ner-scandi\", \"Domain\": \"All\", \"Average\": 0.6404158423152169, \"Average Lower CI\": 0.5971249208216451, \"Average Upper CI\": 0.683077088871327, \"Average F1\": \"0.64 (0.60, 0.68)\", \"Person F1\": \"0.66 (0.57, 0.72)\", \"Organization F1\": \"0.52 (0.45, 0.59)\", \"Location F1\": \"0.75 (0.69, 0.81)\", \"Number of docs\": 1500}, {\"Model\": \"alexandrainst/da-ner-base\", \"Domain\": \"Web\", \"Average\": 0.6772797119549435, \"Average Lower CI\": 0.6294598513935309, \"Average Upper CI\": 0.7233569585810175, \"Average F1\": \"0.68 (0.63, 0.72)\", \"Person F1\": \"0.76 (0.70, 0.81)\", \"Organization F1\": \"0.52 (0.42, 0.61)\", \"Location F1\": \"0.77 (0.70, 0.83)\", \"Number of docs\": 783}, {\"Model\": \"alexandrainst/da-ner-base\", \"Domain\": \"Legal\", \"Average\": 0.7073682855388124, \"Average Lower CI\": 0.6587500118253631, \"Average Upper CI\": 0.7551344964967714, \"Average F1\": \"0.71 (0.66, 0.76)\", \"Person F1\": \"0.71 (0.61, 0.79)\", \"Organization F1\": \"0.72 (0.67, 0.77)\", \"Location F1\": \"0.61 (0.43, 0.76)\", \"Number of docs\": 239}, {\"Model\": \"alexandrainst/da-ner-base\", \"Domain\": \"Conversation\", \"Average\": 0.6887403634221382, \"Average Lower CI\": 0.635970040189989, \"Average Upper CI\": 0.7327359617682199, \"Average F1\": \"0.69 (0.64, 0.73)\", \"Person F1\": \"0.53 (0.44, 0.62)\", \"Organization F1\": \"0.34 (0.24, 0.43)\", \"Location F1\": \"0.97 (0.95, 0.99)\", \"Number of docs\": 168}, {\"Model\": \"alexandrainst/da-ner-base\", \"Domain\": \"Wiki & Books\", \"Average\": 0.6335285074209022, \"Average Lower CI\": 0.5778187958095092, \"Average Upper CI\": 0.6857142857142856, \"Average F1\": \"0.63 (0.58, 0.69)\", \"Person F1\": \"0.68 (0.60, 0.77)\", \"Organization F1\": \"0.28 (0.17, 0.38)\", \"Location F1\": \"0.70 (0.63, 0.75)\", \"Number of docs\": 182}, {\"Model\": \"alexandrainst/da-ner-base\", \"Domain\": \"News\", \"Average\": 0.7551014173976767, \"Average Lower CI\": 0.7182013576081971, \"Average Upper CI\": 0.7860757315552063, \"Average F1\": \"0.76 (0.72, 0.79)\", \"Person F1\": \"0.71 (0.65, 0.78)\", \"Organization F1\": \"0.67 (0.61, 0.71)\", \"Location F1\": \"0.82 (0.78, 0.84)\", \"Number of docs\": 39}, {\"Model\": \"alexandrainst/da-ner-base\", \"Domain\": \"Social Media\", \"Average\": 0.7074503583760692, \"Average Lower CI\": 0.6511280005502441, \"Average Upper CI\": 0.7515687250996016, \"Average F1\": \"0.71 (0.65, 0.75)\", \"Person F1\": \"0.73 (0.67, 0.80)\", \"Organization F1\": \"0.66 (0.58, 0.74)\", \"Location F1\": \"0.66 (0.48, 0.82)\", \"Number of docs\": 64}, {\"Model\": \"alexandrainst/da-ner-base\", \"Domain\": \"All\", \"Average\": 0.6712384598623063, \"Average Lower CI\": 0.6261242441439502, \"Average Upper CI\": 0.7250227766849571, \"Average F1\": \"0.67 (0.63, 0.73)\", \"Person F1\": \"0.70 (0.61, 0.76)\", \"Organization F1\": \"0.55 (0.47, 0.63)\", \"Location F1\": \"0.77 (0.71, 0.83)\", \"Number of docs\": 1500}, {\"Model\": \"da_core_news_trf-3.5.0\", \"Domain\": \"Web\", \"Average\": 0.5876969094308764, \"Average Lower CI\": 0.5434579199558514, \"Average Upper CI\": 0.6422389070437741, \"Average F1\": \"0.59 (0.54, 0.64)\", \"Person F1\": \"0.67 (0.59, 0.73)\", \"Organization F1\": \"0.41 (0.31, 0.50)\", \"Location F1\": \"0.70 (0.63, 0.76)\", \"Number of docs\": 783}, {\"Model\": \"da_core_news_trf-3.5.0\", \"Domain\": \"Legal\", \"Average\": 0.6430997167163972, \"Average Lower CI\": 0.6000158312452166, \"Average Upper CI\": 0.6841037360890302, \"Average F1\": \"0.64 (0.60, 0.68)\", \"Person F1\": \"0.66 (0.58, 0.75)\", \"Organization F1\": \"0.64 (0.60, 0.69)\", \"Location F1\": \"0.56 (0.39, 0.71)\", \"Number of docs\": 239}, {\"Model\": \"da_core_news_trf-3.5.0\", \"Domain\": \"Conversation\", \"Average\": 0.6953852175246458, \"Average Lower CI\": 0.6289815072302558, \"Average Upper CI\": 0.7450399759883073, \"Average F1\": \"0.70 (0.63, 0.75)\", \"Person F1\": \"0.56 (0.47, 0.64)\", \"Organization F1\": \"0.25 (0.15, 0.38)\", \"Location F1\": \"0.97 (0.95, 0.99)\", \"Number of docs\": 168}, {\"Model\": \"da_core_news_trf-3.5.0\", \"Domain\": \"Wiki & Books\", \"Average\": 0.6109104275242887, \"Average Lower CI\": 0.5573528623099887, \"Average Upper CI\": 0.6695693380449564, \"Average F1\": \"0.61 (0.56, 0.67)\", \"Person F1\": \"0.61 (0.54, 0.69)\", \"Organization F1\": \"0.23 (0.13, 0.33)\", \"Location F1\": \"0.74 (0.67, 0.79)\", \"Number of docs\": 182}, {\"Model\": \"da_core_news_trf-3.5.0\", \"Domain\": \"News\", \"Average\": 0.7366176070639676, \"Average Lower CI\": 0.6915650777625022, \"Average Upper CI\": 0.7704920977011495, \"Average F1\": \"0.74 (0.69, 0.77)\", \"Person F1\": \"0.71 (0.65, 0.77)\", \"Organization F1\": \"0.28 (0.20, 0.35)\", \"Location F1\": \"0.88 (0.84, 0.91)\", \"Number of docs\": 39}, {\"Model\": \"da_core_news_trf-3.5.0\", \"Domain\": \"Social Media\", \"Average\": 0.6051651691797325, \"Average Lower CI\": 0.5523764113518463, \"Average Upper CI\": 0.6585234798957399, \"Average F1\": \"0.61 (0.55, 0.66)\", \"Person F1\": \"0.70 (0.63, 0.78)\", \"Organization F1\": \"0.40 (0.32, 0.49)\", \"Location F1\": \"0.66 (0.48, 0.82)\", \"Number of docs\": 64}, {\"Model\": \"da_core_news_trf-3.5.0\", \"Domain\": \"All\", \"Average\": 0.6011565821198456, \"Average Lower CI\": 0.5541557024381119, \"Average Upper CI\": 0.6540204188481675, \"Average F1\": \"0.60 (0.55, 0.65)\", \"Person F1\": \"0.63 (0.55, 0.71)\", \"Organization F1\": \"0.44 (0.35, 0.52)\", \"Location F1\": \"0.74 (0.68, 0.79)\", \"Number of docs\": 1500}, {\"Model\": \"da_core_news_lg-3.5.0\", \"Domain\": \"Web\", \"Average\": 0.5470658597979223, \"Average Lower CI\": 0.5044154936607766, \"Average Upper CI\": 0.5882707104092878, \"Average F1\": \"0.55 (0.50, 0.59)\", \"Person F1\": \"0.69 (0.63, 0.74)\", \"Organization F1\": \"0.38 (0.29, 0.46)\", \"Location F1\": \"0.62 (0.54, 0.68)\", \"Number of docs\": 783}, {\"Model\": \"da_core_news_lg-3.5.0\", \"Domain\": \"Legal\", \"Average\": 0.5784534648295648, \"Average Lower CI\": 0.5095322693383038, \"Average Upper CI\": 0.6340613429593958, \"Average F1\": \"0.58 (0.51, 0.63)\", \"Person F1\": \"0.60 (0.49, 0.69)\", \"Organization F1\": \"0.53 (0.46, 0.60)\", \"Location F1\": \"0.77 (0.64, 0.88)\", \"Number of docs\": 239}, {\"Model\": \"da_core_news_lg-3.5.0\", \"Domain\": \"Conversation\", \"Average\": 0.5995742936928079, \"Average Lower CI\": 0.5269780219780219, \"Average Upper CI\": 0.6469814996904475, \"Average F1\": \"0.60 (0.53, 0.65)\", \"Person F1\": \"0.31 (0.22, 0.41)\", \"Organization F1\": \"0.39 (0.26, 0.52)\", \"Location F1\": \"0.95 (0.93, 0.97)\", \"Number of docs\": 168}, {\"Model\": \"da_core_news_lg-3.5.0\", \"Domain\": \"Wiki & Books\", \"Average\": 0.4283896843283366, \"Average Lower CI\": 0.37074527815617986, \"Average Upper CI\": 0.4901084860173577, \"Average F1\": \"0.43 (0.37, 0.49)\", \"Person F1\": \"0.28 (0.22, 0.37)\", \"Organization F1\": \"0.19 (0.11, 0.30)\", \"Location F1\": \"0.68 (0.62, 0.74)\", \"Number of docs\": 182}, {\"Model\": \"da_core_news_lg-3.5.0\", \"Domain\": \"News\", \"Average\": 0.6628651420839026, \"Average Lower CI\": 0.6025909090909091, \"Average Upper CI\": 0.7117209528439474, \"Average F1\": \"0.66 (0.60, 0.71)\", \"Person F1\": \"0.53 (0.45, 0.58)\", \"Organization F1\": \"0.25 (0.16, 0.32)\", \"Location F1\": \"0.91 (0.88, 0.93)\", \"Number of docs\": 39}, {\"Model\": \"da_core_news_lg-3.5.0\", \"Domain\": \"Social Media\", \"Average\": 0.48286163699360757, \"Average Lower CI\": 0.4253190422123371, \"Average Upper CI\": 0.5409267431597528, \"Average F1\": \"0.48 (0.43, 0.54)\", \"Person F1\": \"0.80 (0.76, 0.84)\", \"Organization F1\": \"0.23 (0.15, 0.32)\", \"Location F1\": \"0.22 (0.12, 0.29)\", \"Number of docs\": 64}, {\"Model\": \"da_core_news_lg-3.5.0\", \"Domain\": \"All\", \"Average\": 0.5368726417033386, \"Average Lower CI\": 0.48491520546368416, \"Average Upper CI\": 0.5913509919493961, \"Average F1\": \"0.54 (0.48, 0.59)\", \"Person F1\": \"0.57 (0.47, 0.65)\", \"Organization F1\": \"0.39 (0.32, 0.46)\", \"Location F1\": \"0.67 (0.59, 0.76)\", \"Number of docs\": 1500}, {\"Model\": \"da_core_news_md-3.5.0\", \"Domain\": \"Web\", \"Average\": 0.5136992348502423, \"Average Lower CI\": 0.4615804480651731, \"Average Upper CI\": 0.5552635415950478, \"Average F1\": \"0.51 (0.46, 0.56)\", \"Person F1\": \"0.64 (0.55, 0.71)\", \"Organization F1\": \"0.32 (0.24, 0.39)\", \"Location F1\": \"0.66 (0.58, 0.72)\", \"Number of docs\": 783}, {\"Model\": \"da_core_news_md-3.5.0\", \"Domain\": \"Legal\", \"Average\": 0.5378431117397959, \"Average Lower CI\": 0.47948233599514806, \"Average Upper CI\": 0.5903539633624378, \"Average F1\": \"0.54 (0.48, 0.59)\", \"Person F1\": \"0.66 (0.55, 0.75)\", \"Organization F1\": \"0.47 (0.40, 0.54)\", \"Location F1\": \"0.53 (0.38, 0.67)\", \"Number of docs\": 239}, {\"Model\": \"da_core_news_md-3.5.0\", \"Domain\": \"Conversation\", \"Average\": 0.6032162945763784, \"Average Lower CI\": 0.5328329918032786, \"Average Upper CI\": 0.6558421362582355, \"Average F1\": \"0.60 (0.53, 0.66)\", \"Person F1\": \"0.36 (0.25, 0.46)\", \"Organization F1\": \"0.27 (0.14, 0.39)\", \"Location F1\": \"0.90 (0.86, 0.94)\", \"Number of docs\": 168}, {\"Model\": \"da_core_news_md-3.5.0\", \"Domain\": \"Wiki & Books\", \"Average\": 0.41435781484086653, \"Average Lower CI\": 0.3706879483500717, \"Average Upper CI\": 0.47429130403638836, \"Average F1\": \"0.41 (0.37, 0.47)\", \"Person F1\": \"0.39 (0.30, 0.51)\", \"Organization F1\": \"0.19 (0.12, 0.26)\", \"Location F1\": \"0.56 (0.48, 0.62)\", \"Number of docs\": 182}, {\"Model\": \"da_core_news_md-3.5.0\", \"Domain\": \"News\", \"Average\": 0.8064540034418824, \"Average Lower CI\": 0.7688224313603484, \"Average Upper CI\": 0.8384070872947277, \"Average F1\": \"0.81 (0.77, 0.84)\", \"Person F1\": \"0.86 (0.81, 0.89)\", \"Organization F1\": \"0.33 (0.23, 0.42)\", \"Location F1\": \"0.91 (0.87, 0.93)\", \"Number of docs\": 39}, {\"Model\": \"da_core_news_md-3.5.0\", \"Domain\": \"Social Media\", \"Average\": 0.4651086913925664, \"Average Lower CI\": 0.41810312426591495, \"Average Upper CI\": 0.5236993464052287, \"Average F1\": \"0.47 (0.42, 0.52)\", \"Person F1\": \"0.62 (0.57, 0.68)\", \"Organization F1\": \"0.40 (0.32, 0.48)\", \"Location F1\": \"0.00 (0.00, 0.00)\", \"Number of docs\": 64}, {\"Model\": \"da_core_news_md-3.5.0\", \"Domain\": \"All\", \"Average\": 0.5165210406027024, \"Average Lower CI\": 0.4613247974915077, \"Average Upper CI\": 0.566645987527782, \"Average F1\": \"0.52 (0.46, 0.57)\", \"Person F1\": \"0.60 (0.51, 0.68)\", \"Organization F1\": \"0.34 (0.28, 0.41)\", \"Location F1\": \"0.67 (0.59, 0.74)\", \"Number of docs\": 1500}, {\"Model\": \"da_core_news_sm-3.5.0\", \"Domain\": \"Web\", \"Average\": 0.3010391861079031, \"Average Lower CI\": 0.26244658567239215, \"Average Upper CI\": 0.33846847078287406, \"Average F1\": \"0.30 (0.26, 0.34)\", \"Person F1\": \"0.36 (0.30, 0.42)\", \"Organization F1\": \"0.16 (0.10, 0.23)\", \"Location F1\": \"0.41 (0.33, 0.48)\", \"Number of docs\": 783}, {\"Model\": \"da_core_news_sm-3.5.0\", \"Domain\": \"Legal\", \"Average\": 0.45884372852577193, \"Average Lower CI\": 0.40300316018171045, \"Average Upper CI\": 0.5209330567629291, \"Average F1\": \"0.46 (0.40, 0.52)\", \"Person F1\": \"0.53 (0.42, 0.63)\", \"Organization F1\": \"0.43 (0.36, 0.49)\", \"Location F1\": \"0.35 (0.20, 0.51)\", \"Number of docs\": 239}, {\"Model\": \"da_core_news_sm-3.5.0\", \"Domain\": \"Conversation\", \"Average\": 0.5301938027694156, \"Average Lower CI\": 0.46568268969100224, \"Average Upper CI\": 0.5762441474834178, \"Average F1\": \"0.53 (0.47, 0.58)\", \"Person F1\": \"0.21 (0.14, 0.28)\", \"Organization F1\": \"0.14 (0.04, 0.25)\", \"Location F1\": \"0.90 (0.86, 0.94)\", \"Number of docs\": 168}, {\"Model\": \"da_core_news_sm-3.5.0\", \"Domain\": \"Wiki & Books\", \"Average\": 0.299486033053285, \"Average Lower CI\": 0.2557559774964838, \"Average Upper CI\": 0.3386012414960058, \"Average F1\": \"0.30 (0.26, 0.34)\", \"Person F1\": \"0.10 (0.05, 0.16)\", \"Organization F1\": \"0.22 (0.13, 0.31)\", \"Location F1\": \"0.46 (0.39, 0.52)\", \"Number of docs\": 182}, {\"Model\": \"da_core_news_sm-3.5.0\", \"Domain\": \"News\", \"Average\": 0.7424269925549841, \"Average Lower CI\": 0.7118787878787879, \"Average Upper CI\": 0.7746618006767242, \"Average F1\": \"0.74 (0.71, 0.77)\", \"Person F1\": \"0.57 (0.51, 0.62)\", \"Organization F1\": \"0.66 (0.62, 0.70)\", \"Location F1\": \"0.90 (0.86, 0.92)\", \"Number of docs\": 39}, {\"Model\": \"da_core_news_sm-3.5.0\", \"Domain\": \"Social Media\", \"Average\": 0.18544554542239552, \"Average Lower CI\": 0.1408037757437071, \"Average Upper CI\": 0.23491625287185944, \"Average F1\": \"0.19 (0.14, 0.23)\", \"Person F1\": \"0.42 (0.34, 0.49)\", \"Organization F1\": \"0.00 (0.00, 0.00)\", \"Location F1\": \"0.00 (0.00, 0.00)\", \"Number of docs\": 64}, {\"Model\": \"da_core_news_sm-3.5.0\", \"Domain\": \"All\", \"Average\": 0.3441244538142977, \"Average Lower CI\": 0.2946728008833272, \"Average Upper CI\": 0.3917402862168212, \"Average F1\": \"0.34 (0.29, 0.39)\", \"Person F1\": \"0.34 (0.26, 0.42)\", \"Organization F1\": \"0.22 (0.17, 0.30)\", \"Location F1\": \"0.48 (0.39, 0.56)\", \"Number of docs\": 1500}]}}, {\"mode\": \"vega-lite\"});\n",
       "</script>"
      ],
      "text/plain": [
       "alt.LayerChart(...)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create altair viz\n",
    "selection = alt.selection_point(\n",
    "    fields=[\"Domain\"],\n",
    "    bind=\"legend\",\n",
    "    value=[{\"Domain\": \"All\"}],\n",
    ")\n",
    "bind_checkbox = alt.binding_checkbox(\n",
    "    name=\"Scale point size by number of documents: \",\n",
    ")\n",
    "param_checkbox = alt.param(bind=bind_checkbox)\n",
    "\n",
    "sort_order = list(dansk.keys())\n",
    "\n",
    "base = (\n",
    "    alt.Chart(df)\n",
    "    .mark_point(filled=True)\n",
    "    .encode(\n",
    "        x=alt.X(\"Average\", title=\"F1\", scale=alt.Scale(domain=[0.0, 1.0])),\n",
    "        y=alt.Y(\"Model\", sort=sort_order),\n",
    "        color=\"Domain\",\n",
    "        size=alt.condition(\n",
    "            param_checkbox, \"Number of docs\", alt.value(100), legend=None\n",
    "        ),\n",
    "        tooltip=[\n",
    "            \"Model\",\n",
    "            \"Domain\",\n",
    "            \"Average F1\",\n",
    "            \"Person F1\",\n",
    "            \"Location F1\",\n",
    "            \"Organization F1\",\n",
    "        ],\n",
    "        opacity=alt.condition(selection, alt.value(1), alt.value(0.0)),\n",
    "    )\n",
    ")\n",
    "error_bars = (\n",
    "    alt.Chart(df)\n",
    "    .mark_errorbar(ticks=False)\n",
    "    .encode(\n",
    "        x=alt.X(\"Average Lower CI\", title=\"F1\"),\n",
    "        x2=\"Average Upper CI\",\n",
    "        y=alt.Y(\"Model\", sort=sort_order),\n",
    "        color=\"Domain\",\n",
    "        opacity=alt.condition(selection, alt.value(1), alt.value(0.0)),\n",
    "    )\n",
    ")\n",
    "\n",
    "chart = base + error_bars\n",
    "\n",
    "chart = chart.add_params(selection, param_checkbox).properties(\n",
    "    width=400,\n",
    "    height=300,\n",
    "    title=\"Generalization to Unseen Domains\",\n",
    ")\n",
    "\n",
    "\n",
    "chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\\begin{tabular}{lllll}\n",
      "\\toprule\n",
      " & Average F1 & Person F1 & Organization F1 & Location F1 \\\\\n",
      "Model &  &  &  &  \\\\\n",
      "\\midrule\n",
      "da\\_dacy\\_large\\_trf-0.2.0 & 0.67 (0.63, 0.71) & 0.69 (0.62, 0.76) & 0.54 (0.47, 0.60) & 0.81 (0.76, 0.86) \\\\\n",
      "da\\_dacy\\_medium\\_trf-0.2.0 & 0.56 (0.51, 0.61) & 0.60 (0.52, 0.67) & 0.42 (0.34, 0.52) & 0.70 (0.63, 0.76) \\\\\n",
      "da\\_dacy\\_small\\_trf-0.2.0 & 0.57 (0.52, 0.62) & 0.66 (0.57, 0.74) & 0.41 (0.35, 0.48) & 0.70 (0.65, 0.75) \\\\\n",
      "da\\_dacy\\_large\\_ner\\_fine\\_grained-0.1.0 & 0.85 (0.81, 0.88) & 0.86 (0.80, 0.90) & 0.79 (0.73, 0.85) & 0.93 (0.89, 0.96) \\\\\n",
      "da\\_dacy\\_medium\\_ner\\_fine\\_grained-0.1.0 & 0.85 (0.81, 0.88) & 0.85 (0.79, 0.90) & 0.80 (0.76, 0.85) & 0.91 (0.86, 0.96) \\\\\n",
      "da\\_dacy\\_small\\_ner\\_fine\\_grained-0.1.0 & 0.83 (0.8, 0.86) & 0.87 (0.82, 0.92) & 0.79 (0.74, 0.83) & 0.85 (0.78, 0.92) \\\\\n",
      "saattrupdan/nbailab-base-ner-scandi & 0.64 (0.6, 0.68) & 0.66 (0.57, 0.72) & 0.52 (0.45, 0.59) & 0.75 (0.69, 0.81) \\\\\n",
      "alexandrainst/da-ner-base & 0.67 (0.63, 0.73) & 0.70 (0.61, 0.76) & 0.55 (0.47, 0.63) & 0.77 (0.71, 0.83) \\\\\n",
      "da\\_core\\_news\\_trf-3.5.0 & 0.6 (0.55, 0.65) & 0.63 (0.55, 0.71) & 0.44 (0.35, 0.52) & 0.74 (0.68, 0.79) \\\\\n",
      "da\\_core\\_news\\_lg-3.5.0 & 0.54 (0.48, 0.59) & 0.57 (0.47, 0.65) & 0.39 (0.32, 0.46) & 0.67 (0.59, 0.76) \\\\\n",
      "da\\_core\\_news\\_md-3.5.0 & 0.52 (0.46, 0.57) & 0.60 (0.51, 0.68) & 0.34 (0.28, 0.41) & 0.67 (0.59, 0.74) \\\\\n",
      "da\\_core\\_news\\_sm-3.5.0 & 0.34 (0.29, 0.39) & 0.34 (0.26, 0.42) & 0.22 (0.17, 0.30) & 0.48 (0.39, 0.56) \\\\\n",
      "\\bottomrule\n",
      "\\end{tabular}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# add ci to average\n",
    "\n",
    "df = tables\n",
    "df = df[df[\"Domain\"] != \"dannet\"]  # type: ignore\n",
    "df = df[df[\"Domain\"].notnull()]\n",
    "\n",
    "df[\"Average F1\"] = (\n",
    "    df[\"Average\"].round(2).astype(str)\n",
    "    + \" (\"\n",
    "    + df[\"Average Lower CI\"].round(2).astype(str)\n",
    "    + \", \"\n",
    "    + df[\"Average Upper CI\"].round(2).astype(str)\n",
    "    + \")\"\n",
    ")\n",
    "\n",
    "df.drop([\"Average Lower CI\", \"Average Upper CI\"], axis=1, inplace=True)\n",
    "df.drop([\"Number of docs\", \"Average\"], axis=1, inplace=True)\n",
    "\n",
    "\n",
    "# filter all but average\n",
    "df = df[df[\"Domain\"] == \"All\"]\n",
    "df.drop([\"Domain\"], axis=1, inplace=True)\n",
    "df.set_index(\"Model\", inplace=True)\n",
    "df\n",
    "\n",
    "# convert to latex using styler\n",
    "style = df.style.format_index(escape=\"latex\", axis=1).format_index(\n",
    "    escape=\"latex\", axis=0\n",
    ")\n",
    "\n",
    "# print latex\n",
    "latex = style.to_latex(\n",
    "    hrules=True,\n",
    "    convert_css=True,\n",
    ")\n",
    "\n",
    "print(latex)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Biases\n",
    "\n",
    "To examine the biases in Danish models we use augmentation to replace names in the Danish dataset DaNE {cite}`hvingelby2020dane`, this approach\n",
    "is similar to that introduced in the initial DaCy paper {cite}`enevoldsen2021dacy`.\n",
    "\n",
    "Here is a short example of how the augmentation might look like:\n",
    "\n",
    "\n",
    "````{admonition} Example\n",
    "\n",
    "```{admonition} Original\n",
    ":class: note\n",
    "\n",
    "\n",
    "Peter Schmeichel mener også, at det danske landshold anno 2021 tilhører verdenstoppen og kan vinde den kommende kamp mod England.\n",
    "```\n",
    "\n",
    "```{admonition} Female name augmentation\n",
    ":class: important\n",
    "\n",
    "Anne Østergaard mener også, at det danske landshold anno 2021 tilhører verdenstoppen og kan vinde den kommende kamp mod England.\n",
    "```\n",
    "````\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gender_bias_dane (test): Loading prediction for saattrupdan/nbailab-base-ner-scandi\n",
      "gender_bias_dane (test): Loading prediction for da_dacy_large_trf-0.2.0\n",
      "gender_bias_dane (test): Loading prediction for da_dacy_medium_trf-0.2.0\n",
      "gender_bias_dane (test): Loading prediction for da_dacy_small_trf-0.2.0\n",
      "gender_bias_dane (test): Loading prediction for alexandrainst/da-ner-base\n",
      "gender_bias_dane (test): Loading prediction for da_core_news_trf-3.5.0\n",
      "gender_bias_dane (test): Loading prediction for da_core_news_lg-3.5.0\n",
      "gender_bias_dane (test): Loading prediction for da_core_news_md-3.5.0\n",
      "gender_bias_dane (test): Loading prediction for da_core_news_sm-3.5.0\n"
     ]
    }
   ],
   "source": [
    "gbiases = {}\n",
    "for mdl_name, model_getter in MODELS.items():\n",
    "    if \"fine_grained\" in mdl_name:\n",
    "        continue\n",
    "    mdl_results = apply_models(\n",
    "        mdl_name, model_getter, dataset=\"gender_bias_dane\", splits=[\"test\"]\n",
    "    )\n",
    "    gbiases[mdl_name] = mdl_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "\n",
    "def augmentation_specific_examples(examples):\n",
    "    aug_group = defaultdict(list)\n",
    "    for example in examples:\n",
    "        aug_name = example.y._.meta[\"augmenter\"]\n",
    "        aug_group[aug_name].append(example)\n",
    "    return aug_group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saattrupdan/nbailab-base-ner-scandi\n",
      "da_dacy_large_trf-0.2.0\n",
      "da_dacy_medium_trf-0.2.0\n",
      "da_dacy_small_trf-0.2.0\n",
      "alexandrainst/da-ner-base\n",
      "da_core_news_trf-3.5.0\n",
      "da_core_news_lg-3.5.0\n",
      "da_core_news_md-3.5.0\n",
      "da_core_news_sm-3.5.0\n"
     ]
    }
   ],
   "source": [
    "tables = []\n",
    "for mdl in gbiases:\n",
    "    print(mdl)\n",
    "    examples = gbiases[mdl][\"test\"][\"examples\"]\n",
    "\n",
    "    aug_group = augmentation_specific_examples(examples)\n",
    "    for aug_name, _examples in aug_group.items():\n",
    "        _examples = convert_to_conll_2003(_examples)  # also removes misc.\n",
    "        table = create_dataframe(_examples, mdl, n_rep=100, n_samples=1000)\n",
    "        table[\"Augmentation\"] = aug_name\n",
    "        tables.append(table)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "df = pd.concat(tables)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "# create the table\n",
    "def create_table(df, model_order: list[str], baseline=df_average):\n",
    "    table_df = df[[\"Models\", \"Augmentation\", \"Average\"]]\n",
    "\n",
    "    table_df = table_df.pivot(index=\"Models\", columns=\"Augmentation\", values=\"Average\")\n",
    "\n",
    "    # order the models column\n",
    "    table_df = table_df.reindex(model_order)\n",
    "\n",
    "    # add baseline\n",
    "    table_df[\"Baseline\"] = list(baseline)\n",
    "    # order the columns\n",
    "    table_df = table_df[[\"Baseline\"] + list(table_df.columns[:-1])]\n",
    "\n",
    "    # create augmentation superheader\n",
    "\n",
    "    aug_superheader = [(\"\", \"Baseline\")]\n",
    "    for aug_name in table_df.columns[1:]:\n",
    "        aug_superheader.append((\"Augmentation\", aug_name))\n",
    "\n",
    "    aug_superheader = pd.MultiIndex.from_tuples(aug_superheader)\n",
    "    table_df.columns = aug_superheader\n",
    "    df = table_df.reset_index()\n",
    "    s = df.style.apply(highlight_max, axis=0, subset=df.columns[1:])\n",
    "    s = s.apply(underline_second_max, axis=0, subset=df.columns[1:])\n",
    "\n",
    "    # Add a caption\n",
    "    s = s.set_caption(\n",
    "        \"F1 score for each augmentation with 95% confidence interval calculated over 100 repetitions\"\n",
    "    )\n",
    "\n",
    "    # Center the header and left align the model names\n",
    "    s = s.set_properties(subset=df.columns[1:], **{\"text-align\": \"right\"})\n",
    "\n",
    "    super_header_style = [\n",
    "        {\"selector\": \".level0\", \"props\": [(\"text-align\", \"center\")]},\n",
    "        {\"selector\": \".col_heading\", \"props\": [(\"text-align\", \"center\")]},\n",
    "    ]\n",
    "    # Apply the CSS style to the styler\n",
    "    s = s.set_table_styles(super_header_style)  # type: ignore\n",
    "    # s = s.set_properties(subset=[(\"\", \"Models\")],\n",
    "    #                       **{\"text-align\": \"left\"})\n",
    "    # remove the index\n",
    "    s = s.hide(axis=\"index\")\n",
    "    # smaller font\n",
    "    s = s.set_table_attributes('style=\"font-size: 0.65em\"')\n",
    "    return s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_1ce53 .level0 {\n",
       "  text-align: center;\n",
       "}\n",
       "#T_1ce53 .col_heading {\n",
       "  text-align: center;\n",
       "}\n",
       "#T_1ce53_row0_col1, #T_1ce53_row0_col2, #T_1ce53_row0_col3, #T_1ce53_row0_col4, #T_1ce53_row0_col5 {\n",
       "  font-weight: bold;\n",
       "  text-align: right;\n",
       "}\n",
       "#T_1ce53_row1_col1, #T_1ce53_row1_col2, #T_1ce53_row1_col3, #T_1ce53_row1_col4, #T_1ce53_row1_col5 {\n",
       "  text-decoration: underline;\n",
       "  text-align: right;\n",
       "}\n",
       "#T_1ce53_row2_col1, #T_1ce53_row2_col2, #T_1ce53_row2_col3, #T_1ce53_row2_col4, #T_1ce53_row2_col5, #T_1ce53_row3_col1, #T_1ce53_row3_col2, #T_1ce53_row3_col3, #T_1ce53_row3_col4, #T_1ce53_row3_col5, #T_1ce53_row4_col1, #T_1ce53_row4_col2, #T_1ce53_row4_col3, #T_1ce53_row4_col4, #T_1ce53_row4_col5, #T_1ce53_row5_col1, #T_1ce53_row5_col2, #T_1ce53_row5_col3, #T_1ce53_row5_col4, #T_1ce53_row5_col5, #T_1ce53_row6_col1, #T_1ce53_row6_col2, #T_1ce53_row6_col3, #T_1ce53_row6_col4, #T_1ce53_row6_col5, #T_1ce53_row7_col1, #T_1ce53_row7_col2, #T_1ce53_row7_col3, #T_1ce53_row7_col4, #T_1ce53_row7_col5, #T_1ce53_row8_col1, #T_1ce53_row8_col2, #T_1ce53_row8_col3, #T_1ce53_row8_col4, #T_1ce53_row8_col5 {\n",
       "  text-align: right;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_1ce53\" style=\"font-size: 0.65em\">\n",
       "  <caption>F1 score for each augmentation with 95% confidence interval calculated over 100 repetitions</caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th id=\"T_1ce53_level0_col0\" class=\"col_heading level0 col0\" >Models</th>\n",
       "      <th id=\"T_1ce53_level0_col1\" class=\"col_heading level0 col1\" ></th>\n",
       "      <th id=\"T_1ce53_level0_col2\" class=\"col_heading level0 col2\" colspan=\"4\">Augmentation</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_1ce53_level1_col0\" class=\"col_heading level1 col0\" ></th>\n",
       "      <th id=\"T_1ce53_level1_col1\" class=\"col_heading level1 col1\" >Baseline</th>\n",
       "      <th id=\"T_1ce53_level1_col2\" class=\"col_heading level1 col2\" >Danish Names</th>\n",
       "      <th id=\"T_1ce53_level1_col3\" class=\"col_heading level1 col3\" >Female Names</th>\n",
       "      <th id=\"T_1ce53_level1_col4\" class=\"col_heading level1 col4\" >Male Names</th>\n",
       "      <th id=\"T_1ce53_level1_col5\" class=\"col_heading level1 col5\" >Muslim Names</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td id=\"T_1ce53_row0_col0\" class=\"data row0 col0\" >saattrupdan/nbailab-base-ner-scandi</td>\n",
       "      <td id=\"T_1ce53_row0_col1\" class=\"data row0 col1\" >86.3 (82.4, 89.7)</td>\n",
       "      <td id=\"T_1ce53_row0_col2\" class=\"data row0 col2\" >89.0 (86.8, 91.1)</td>\n",
       "      <td id=\"T_1ce53_row0_col3\" class=\"data row0 col3\" >88.9 (86.9, 91.1)</td>\n",
       "      <td id=\"T_1ce53_row0_col4\" class=\"data row0 col4\" >88.9 (86.9, 91.1)</td>\n",
       "      <td id=\"T_1ce53_row0_col5\" class=\"data row0 col5\" >88.1 (85.9, 90.4)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_1ce53_row1_col0\" class=\"data row1 col0\" >da_dacy_large_trf-0.2.0</td>\n",
       "      <td id=\"T_1ce53_row1_col1\" class=\"data row1 col1\" >85.4 (81.2, 88.9)</td>\n",
       "      <td id=\"T_1ce53_row1_col2\" class=\"data row1 col2\" >87.7 (85.2, 90.4)</td>\n",
       "      <td id=\"T_1ce53_row1_col3\" class=\"data row1 col3\" >87.8 (85.2, 90.2)</td>\n",
       "      <td id=\"T_1ce53_row1_col4\" class=\"data row1 col4\" >87.5 (84.3, 90.3)</td>\n",
       "      <td id=\"T_1ce53_row1_col5\" class=\"data row1 col5\" >85.6 (82.9, 88.3)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_1ce53_row2_col0\" class=\"data row2 col0\" >da_dacy_medium_trf-0.2.0</td>\n",
       "      <td id=\"T_1ce53_row2_col1\" class=\"data row2 col1\" >84.9 (81.0, 88.5)</td>\n",
       "      <td id=\"T_1ce53_row2_col2\" class=\"data row2 col2\" >86.2 (83.9, 88.8)</td>\n",
       "      <td id=\"T_1ce53_row2_col3\" class=\"data row2 col3\" >86.1 (83.8, 89.1)</td>\n",
       "      <td id=\"T_1ce53_row2_col4\" class=\"data row2 col4\" >86.1 (83.6, 89.2)</td>\n",
       "      <td id=\"T_1ce53_row2_col5\" class=\"data row2 col5\" >84.2 (81.7, 87.4)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_1ce53_row3_col0\" class=\"data row3 col0\" >da_dacy_small_trf-0.2.0</td>\n",
       "      <td id=\"T_1ce53_row3_col1\" class=\"data row3 col1\" >82.7 (79.3, 85.9)</td>\n",
       "      <td id=\"T_1ce53_row3_col2\" class=\"data row3 col2\" >82.4 (79.6, 85.3)</td>\n",
       "      <td id=\"T_1ce53_row3_col3\" class=\"data row3 col3\" >82.2 (79.9, 84.7)</td>\n",
       "      <td id=\"T_1ce53_row3_col4\" class=\"data row3 col4\" >82.1 (79.2, 85.2)</td>\n",
       "      <td id=\"T_1ce53_row3_col5\" class=\"data row3 col5\" >81.2 (78.6, 83.7)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_1ce53_row4_col0\" class=\"data row4 col0\" >alexandrainst/da-ner-base</td>\n",
       "      <td id=\"T_1ce53_row4_col1\" class=\"data row4 col1\" >70.7 (66.2, 75.2)</td>\n",
       "      <td id=\"T_1ce53_row4_col2\" class=\"data row4 col2\" >81.5 (78.2, 84.4)</td>\n",
       "      <td id=\"T_1ce53_row4_col3\" class=\"data row4 col3\" >81.6 (78.3, 84.4)</td>\n",
       "      <td id=\"T_1ce53_row4_col4\" class=\"data row4 col4\" >81.5 (78.2, 84.4)</td>\n",
       "      <td id=\"T_1ce53_row4_col5\" class=\"data row4 col5\" >79.8 (76.7, 82.4)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_1ce53_row5_col0\" class=\"data row5 col0\" >da_core_news_trf-3.5.0</td>\n",
       "      <td id=\"T_1ce53_row5_col1\" class=\"data row5 col1\" >79.0 (75.1, 82.3)</td>\n",
       "      <td id=\"T_1ce53_row5_col2\" class=\"data row5 col2\" >80.7 (77.2, 83.1)</td>\n",
       "      <td id=\"T_1ce53_row5_col3\" class=\"data row5 col3\" >80.9 (78.1, 83.8)</td>\n",
       "      <td id=\"T_1ce53_row5_col4\" class=\"data row5 col4\" >80.6 (77.3, 83.8)</td>\n",
       "      <td id=\"T_1ce53_row5_col5\" class=\"data row5 col5\" >78.7 (75.8, 81.1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_1ce53_row6_col0\" class=\"data row6 col0\" >da_core_news_lg-3.5.0</td>\n",
       "      <td id=\"T_1ce53_row6_col1\" class=\"data row6 col1\" >74.6 (70.8, 78.1)</td>\n",
       "      <td id=\"T_1ce53_row6_col2\" class=\"data row6 col2\" >78.3 (75.5, 80.7)</td>\n",
       "      <td id=\"T_1ce53_row6_col3\" class=\"data row6 col3\" >78.5 (75.9, 81.1)</td>\n",
       "      <td id=\"T_1ce53_row6_col4\" class=\"data row6 col4\" >78.4 (75.4, 81.2)</td>\n",
       "      <td id=\"T_1ce53_row6_col5\" class=\"data row6 col5\" >68.2 (65.4, 71.2)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_1ce53_row7_col0\" class=\"data row7 col0\" >da_core_news_md-3.5.0</td>\n",
       "      <td id=\"T_1ce53_row7_col1\" class=\"data row7 col1\" >71.2 (66.9, 75.2)</td>\n",
       "      <td id=\"T_1ce53_row7_col2\" class=\"data row7 col2\" >75.7 (71.9, 78.7)</td>\n",
       "      <td id=\"T_1ce53_row7_col3\" class=\"data row7 col3\" >75.6 (72.2, 79.1)</td>\n",
       "      <td id=\"T_1ce53_row7_col4\" class=\"data row7 col4\" >75.5 (72.3, 78.9)</td>\n",
       "      <td id=\"T_1ce53_row7_col5\" class=\"data row7 col5\" >64.6 (60.5, 68.1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_1ce53_row8_col0\" class=\"data row8 col0\" >da_core_news_sm-3.5.0</td>\n",
       "      <td id=\"T_1ce53_row8_col1\" class=\"data row8 col1\" >64.4 (59.7, 68.5)</td>\n",
       "      <td id=\"T_1ce53_row8_col2\" class=\"data row8 col2\" >58.8 (55.5, 62.0)</td>\n",
       "      <td id=\"T_1ce53_row8_col3\" class=\"data row8 col3\" >59.1 (56.2, 62.6)</td>\n",
       "      <td id=\"T_1ce53_row8_col4\" class=\"data row8 col4\" >59.1 (56.4, 62.3)</td>\n",
       "      <td id=\"T_1ce53_row8_col5\" class=\"data row8 col5\" >53.4 (50.2, 56.4)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x17bc48e50>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "create_table(df, model_order=list(gbiases.keys()))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Robustness"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the paper *'DaCy: A Unified Framework for Danish NLP'* {cite}`enevoldsen2021dacy` we conducted a series on augmentation on the DaNE test set to estimate the robustness and biases of DaCy and other Danish language processing pipelines. This page represents only parts of the paper. We recommend reading the paper for a more thorough and nuanced overview.\n",
    "\n",
    "The augmentation we will be using in this test are performed on the DaNE test set and include the following:\n",
    "\n",
    "- **Spelling Error**: Intended to similar domains with inconsistent spelling, OCR errors, conversational data, etc.. The augmentation includes a series of smaller augmentation:\n",
    "  - Keystroke error: The augmentation is used to introduce errors by replacing a character with a character that is close on the keyboard.\n",
    "  - Character swap: The augmentation is used to introduce errors by swapping two neighboring characters.\n",
    "  - Token swap: The augmentation is used to introduce errors by swapping two neighboring tokens.\n",
    "- **Inconsistent Casing**: This augmentation is used to simulate inconsistent casing in the language and uses two different methods by either randomly capitalizing or lowercasing tokens.\n",
    "- **Synonym Augmentation**: This augmentation is used to simulate the variation and slight grammatical errors in the language and uses two different methods:\n",
    "  - Wordnet Synonym replacement: The augmentation replaces a token with a synonym in WordNet while respecting its syntactic role.\n",
    "  - Embedding Synonym replacement: This augmentation replaces a token with a synonym which tends to appear in similar contexts.\n",
    "- **Inconsistent Spacing**: This augmentation is used to simulate inconsistent spacing in the language and uses two different methods by either randomly adding or removing spaces.\n",
    "- **Historical Spelling**: This augmentation is used to simulate historical spelling in Danish including ASCII spellings of the letters Æ (Ae), Ø (Oe), and Å (Aa) as well as uppercasing nouns.\n",
    "\n",
    "For all of the augmentations the probability of an augmentation is set to augment 5% of the spaces where the targeted augmentation can take place. The augmentations are performed using the [augmenty](https://kennethenevoldsen.github.io/augmenty/index.html).\n",
    "\n",
    "The underlying assumption of making these augmentations is that the annotations of the tokens do not change with augmentation. This can naturally sometimes be the case. A single letter *\"hun læste gåden\"* (*she read the puzzle*) and *\"hun løste gåden\"* (*she solved the puzzle*) have quite a different meaning. So while we expect the performance to drop the degree of the drop is interesting to examine and often in comparison to the other models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "robustness_dane (test): Loading prediction for saattrupdan/nbailab-base-ner-scandi\n",
      "robustness_dane (test): Loading prediction for da_dacy_large_trf-0.2.0\n",
      "robustness_dane (test): Loading prediction for da_dacy_medium_trf-0.2.0\n",
      "robustness_dane (test): Loading prediction for da_dacy_small_trf-0.2.0\n",
      "robustness_dane (test): Loading prediction for alexandrainst/da-ner-base\n",
      "robustness_dane (test): Loading prediction for da_core_news_trf-3.5.0\n",
      "robustness_dane (test): Loading prediction for da_core_news_lg-3.5.0\n",
      "robustness_dane (test): Loading prediction for da_core_news_md-3.5.0\n",
      "robustness_dane (test): Loading prediction for da_core_news_sm-3.5.0\n"
     ]
    }
   ],
   "source": [
    "robustness = {}\n",
    "for mdl_name, model_getter in MODELS.items():\n",
    "    if \"fine_grained\" in mdl_name:\n",
    "        continue\n",
    "    mdl_results = apply_models(\n",
    "        mdl_name, model_getter, dataset=\"robustness_dane\", splits=[\"test\"]\n",
    "    )\n",
    "    robustness[mdl_name] = mdl_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saattrupdan/nbailab-base-ner-scandi\n",
      "da_dacy_large_trf-0.2.0\n",
      "da_dacy_medium_trf-0.2.0\n",
      "da_dacy_small_trf-0.2.0\n",
      "alexandrainst/da-ner-base\n",
      "da_core_news_trf-3.5.0\n",
      "da_core_news_lg-3.5.0\n",
      "da_core_news_md-3.5.0\n",
      "da_core_news_sm-3.5.0\n"
     ]
    }
   ],
   "source": [
    "tables = []\n",
    "for mdl in robustness:\n",
    "    print(mdl)\n",
    "    examples = robustness[mdl][\"test\"][\"examples\"]\n",
    "\n",
    "    aug_group = augmentation_specific_examples(examples)\n",
    "    for aug_name, _examples in aug_group.items():\n",
    "        _examples = convert_to_conll_2003(_examples)  # also removes misc.\n",
    "        table = create_dataframe(_examples, mdl, n_rep=100, n_samples=1000)\n",
    "        table[\"Augmentation\"] = aug_name\n",
    "        tables.append(table)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_f0a31 .level0 {\n",
       "  text-align: center;\n",
       "}\n",
       "#T_f0a31 .col_heading {\n",
       "  text-align: center;\n",
       "}\n",
       "#T_f0a31_row0_col1, #T_f0a31_row0_col4, #T_f0a31_row0_col5, #T_f0a31_row0_col6, #T_f0a31_row1_col2, #T_f0a31_row1_col3 {\n",
       "  font-weight: bold;\n",
       "  text-align: right;\n",
       "}\n",
       "#T_f0a31_row0_col2, #T_f0a31_row0_col3, #T_f0a31_row1_col1, #T_f0a31_row1_col6, #T_f0a31_row2_col4, #T_f0a31_row2_col5 {\n",
       "  text-decoration: underline;\n",
       "  text-align: right;\n",
       "}\n",
       "#T_f0a31_row1_col4, #T_f0a31_row1_col5, #T_f0a31_row2_col1, #T_f0a31_row2_col2, #T_f0a31_row2_col3, #T_f0a31_row2_col6, #T_f0a31_row3_col1, #T_f0a31_row3_col2, #T_f0a31_row3_col3, #T_f0a31_row3_col4, #T_f0a31_row3_col5, #T_f0a31_row3_col6, #T_f0a31_row4_col1, #T_f0a31_row4_col2, #T_f0a31_row4_col3, #T_f0a31_row4_col4, #T_f0a31_row4_col5, #T_f0a31_row4_col6, #T_f0a31_row5_col1, #T_f0a31_row5_col2, #T_f0a31_row5_col3, #T_f0a31_row5_col4, #T_f0a31_row5_col5, #T_f0a31_row5_col6, #T_f0a31_row6_col1, #T_f0a31_row6_col2, #T_f0a31_row6_col3, #T_f0a31_row6_col4, #T_f0a31_row6_col5, #T_f0a31_row6_col6, #T_f0a31_row7_col1, #T_f0a31_row7_col2, #T_f0a31_row7_col3, #T_f0a31_row7_col4, #T_f0a31_row7_col5, #T_f0a31_row7_col6, #T_f0a31_row8_col1, #T_f0a31_row8_col2, #T_f0a31_row8_col3, #T_f0a31_row8_col4, #T_f0a31_row8_col5, #T_f0a31_row8_col6 {\n",
       "  text-align: right;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_f0a31\" style=\"font-size: 0.65em\">\n",
       "  <caption>F1 score for each augmentation with 95% confidence interval calculated over 100 repetitions</caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th id=\"T_f0a31_level0_col0\" class=\"col_heading level0 col0\" >Models</th>\n",
       "      <th id=\"T_f0a31_level0_col1\" class=\"col_heading level0 col1\" ></th>\n",
       "      <th id=\"T_f0a31_level0_col2\" class=\"col_heading level0 col2\" colspan=\"5\">Augmentation</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_f0a31_level1_col0\" class=\"col_heading level1 col0\" ></th>\n",
       "      <th id=\"T_f0a31_level1_col1\" class=\"col_heading level1 col1\" >Baseline</th>\n",
       "      <th id=\"T_f0a31_level1_col2\" class=\"col_heading level1 col2\" >Historical Spelling</th>\n",
       "      <th id=\"T_f0a31_level1_col3\" class=\"col_heading level1 col3\" >Inconsistent Casing</th>\n",
       "      <th id=\"T_f0a31_level1_col4\" class=\"col_heading level1 col4\" >Inconsistent Spacing</th>\n",
       "      <th id=\"T_f0a31_level1_col5\" class=\"col_heading level1 col5\" >Spelling Error</th>\n",
       "      <th id=\"T_f0a31_level1_col6\" class=\"col_heading level1 col6\" >Synonym replacement</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td id=\"T_f0a31_row0_col0\" class=\"data row0 col0\" >saattrupdan/nbailab-base-ner-scandi</td>\n",
       "      <td id=\"T_f0a31_row0_col1\" class=\"data row0 col1\" >86.3 (82.4, 89.7)</td>\n",
       "      <td id=\"T_f0a31_row0_col2\" class=\"data row0 col2\" >81.9 (79.1, 85.0)</td>\n",
       "      <td id=\"T_f0a31_row0_col3\" class=\"data row0 col3\" >86.5 (84.4, 89.0)</td>\n",
       "      <td id=\"T_f0a31_row0_col4\" class=\"data row0 col4\" >78.8 (75.7, 81.6)</td>\n",
       "      <td id=\"T_f0a31_row0_col5\" class=\"data row0 col5\" >73.3 (69.9, 76.8)</td>\n",
       "      <td id=\"T_f0a31_row0_col6\" class=\"data row0 col6\" >87.1 (84.9, 89.6)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_f0a31_row1_col0\" class=\"data row1 col0\" >da_dacy_large_trf-0.2.0</td>\n",
       "      <td id=\"T_f0a31_row1_col1\" class=\"data row1 col1\" >85.4 (81.2, 88.9)</td>\n",
       "      <td id=\"T_f0a31_row1_col2\" class=\"data row1 col2\" >86.0 (82.8, 88.9)</td>\n",
       "      <td id=\"T_f0a31_row1_col3\" class=\"data row1 col3\" >86.9 (83.9, 89.4)</td>\n",
       "      <td id=\"T_f0a31_row1_col4\" class=\"data row1 col4\" >69.7 (66.4, 72.4)</td>\n",
       "      <td id=\"T_f0a31_row1_col5\" class=\"data row1 col5\" >59.7 (56.4, 63.9)</td>\n",
       "      <td id=\"T_f0a31_row1_col6\" class=\"data row1 col6\" >85.9 (82.9, 88.8)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_f0a31_row2_col0\" class=\"data row2 col0\" >da_dacy_medium_trf-0.2.0</td>\n",
       "      <td id=\"T_f0a31_row2_col1\" class=\"data row2 col1\" >84.9 (81.0, 88.5)</td>\n",
       "      <td id=\"T_f0a31_row2_col2\" class=\"data row2 col2\" >69.6 (66.7, 72.1)</td>\n",
       "      <td id=\"T_f0a31_row2_col3\" class=\"data row2 col3\" >83.7 (81.3, 86.3)</td>\n",
       "      <td id=\"T_f0a31_row2_col4\" class=\"data row2 col4\" >70.5 (66.6, 74.0)</td>\n",
       "      <td id=\"T_f0a31_row2_col5\" class=\"data row2 col5\" >65.4 (62.6, 68.5)</td>\n",
       "      <td id=\"T_f0a31_row2_col6\" class=\"data row2 col6\" >85.1 (82.5, 88.3)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_f0a31_row3_col0\" class=\"data row3 col0\" >da_dacy_small_trf-0.2.0</td>\n",
       "      <td id=\"T_f0a31_row3_col1\" class=\"data row3 col1\" >82.7 (79.3, 85.9)</td>\n",
       "      <td id=\"T_f0a31_row3_col2\" class=\"data row3 col2\" >51.7 (49.1, 54.6)</td>\n",
       "      <td id=\"T_f0a31_row3_col3\" class=\"data row3 col3\" >81.1 (78.6, 83.5)</td>\n",
       "      <td id=\"T_f0a31_row3_col4\" class=\"data row3 col4\" >64.3 (60.4, 67.2)</td>\n",
       "      <td id=\"T_f0a31_row3_col5\" class=\"data row3 col5\" >63.1 (59.9, 66.5)</td>\n",
       "      <td id=\"T_f0a31_row3_col6\" class=\"data row3 col6\" >83.4 (81.0, 85.7)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_f0a31_row4_col0\" class=\"data row4 col0\" >alexandrainst/da-ner-base</td>\n",
       "      <td id=\"T_f0a31_row4_col1\" class=\"data row4 col1\" >70.7 (66.2, 75.2)</td>\n",
       "      <td id=\"T_f0a31_row4_col2\" class=\"data row4 col2\" >78.7 (75.3, 81.6)</td>\n",
       "      <td id=\"T_f0a31_row4_col3\" class=\"data row4 col3\" >80.8 (77.6, 83.2)</td>\n",
       "      <td id=\"T_f0a31_row4_col4\" class=\"data row4 col4\" >63.4 (59.4, 66.3)</td>\n",
       "      <td id=\"T_f0a31_row4_col5\" class=\"data row4 col5\" >49.9 (47.3, 53.6)</td>\n",
       "      <td id=\"T_f0a31_row4_col6\" class=\"data row4 col6\" >80.1 (77.1, 82.8)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_f0a31_row5_col0\" class=\"data row5 col0\" >da_core_news_trf-3.5.0</td>\n",
       "      <td id=\"T_f0a31_row5_col1\" class=\"data row5 col1\" >79.0 (75.1, 82.3)</td>\n",
       "      <td id=\"T_f0a31_row5_col2\" class=\"data row5 col2\" >75.1 (72.4, 77.3)</td>\n",
       "      <td id=\"T_f0a31_row5_col3\" class=\"data row5 col3\" >81.3 (78.5, 84.1)</td>\n",
       "      <td id=\"T_f0a31_row5_col4\" class=\"data row5 col4\" >58.9 (55.8, 62.3)</td>\n",
       "      <td id=\"T_f0a31_row5_col5\" class=\"data row5 col5\" >41.2 (38.5, 44.0)</td>\n",
       "      <td id=\"T_f0a31_row5_col6\" class=\"data row5 col6\" >80.4 (77.6, 83.3)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_f0a31_row6_col0\" class=\"data row6 col0\" >da_core_news_lg-3.5.0</td>\n",
       "      <td id=\"T_f0a31_row6_col1\" class=\"data row6 col1\" >74.6 (70.8, 78.1)</td>\n",
       "      <td id=\"T_f0a31_row6_col2\" class=\"data row6 col2\" >47.0 (44.5, 49.7)</td>\n",
       "      <td id=\"T_f0a31_row6_col3\" class=\"data row6 col3\" >74.5 (71.6, 77.7)</td>\n",
       "      <td id=\"T_f0a31_row6_col4\" class=\"data row6 col4\" >51.1 (48.1, 53.8)</td>\n",
       "      <td id=\"T_f0a31_row6_col5\" class=\"data row6 col5\" >44.9 (42.0, 47.9)</td>\n",
       "      <td id=\"T_f0a31_row6_col6\" class=\"data row6 col6\" >76.3 (73.6, 79.1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_f0a31_row7_col0\" class=\"data row7 col0\" >da_core_news_md-3.5.0</td>\n",
       "      <td id=\"T_f0a31_row7_col1\" class=\"data row7 col1\" >71.2 (66.9, 75.2)</td>\n",
       "      <td id=\"T_f0a31_row7_col2\" class=\"data row7 col2\" >48.7 (45.7, 51.6)</td>\n",
       "      <td id=\"T_f0a31_row7_col3\" class=\"data row7 col3\" >71.6 (68.2, 75.4)</td>\n",
       "      <td id=\"T_f0a31_row7_col4\" class=\"data row7 col4\" >51.1 (47.6, 54.3)</td>\n",
       "      <td id=\"T_f0a31_row7_col5\" class=\"data row7 col5\" >41.8 (38.8, 44.7)</td>\n",
       "      <td id=\"T_f0a31_row7_col6\" class=\"data row7 col6\" >72.8 (69.2, 76.1)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_f0a31_row8_col0\" class=\"data row8 col0\" >da_core_news_sm-3.5.0</td>\n",
       "      <td id=\"T_f0a31_row8_col1\" class=\"data row8 col1\" >64.4 (59.7, 68.5)</td>\n",
       "      <td id=\"T_f0a31_row8_col2\" class=\"data row8 col2\" >31.9 (29.6, 34.1)</td>\n",
       "      <td id=\"T_f0a31_row8_col3\" class=\"data row8 col3\" >61.5 (58.1, 64.6)</td>\n",
       "      <td id=\"T_f0a31_row8_col4\" class=\"data row8 col4\" >46.6 (43.7, 50.4)</td>\n",
       "      <td id=\"T_f0a31_row8_col5\" class=\"data row8 col5\" >49.6 (46.5, 53.0)</td>\n",
       "      <td id=\"T_f0a31_row8_col6\" class=\"data row8 col6\" >64.8 (61.4, 68.1)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f49c7cd0>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "robustness_df = pd.concat(tables)\n",
    "\n",
    "create_table(robustness_df, model_order=list(robustness.keys()))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference Speed\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While performance naturally is important is it also important to know why you might choose one model over another. One of the main reasons for choosing a smaller model is inference speed. The following table shows the inference speed of the different models. The inference speed is measured in words per second (WPS) and is measured on a Apple M1 Pro 16Gb running macOS 13.3.1 (i.e. high-end consumer laptop). The models are tested on the test set of DaNE.\n",
    "\n",
    "```{admonition} GPU Acceleration\n",
    ":class: note\n",
    "\n",
    "These benchmarks does not use GPU acceleration. If you were to use GPU acceleration the inference speed would be much higher, similarly larger models would benefit more from this acceleration.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dane (test): Loading prediction for saattrupdan/nbailab-base-ner-scandi\n",
      "dane (test): Loading prediction for da_dacy_large_trf-0.2.0\n",
      "dane (test): Loading prediction for da_dacy_medium_trf-0.2.0\n",
      "dane (test): Loading prediction for da_dacy_small_trf-0.2.0\n",
      "dane (test): Loading prediction for da_dacy_large_ner_fine_grained-0.1.0\n",
      "dane (test): Loading prediction for da_dacy_medium_ner_fine_grained-0.1.0\n",
      "dane (test): Loading prediction for da_dacy_small_ner_fine_grained-0.1.0\n",
      "dane (test): Loading prediction for alexandrainst/da-ner-base\n",
      "dane (test): Loading prediction for da_core_news_trf-3.5.0\n",
      "dane (test): Loading prediction for da_core_news_lg-3.5.0\n",
      "dane (test): Loading prediction for da_core_news_md-3.5.0\n",
      "dane (test): Loading prediction for da_core_news_sm-3.5.0\n"
     ]
    }
   ],
   "source": [
    "dane = {}\n",
    "for mdl_name, model_getter in MODELS.items():\n",
    "    mdl_results = apply_models(mdl_name, model_getter, dataset=\"dane\", splits=[\"test\"])\n",
    "    dane[mdl_name] = mdl_results[\"test\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "rows = []\n",
    "n_words = None\n",
    "for mdl_name, model_getter in MODELS.items():\n",
    "    total_time = dane[mdl_name][\"time_in_seconds\"]\n",
    "    if n_words is None:\n",
    "        examples = dane[mdl_name][\"examples\"]\n",
    "        n_words = sum(len(e.y) for e in examples)\n",
    "    wps = n_words / total_time\n",
    "    rows.append(\n",
    "        {\"Model\": mdl_name, \"Words per second\": wps, \"Total time (sec)\": total_time}\n",
    "    )\n",
    "\n",
    "speed = pd.DataFrame(rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_de718_row0_col0, #T_de718_row1_col0, #T_de718_row2_col0, #T_de718_row3_col0, #T_de718_row4_col0, #T_de718_row5_col0, #T_de718_row6_col0, #T_de718_row7_col0, #T_de718_row8_col0, #T_de718_row9_col0, #T_de718_row10_col0, #T_de718_row11_col0 {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_de718_row0_col1, #T_de718_row0_col2, #T_de718_row1_col1, #T_de718_row1_col2, #T_de718_row2_col1, #T_de718_row2_col2, #T_de718_row3_col1, #T_de718_row3_col2, #T_de718_row4_col1, #T_de718_row4_col2, #T_de718_row5_col1, #T_de718_row5_col2, #T_de718_row6_col1, #T_de718_row6_col2, #T_de718_row7_col1, #T_de718_row7_col2, #T_de718_row8_col1, #T_de718_row8_col2, #T_de718_row9_col1, #T_de718_row9_col2, #T_de718_row10_col1, #T_de718_row10_col2 {\n",
       "  text-align: right;\n",
       "}\n",
       "#T_de718_row11_col1, #T_de718_row11_col2 {\n",
       "  font-weight: bold;\n",
       "  text-align: right;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_de718\">\n",
       "  <caption>Inference speed on DANE test set</caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th id=\"T_de718_level0_col0\" class=\"col_heading level0 col0\" >Model</th>\n",
       "      <th id=\"T_de718_level0_col1\" class=\"col_heading level0 col1\" >Words per second</th>\n",
       "      <th id=\"T_de718_level0_col2\" class=\"col_heading level0 col2\" >Total time (sec)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td id=\"T_de718_row0_col0\" class=\"data row0 col0\" >saattrupdan/nbailab-base-ner-scandi</td>\n",
       "      <td id=\"T_de718_row0_col1\" class=\"data row0 col1\" >1438.8</td>\n",
       "      <td id=\"T_de718_row0_col2\" class=\"data row0 col2\" >6.97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_de718_row1_col0\" class=\"data row1 col0\" >da_dacy_large_trf-0.2.0</td>\n",
       "      <td id=\"T_de718_row1_col1\" class=\"data row1 col1\" >353.3</td>\n",
       "      <td id=\"T_de718_row1_col2\" class=\"data row1 col2\" >28.37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_de718_row2_col0\" class=\"data row2 col0\" >da_dacy_medium_trf-0.2.0</td>\n",
       "      <td id=\"T_de718_row2_col1\" class=\"data row2 col1\" >770.2</td>\n",
       "      <td id=\"T_de718_row2_col2\" class=\"data row2 col2\" >13.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_de718_row3_col0\" class=\"data row3 col0\" >da_dacy_small_trf-0.2.0</td>\n",
       "      <td id=\"T_de718_row3_col1\" class=\"data row3 col1\" >2024.6</td>\n",
       "      <td id=\"T_de718_row3_col2\" class=\"data row3 col2\" >4.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_de718_row4_col0\" class=\"data row4 col0\" >da_dacy_large_ner_fine_grained-0.1.0</td>\n",
       "      <td id=\"T_de718_row4_col1\" class=\"data row4 col1\" >567.9</td>\n",
       "      <td id=\"T_de718_row4_col2\" class=\"data row4 col2\" >17.65</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_de718_row5_col0\" class=\"data row5 col0\" >da_dacy_medium_ner_fine_grained-0.1.0</td>\n",
       "      <td id=\"T_de718_row5_col1\" class=\"data row5 col1\" >1670.3</td>\n",
       "      <td id=\"T_de718_row5_col2\" class=\"data row5 col2\" >6.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_de718_row6_col0\" class=\"data row6 col0\" >da_dacy_small_ner_fine_grained-0.1.0</td>\n",
       "      <td id=\"T_de718_row6_col1\" class=\"data row6 col1\" >5717.6</td>\n",
       "      <td id=\"T_de718_row6_col2\" class=\"data row6 col2\" >1.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_de718_row7_col0\" class=\"data row7 col0\" >alexandrainst/da-ner-base</td>\n",
       "      <td id=\"T_de718_row7_col1\" class=\"data row7 col1\" >1618.7</td>\n",
       "      <td id=\"T_de718_row7_col2\" class=\"data row7 col2\" >6.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_de718_row8_col0\" class=\"data row8 col0\" >da_core_news_trf-3.5.0</td>\n",
       "      <td id=\"T_de718_row8_col1\" class=\"data row8 col1\" >1125.1</td>\n",
       "      <td id=\"T_de718_row8_col2\" class=\"data row8 col2\" >8.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_de718_row9_col0\" class=\"data row9 col0\" >da_core_news_lg-3.5.0</td>\n",
       "      <td id=\"T_de718_row9_col1\" class=\"data row9 col1\" >31364.7</td>\n",
       "      <td id=\"T_de718_row9_col2\" class=\"data row9 col2\" >0.32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_de718_row10_col0\" class=\"data row10 col0\" >da_core_news_md-3.5.0</td>\n",
       "      <td id=\"T_de718_row10_col1\" class=\"data row10 col1\" >32571.3</td>\n",
       "      <td id=\"T_de718_row10_col2\" class=\"data row10 col2\" >0.31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td id=\"T_de718_row11_col0\" class=\"data row11 col0\" >da_core_news_sm-3.5.0</td>\n",
       "      <td id=\"T_de718_row11_col1\" class=\"data row11 col1\" >34624.4</td>\n",
       "      <td id=\"T_de718_row11_col2\" class=\"data row11 col2\" >0.29</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x2a5a6e950>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create the table\n",
    "style = speed.style.set_caption(\"Inference speed on DANE test set\")\n",
    "\n",
    "\n",
    "def highlight_min(s):\n",
    "    \"\"\"highlight the minimum in a series with bold\"\"\"\n",
    "    is_min = s == s.min()\n",
    "    return [\"font-weight: bold\" if v else \"\" for v in is_min]\n",
    "\n",
    "\n",
    "def highlight_max(s):\n",
    "    \"\"\"highlight the minimum in a series with bold\"\"\"\n",
    "    is_max = s == s.max()\n",
    "    return [\"font-weight: bold\" if v else \"\" for v in is_max]\n",
    "\n",
    "\n",
    "style = style.apply(highlight_min, axis=0, subset=[\"Total time (sec)\"])\n",
    "style = style.apply(highlight_max, axis=0, subset=[\"Words per second\"])\n",
    "\n",
    "style = style.set_properties(\n",
    "    subset=[\"Words per second\", \"Total time (sec)\"], **{\"text-align\": \"right\"}\n",
    ")\n",
    "# set decimal places\n",
    "style = style.format({\"Words per second\": \"{:.1f}\", \"Total time (sec)\": \"{:.2f}\"})\n",
    "\n",
    "style = style.hide(axis=\"index\")\n",
    "style = style.set_properties(subset=[\"Model\"], **{\"text-align\": \"left\"})\n",
    "style"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note here that the `da_dacy_{size}_trf-{version}` models from DaCy and the `da_core_news_{size}-{version}` models from spaCy are multi-task models so performs multiple tasks at once. This means that the inference speed is not directly comparable to the other models."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "```{bibliography} references.bib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dacy",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}