docs/tutorials/robustness.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating Robustness\n",
    "This tutorial walks through how to use `Augmenty`/`SpaCy` augmenters to evalutate robustness of any NLP pipeline. As an example we'll start out by evaluating SpaCy small and DaCy small on the test set of [DaNE](https://github.com/alexandrainst/danlp/blob/master/docs/docs/datasets.md#dane). DaNE is the Danish Dependency treebank tagged for part-of-speech tags, dependency relations and named entities. Lastly we will show how to use this framework on any other type of model using [DaNLP's BERT](https://github.com/alexandrainst/danlp/blob/master/docs/docs/tasks/ner.md#-bert-bert) as an example. \n",
    "\n",
    "Let us start of with installing the required packages and loading the models and dataset we wish to test on."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing packages\n",
    "\n",
    "To get started we will first need to install a few packages:\n",
    "\n",
    "```bash\n",
    "# install models\n",
    "pip install dacy\n",
    "python -m spacy download da_core_news_sm\n",
    "\n",
    "# install augmentation library\n",
    "pip install \"augmenty>1.3.0\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading models and data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "from dacy.datasets import dane\n",
    "\n",
    "# load the DaNE test set\n",
    "test = dane(splits=[\"test\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": [
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "import dacy\n",
    "import spacy\n",
    "\n",
    "# load models\n",
    "spacy_small = spacy.load(\"da_core_news_sm\")\n",
    "dacy_small = dacy.load(\"small\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimating performance\n",
    "Evaluating models already in the `SpaCy` framework is very straightforward. Simply call the `score` function on your nlp pipeline and choose which metrics you want to calculate performance for. `score` is a wrapper for `SpaCy.scorer.Scorer` that outputs a nicely formatted dataframe. `score` calculates performance for NER, POS, tokenization, and dependency parsing by default, which can be changed with the score_fn argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": [
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "from dacy.score import score\n",
    "\n",
    "spacy_baseline = score(test, apply_fn=spacy_small, score_fn=[\"ents\", \"pos\"])\n",
    "dacy_baseline = score(test, apply_fn=dacy_small, score_fn=[\"ents\", \"pos\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wall_time</th>\n",
       "      <th>ents_p</th>\n",
       "      <th>ents_r</th>\n",
       "      <th>ents_f</th>\n",
       "      <th>ents_per_type_LOC_p</th>\n",
       "      <th>ents_per_type_LOC_r</th>\n",
       "      <th>ents_per_type_LOC_f</th>\n",
       "      <th>ents_per_type_MISC_p</th>\n",
       "      <th>ents_per_type_MISC_r</th>\n",
       "      <th>ents_per_type_MISC_f</th>\n",
       "      <th>...</th>\n",
       "      <th>ents_per_type_PER_f</th>\n",
       "      <th>ents_per_type_ORG_p</th>\n",
       "      <th>ents_per_type_ORG_r</th>\n",
       "      <th>ents_per_type_ORG_f</th>\n",
       "      <th>ents_excl_MISC_ents_p</th>\n",
       "      <th>ents_excl_MISC_ents_r</th>\n",
       "      <th>ents_excl_MISC_ents_f</th>\n",
       "      <th>pos_acc</th>\n",
       "      <th>tag_acc</th>\n",
       "      <th>k</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.862225</td>\n",
       "      <td>0.685598</td>\n",
       "      <td>0.605735</td>\n",
       "      <td>0.643197</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.615385</td>\n",
       "      <td>0.628571</td>\n",
       "      <td>0.545455</td>\n",
       "      <td>0.584071</td>\n",
       "      <td>...</td>\n",
       "      <td>0.798898</td>\n",
       "      <td>0.677419</td>\n",
       "      <td>0.391304</td>\n",
       "      <td>0.496063</td>\n",
       "      <td>0.701031</td>\n",
       "      <td>0.622426</td>\n",
       "      <td>0.659394</td>\n",
       "      <td>0.947658</td>\n",
       "      <td>0.947658</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   wall_time    ents_p    ents_r    ents_f  ents_per_type_LOC_p  \\\n",
       "0   1.862225  0.685598  0.605735  0.643197             0.571429   \n",
       "\n",
       "   ents_per_type_LOC_r  ents_per_type_LOC_f  ents_per_type_MISC_p  \\\n",
       "0             0.666667             0.615385              0.628571   \n",
       "\n",
       "   ents_per_type_MISC_r  ents_per_type_MISC_f  ...  ents_per_type_PER_f  \\\n",
       "0              0.545455              0.584071  ...             0.798898   \n",
       "\n",
       "   ents_per_type_ORG_p  ents_per_type_ORG_r  ents_per_type_ORG_f  \\\n",
       "0             0.677419             0.391304             0.496063   \n",
       "\n",
       "   ents_excl_MISC_ents_p  ents_excl_MISC_ents_r  ents_excl_MISC_ents_f  \\\n",
       "0               0.701031               0.622426               0.659394   \n",
       "\n",
       "    pos_acc   tag_acc  k  \n",
       "0  0.947658  0.947658  0  \n",
       "\n",
       "[1 rows x 22 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spacy_baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wall_time</th>\n",
       "      <th>ents_p</th>\n",
       "      <th>ents_r</th>\n",
       "      <th>ents_f</th>\n",
       "      <th>ents_per_type_LOC_p</th>\n",
       "      <th>ents_per_type_LOC_r</th>\n",
       "      <th>ents_per_type_LOC_f</th>\n",
       "      <th>ents_per_type_MISC_p</th>\n",
       "      <th>ents_per_type_MISC_r</th>\n",
       "      <th>ents_per_type_MISC_f</th>\n",
       "      <th>...</th>\n",
       "      <th>ents_per_type_PER_f</th>\n",
       "      <th>ents_per_type_ORG_p</th>\n",
       "      <th>ents_per_type_ORG_r</th>\n",
       "      <th>ents_per_type_ORG_f</th>\n",
       "      <th>ents_excl_MISC_ents_p</th>\n",
       "      <th>ents_excl_MISC_ents_r</th>\n",
       "      <th>ents_excl_MISC_ents_f</th>\n",
       "      <th>pos_acc</th>\n",
       "      <th>tag_acc</th>\n",
       "      <th>k</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.808233</td>\n",
       "      <td>0.82852</td>\n",
       "      <td>0.822581</td>\n",
       "      <td>0.82554</td>\n",
       "      <td>0.767241</td>\n",
       "      <td>0.927083</td>\n",
       "      <td>0.839623</td>\n",
       "      <td>0.764706</td>\n",
       "      <td>0.752066</td>\n",
       "      <td>0.758333</td>\n",
       "      <td>...</td>\n",
       "      <td>0.920904</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.720497</td>\n",
       "      <td>0.75817</td>\n",
       "      <td>0.845977</td>\n",
       "      <td>0.842105</td>\n",
       "      <td>0.844037</td>\n",
       "      <td>0.978324</td>\n",
       "      <td>0.978972</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   wall_time   ents_p    ents_r   ents_f  ents_per_type_LOC_p  \\\n",
       "0   5.808233  0.82852  0.822581  0.82554             0.767241   \n",
       "\n",
       "   ents_per_type_LOC_r  ents_per_type_LOC_f  ents_per_type_MISC_p  \\\n",
       "0             0.927083             0.839623              0.764706   \n",
       "\n",
       "   ents_per_type_MISC_r  ents_per_type_MISC_f  ...  ents_per_type_PER_f  \\\n",
       "0              0.752066              0.758333  ...             0.920904   \n",
       "\n",
       "   ents_per_type_ORG_p  ents_per_type_ORG_r  ents_per_type_ORG_f  \\\n",
       "0                  0.8             0.720497              0.75817   \n",
       "\n",
       "   ents_excl_MISC_ents_p  ents_excl_MISC_ents_r  ents_excl_MISC_ents_f  \\\n",
       "0               0.845977               0.842105               0.844037   \n",
       "\n",
       "    pos_acc   tag_acc  k  \n",
       "0  0.978324  0.978972  0  \n",
       "\n",
       "[1 rows x 22 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dacy_baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Estimating robustness and biases\n",
    "To obtain performance estimates on augmented data, simply provide a list of augmenters as the `augmenters` argument. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from augmenty.span.entities import create_per_replace_augmenter_v1\n",
    "from dacy.datasets import female_names\n",
    "from spacy.training.augment import create_lower_casing_augmenter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": [
     "hide-output"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/au561649/Github/DaCy/src/dacy/datasets/names.py:53: FutureWarning: Not prepending group keys to the result index of transform-like apply. In the future, the group keys will be included in the index, regardless of whether the applied function returns a like-indexed object.\n",
      "To preserve the previous behavior, use\n",
      "\n",
      "\t>>> .groupby(..., group_keys=False)\n",
      "\n",
      "To adopt the future behavior and silence this warning, use \n",
      "\n",
      "\t>>> .groupby(..., group_keys=True)\n",
      "  names = names.groupby(level=0).apply(lambda x: x / float(x.sum()))\n"
     ]
    }
   ],
   "source": [
    "lower_aug = create_lower_casing_augmenter(level=1)\n",
    "female_name_dict = female_names()\n",
    "# Augmenter that replaces names with random Danish female names. Keep the format of the name as is (force_pattern_size=False)\n",
    "# but replace the name with one of the two defined patterns\n",
    "\n",
    "patterns = [\n",
    "    [\"firstname\"],\n",
    "    [\"firstname\", \"lastname\"],\n",
    "    [\"firstname\", \"firstname\", \"lastname\"],\n",
    "]\n",
    "female_aug = create_per_replace_augmenter_v1(female_name_dict, patterns, level=0.1)\n",
    "\n",
    "spacy_aug = score(\n",
    "    test,\n",
    "    apply_fn=spacy_small,\n",
    "    score_fn=[\"ents\", \"pos\"],\n",
    "    augmenters=[lower_aug, female_aug],\n",
    ")\n",
    "dacy_aug = score(\n",
    "    test,\n",
    "    apply_fn=dacy_small,\n",
    "    score_fn=[\"ents\", \"pos\"],\n",
    "    augmenters=[lower_aug, female_aug],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wall_time</th>\n",
       "      <th>ents_p</th>\n",
       "      <th>ents_r</th>\n",
       "      <th>ents_f</th>\n",
       "      <th>ents_per_type_LOC_p</th>\n",
       "      <th>ents_per_type_LOC_r</th>\n",
       "      <th>ents_per_type_LOC_f</th>\n",
       "      <th>ents_per_type_MISC_p</th>\n",
       "      <th>ents_per_type_MISC_r</th>\n",
       "      <th>ents_per_type_MISC_f</th>\n",
       "      <th>...</th>\n",
       "      <th>ents_per_type_PER_f</th>\n",
       "      <th>ents_per_type_ORG_p</th>\n",
       "      <th>ents_per_type_ORG_r</th>\n",
       "      <th>ents_per_type_ORG_f</th>\n",
       "      <th>ents_excl_MISC_ents_p</th>\n",
       "      <th>ents_excl_MISC_ents_r</th>\n",
       "      <th>ents_excl_MISC_ents_f</th>\n",
       "      <th>pos_acc</th>\n",
       "      <th>tag_acc</th>\n",
       "      <th>k</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.862225</td>\n",
       "      <td>0.685598</td>\n",
       "      <td>0.605735</td>\n",
       "      <td>0.643197</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.615385</td>\n",
       "      <td>0.628571</td>\n",
       "      <td>0.545455</td>\n",
       "      <td>0.584071</td>\n",
       "      <td>...</td>\n",
       "      <td>0.798898</td>\n",
       "      <td>0.677419</td>\n",
       "      <td>0.391304</td>\n",
       "      <td>0.496063</td>\n",
       "      <td>0.701031</td>\n",
       "      <td>0.622426</td>\n",
       "      <td>0.659394</td>\n",
       "      <td>0.947658</td>\n",
       "      <td>0.947658</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.873839</td>\n",
       "      <td>0.695652</td>\n",
       "      <td>0.286738</td>\n",
       "      <td>0.406091</td>\n",
       "      <td>0.687500</td>\n",
       "      <td>0.343750</td>\n",
       "      <td>0.458333</td>\n",
       "      <td>0.720000</td>\n",
       "      <td>0.446281</td>\n",
       "      <td>0.551020</td>\n",
       "      <td>...</td>\n",
       "      <td>0.412451</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.124224</td>\n",
       "      <td>0.209424</td>\n",
       "      <td>0.683871</td>\n",
       "      <td>0.242563</td>\n",
       "      <td>0.358108</td>\n",
       "      <td>0.922885</td>\n",
       "      <td>0.922885</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.699737</td>\n",
       "      <td>0.685598</td>\n",
       "      <td>0.605735</td>\n",
       "      <td>0.643197</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.615385</td>\n",
       "      <td>0.628571</td>\n",
       "      <td>0.545455</td>\n",
       "      <td>0.584071</td>\n",
       "      <td>...</td>\n",
       "      <td>0.798898</td>\n",
       "      <td>0.677419</td>\n",
       "      <td>0.391304</td>\n",
       "      <td>0.496063</td>\n",
       "      <td>0.701031</td>\n",
       "      <td>0.622426</td>\n",
       "      <td>0.659394</td>\n",
       "      <td>0.947658</td>\n",
       "      <td>0.947658</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   wall_time    ents_p    ents_r    ents_f  ents_per_type_LOC_p  \\\n",
       "0   1.862225  0.685598  0.605735  0.643197             0.571429   \n",
       "0   1.873839  0.695652  0.286738  0.406091             0.687500   \n",
       "0   1.699737  0.685598  0.605735  0.643197             0.571429   \n",
       "\n",
       "   ents_per_type_LOC_r  ents_per_type_LOC_f  ents_per_type_MISC_p  \\\n",
       "0             0.666667             0.615385              0.628571   \n",
       "0             0.343750             0.458333              0.720000   \n",
       "0             0.666667             0.615385              0.628571   \n",
       "\n",
       "   ents_per_type_MISC_r  ents_per_type_MISC_f  ...  ents_per_type_PER_f  \\\n",
       "0              0.545455              0.584071  ...             0.798898   \n",
       "0              0.446281              0.551020  ...             0.412451   \n",
       "0              0.545455              0.584071  ...             0.798898   \n",
       "\n",
       "   ents_per_type_ORG_p  ents_per_type_ORG_r  ents_per_type_ORG_f  \\\n",
       "0             0.677419             0.391304             0.496063   \n",
       "0             0.666667             0.124224             0.209424   \n",
       "0             0.677419             0.391304             0.496063   \n",
       "\n",
       "   ents_excl_MISC_ents_p  ents_excl_MISC_ents_r  ents_excl_MISC_ents_f  \\\n",
       "0               0.701031               0.622426               0.659394   \n",
       "0               0.683871               0.242563               0.358108   \n",
       "0               0.701031               0.622426               0.659394   \n",
       "\n",
       "    pos_acc   tag_acc  k  \n",
       "0  0.947658  0.947658  0  \n",
       "0  0.922885  0.922885  0  \n",
       "0  0.947658  0.947658  0  \n",
       "\n",
       "[3 rows x 22 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "pd.concat([spacy_baseline, spacy_aug])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wall_time</th>\n",
       "      <th>ents_p</th>\n",
       "      <th>ents_r</th>\n",
       "      <th>ents_f</th>\n",
       "      <th>ents_per_type_LOC_p</th>\n",
       "      <th>ents_per_type_LOC_r</th>\n",
       "      <th>ents_per_type_LOC_f</th>\n",
       "      <th>ents_per_type_MISC_p</th>\n",
       "      <th>ents_per_type_MISC_r</th>\n",
       "      <th>ents_per_type_MISC_f</th>\n",
       "      <th>...</th>\n",
       "      <th>ents_per_type_PER_f</th>\n",
       "      <th>ents_per_type_ORG_p</th>\n",
       "      <th>ents_per_type_ORG_r</th>\n",
       "      <th>ents_per_type_ORG_f</th>\n",
       "      <th>ents_excl_MISC_ents_p</th>\n",
       "      <th>ents_excl_MISC_ents_r</th>\n",
       "      <th>ents_excl_MISC_ents_f</th>\n",
       "      <th>pos_acc</th>\n",
       "      <th>tag_acc</th>\n",
       "      <th>k</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.808233</td>\n",
       "      <td>0.828520</td>\n",
       "      <td>0.822581</td>\n",
       "      <td>0.82554</td>\n",
       "      <td>0.767241</td>\n",
       "      <td>0.927083</td>\n",
       "      <td>0.839623</td>\n",
       "      <td>0.764706</td>\n",
       "      <td>0.752066</td>\n",
       "      <td>0.758333</td>\n",
       "      <td>...</td>\n",
       "      <td>0.920904</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.720497</td>\n",
       "      <td>0.758170</td>\n",
       "      <td>0.845977</td>\n",
       "      <td>0.842105</td>\n",
       "      <td>0.844037</td>\n",
       "      <td>0.978324</td>\n",
       "      <td>0.978972</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.315962</td>\n",
       "      <td>0.607143</td>\n",
       "      <td>0.213262</td>\n",
       "      <td>0.31565</td>\n",
       "      <td>0.677419</td>\n",
       "      <td>0.218750</td>\n",
       "      <td>0.330709</td>\n",
       "      <td>0.490566</td>\n",
       "      <td>0.429752</td>\n",
       "      <td>0.458150</td>\n",
       "      <td>...</td>\n",
       "      <td>0.245283</td>\n",
       "      <td>0.740741</td>\n",
       "      <td>0.124224</td>\n",
       "      <td>0.212766</td>\n",
       "      <td>0.744444</td>\n",
       "      <td>0.153318</td>\n",
       "      <td>0.254269</td>\n",
       "      <td>0.933873</td>\n",
       "      <td>0.931722</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.710288</td>\n",
       "      <td>0.828520</td>\n",
       "      <td>0.822581</td>\n",
       "      <td>0.82554</td>\n",
       "      <td>0.767241</td>\n",
       "      <td>0.927083</td>\n",
       "      <td>0.839623</td>\n",
       "      <td>0.764706</td>\n",
       "      <td>0.752066</td>\n",
       "      <td>0.758333</td>\n",
       "      <td>...</td>\n",
       "      <td>0.920904</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.720497</td>\n",
       "      <td>0.758170</td>\n",
       "      <td>0.845977</td>\n",
       "      <td>0.842105</td>\n",
       "      <td>0.844037</td>\n",
       "      <td>0.978324</td>\n",
       "      <td>0.978972</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   wall_time    ents_p    ents_r   ents_f  ents_per_type_LOC_p  \\\n",
       "0   5.808233  0.828520  0.822581  0.82554             0.767241   \n",
       "0   5.315962  0.607143  0.213262  0.31565             0.677419   \n",
       "0   5.710288  0.828520  0.822581  0.82554             0.767241   \n",
       "\n",
       "   ents_per_type_LOC_r  ents_per_type_LOC_f  ents_per_type_MISC_p  \\\n",
       "0             0.927083             0.839623              0.764706   \n",
       "0             0.218750             0.330709              0.490566   \n",
       "0             0.927083             0.839623              0.764706   \n",
       "\n",
       "   ents_per_type_MISC_r  ents_per_type_MISC_f  ...  ents_per_type_PER_f  \\\n",
       "0              0.752066              0.758333  ...             0.920904   \n",
       "0              0.429752              0.458150  ...             0.245283   \n",
       "0              0.752066              0.758333  ...             0.920904   \n",
       "\n",
       "   ents_per_type_ORG_p  ents_per_type_ORG_r  ents_per_type_ORG_f  \\\n",
       "0             0.800000             0.720497             0.758170   \n",
       "0             0.740741             0.124224             0.212766   \n",
       "0             0.800000             0.720497             0.758170   \n",
       "\n",
       "   ents_excl_MISC_ents_p  ents_excl_MISC_ents_r  ents_excl_MISC_ents_f  \\\n",
       "0               0.845977               0.842105               0.844037   \n",
       "0               0.744444               0.153318               0.254269   \n",
       "0               0.845977               0.842105               0.844037   \n",
       "\n",
       "    pos_acc   tag_acc  k  \n",
       "0  0.978324  0.978972  0  \n",
       "0  0.933873  0.931722  0  \n",
       "0  0.978324  0.978972  0  \n",
       "\n",
       "[3 rows x 22 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([dacy_baseline, dacy_aug])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the second row, we see that `SpaCy small` is very vulnerable to lower casing as NER recall drops from 0.66 to 0.38. `DaCy small` is slightly more robust lower casing, but still suffers. Changing names also leads to a drop in performance for both models. \n",
    "\n",
    "To better estimate the effect of stochastic augmenters such as those changing names or adding keystroke errors we can use the `k` argument in `score` to run the augmenter multiple times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from augmenty.character.replace import create_keystroke_error_augmenter_v1\n",
    "\n",
    "key_05_aug = create_keystroke_error_augmenter_v1(level=0.5, keyboard=\"da_qwerty_v1\")\n",
    "\n",
    "spacy_key = score(\n",
    "    test, apply_fn=spacy_small, score_fn=[\"ents\", \"pos\"], augmenters=[key_05_aug], k=5\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wall_time</th>\n",
       "      <th>ents_p</th>\n",
       "      <th>ents_r</th>\n",
       "      <th>ents_f</th>\n",
       "      <th>ents_per_type_LOC_p</th>\n",
       "      <th>ents_per_type_LOC_r</th>\n",
       "      <th>ents_per_type_LOC_f</th>\n",
       "      <th>ents_per_type_MISC_p</th>\n",
       "      <th>ents_per_type_MISC_r</th>\n",
       "      <th>ents_per_type_MISC_f</th>\n",
       "      <th>...</th>\n",
       "      <th>ents_per_type_PER_f</th>\n",
       "      <th>ents_per_type_ORG_p</th>\n",
       "      <th>ents_per_type_ORG_r</th>\n",
       "      <th>ents_per_type_ORG_f</th>\n",
       "      <th>ents_excl_MISC_ents_p</th>\n",
       "      <th>ents_excl_MISC_ents_r</th>\n",
       "      <th>ents_excl_MISC_ents_f</th>\n",
       "      <th>pos_acc</th>\n",
       "      <th>tag_acc</th>\n",
       "      <th>k</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.173135</td>\n",
       "      <td>0.096026</td>\n",
       "      <td>0.103943</td>\n",
       "      <td>0.099828</td>\n",
       "      <td>0.109890</td>\n",
       "      <td>0.104167</td>\n",
       "      <td>0.106952</td>\n",
       "      <td>0.060811</td>\n",
       "      <td>0.074380</td>\n",
       "      <td>0.066914</td>\n",
       "      <td>...</td>\n",
       "      <td>0.141732</td>\n",
       "      <td>0.073171</td>\n",
       "      <td>0.074534</td>\n",
       "      <td>0.073846</td>\n",
       "      <td>0.107456</td>\n",
       "      <td>0.112128</td>\n",
       "      <td>0.109742</td>\n",
       "      <td>0.326630</td>\n",
       "      <td>0.326630</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.117777</td>\n",
       "      <td>0.116949</td>\n",
       "      <td>0.123656</td>\n",
       "      <td>0.120209</td>\n",
       "      <td>0.145631</td>\n",
       "      <td>0.156250</td>\n",
       "      <td>0.150754</td>\n",
       "      <td>0.073171</td>\n",
       "      <td>0.074380</td>\n",
       "      <td>0.073770</td>\n",
       "      <td>...</td>\n",
       "      <td>0.181818</td>\n",
       "      <td>0.066298</td>\n",
       "      <td>0.074534</td>\n",
       "      <td>0.070175</td>\n",
       "      <td>0.128480</td>\n",
       "      <td>0.137300</td>\n",
       "      <td>0.132743</td>\n",
       "      <td>0.319308</td>\n",
       "      <td>0.319308</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.094923</td>\n",
       "      <td>0.097603</td>\n",
       "      <td>0.102151</td>\n",
       "      <td>0.099825</td>\n",
       "      <td>0.060000</td>\n",
       "      <td>0.062500</td>\n",
       "      <td>0.061224</td>\n",
       "      <td>0.063830</td>\n",
       "      <td>0.074380</td>\n",
       "      <td>0.068702</td>\n",
       "      <td>...</td>\n",
       "      <td>0.153439</td>\n",
       "      <td>0.089655</td>\n",
       "      <td>0.080745</td>\n",
       "      <td>0.084967</td>\n",
       "      <td>0.108352</td>\n",
       "      <td>0.109840</td>\n",
       "      <td>0.109091</td>\n",
       "      <td>0.321187</td>\n",
       "      <td>0.321187</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2.070100</td>\n",
       "      <td>0.123539</td>\n",
       "      <td>0.132616</td>\n",
       "      <td>0.127917</td>\n",
       "      <td>0.134831</td>\n",
       "      <td>0.125000</td>\n",
       "      <td>0.129730</td>\n",
       "      <td>0.080645</td>\n",
       "      <td>0.082645</td>\n",
       "      <td>0.081633</td>\n",
       "      <td>...</td>\n",
       "      <td>0.153465</td>\n",
       "      <td>0.129630</td>\n",
       "      <td>0.130435</td>\n",
       "      <td>0.130031</td>\n",
       "      <td>0.134737</td>\n",
       "      <td>0.146453</td>\n",
       "      <td>0.140351</td>\n",
       "      <td>0.313382</td>\n",
       "      <td>0.313382</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.069810</td>\n",
       "      <td>0.099831</td>\n",
       "      <td>0.105735</td>\n",
       "      <td>0.102698</td>\n",
       "      <td>0.104762</td>\n",
       "      <td>0.114583</td>\n",
       "      <td>0.109453</td>\n",
       "      <td>0.033613</td>\n",
       "      <td>0.033058</td>\n",
       "      <td>0.033333</td>\n",
       "      <td>...</td>\n",
       "      <td>0.172973</td>\n",
       "      <td>0.067797</td>\n",
       "      <td>0.074534</td>\n",
       "      <td>0.071006</td>\n",
       "      <td>0.116525</td>\n",
       "      <td>0.125858</td>\n",
       "      <td>0.121012</td>\n",
       "      <td>0.315617</td>\n",
       "      <td>0.315617</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   wall_time    ents_p    ents_r    ents_f  ents_per_type_LOC_p  \\\n",
       "0   2.173135  0.096026  0.103943  0.099828             0.109890   \n",
       "1   2.117777  0.116949  0.123656  0.120209             0.145631   \n",
       "2   2.094923  0.097603  0.102151  0.099825             0.060000   \n",
       "3   2.070100  0.123539  0.132616  0.127917             0.134831   \n",
       "4   2.069810  0.099831  0.105735  0.102698             0.104762   \n",
       "\n",
       "   ents_per_type_LOC_r  ents_per_type_LOC_f  ents_per_type_MISC_p  \\\n",
       "0             0.104167             0.106952              0.060811   \n",
       "1             0.156250             0.150754              0.073171   \n",
       "2             0.062500             0.061224              0.063830   \n",
       "3             0.125000             0.129730              0.080645   \n",
       "4             0.114583             0.109453              0.033613   \n",
       "\n",
       "   ents_per_type_MISC_r  ents_per_type_MISC_f  ...  ents_per_type_PER_f  \\\n",
       "0              0.074380              0.066914  ...             0.141732   \n",
       "1              0.074380              0.073770  ...             0.181818   \n",
       "2              0.074380              0.068702  ...             0.153439   \n",
       "3              0.082645              0.081633  ...             0.153465   \n",
       "4              0.033058              0.033333  ...             0.172973   \n",
       "\n",
       "   ents_per_type_ORG_p  ents_per_type_ORG_r  ents_per_type_ORG_f  \\\n",
       "0             0.073171             0.074534             0.073846   \n",
       "1             0.066298             0.074534             0.070175   \n",
       "2             0.089655             0.080745             0.084967   \n",
       "3             0.129630             0.130435             0.130031   \n",
       "4             0.067797             0.074534             0.071006   \n",
       "\n",
       "   ents_excl_MISC_ents_p  ents_excl_MISC_ents_r  ents_excl_MISC_ents_f  \\\n",
       "0               0.107456               0.112128               0.109742   \n",
       "1               0.128480               0.137300               0.132743   \n",
       "2               0.108352               0.109840               0.109091   \n",
       "3               0.134737               0.146453               0.140351   \n",
       "4               0.116525               0.125858               0.121012   \n",
       "\n",
       "    pos_acc   tag_acc  k  \n",
       "0  0.326630  0.326630  0  \n",
       "1  0.319308  0.319308  1  \n",
       "2  0.321187  0.321187  2  \n",
       "3  0.313382  0.313382  3  \n",
       "4  0.315617  0.315617  4  \n",
       "\n",
       "[5 rows x 22 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spacy_key"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this manner, evaluating performance on augmented data for SpaCy pipelines is as easy as defining the augmenters and calling a single function. In the `dacy_paper_replication.py` script you can find the exact script used to evaluate the robustness of Danish NLP models in the [DaCy paper]().\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.7 ('.venv': poetry)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "41a0a94681c1d0d7a5ae694864ee715c089937d98d908faa5b8f75504686895e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}