notebooks/ner_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transformers Interpret NER Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\pablo\\DFKI\\transformers-interpret\\.venv\\lib\\site-packages\\tqdm\\auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModelForTokenClassification, AutoTokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import NER model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_PATH = 'dslim/bert-base-NER'\n",
    "model = AutoModelForTokenClassification.from_pretrained(MODEL_PATH)\n",
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can explore the possible entity tags that this model classifies each token into:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'O',\n",
       " 1: 'B-MISC',\n",
       " 2: 'I-MISC',\n",
       " 3: 'B-PER',\n",
       " 4: 'I-PER',\n",
       " 5: 'B-ORG',\n",
       " 6: 'I-ORG',\n",
       " 7: 'B-LOC',\n",
       " 8: 'I-LOC'}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.config.id2label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Import the **TokenClassificationExplainer** from the *transformers interpret* package. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers_interpret import TokenClassificationExplainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an instance of the explainer providing the model and the tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_explainer = TokenClassificationExplainer(model=model, tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call the explainer to generate the attributions. In the NER task, for each token of the sentence, the predicted class is considered as the target and the attributions of the rest of the tokens are calculated. This can take up to one or two minutes. In the following sections we will see how to limit the computation of the attributions to only certain tokens or only certain NER labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_text = \"We visited Paris last weekend, where Emmanuel Macron lives.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "attributions = ner_explainer(sample_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The explainer returns a dictionary with the predicted label and the attribtuion scores for every token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'[CLS]': {'label': 'O',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', -0.2197283334110176),\n",
       "   ('visited', -0.006874791837538396),\n",
       "   ('paris', 0.18982619777915935),\n",
       "   ('last', -0.21827055690979064),\n",
       "   ('weekend', 0.3766208981865305),\n",
       "   (',', 0.4892531296167725),\n",
       "   ('where', 0.05182857620198211),\n",
       "   ('emmanuel', 0.03630020660142918),\n",
       "   ('macro', 0.1923832558634223),\n",
       "   ('##n', -0.06223924004251359),\n",
       "   ('lives', 0.27938115094347166),\n",
       "   ('.', -0.6031927350485571),\n",
       "   ('[SEP]', 0.0)]},\n",
       " 'we': {'label': 'O',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', 0.8284940046000231),\n",
       "   ('visited', 0.10991536556232688),\n",
       "   ('paris', -0.1125062256706944),\n",
       "   ('last', -0.15386756933829704),\n",
       "   ('weekend', -0.005880473809398968),\n",
       "   (',', 0.004343130287280704),\n",
       "   ('where', -0.28342447030227175),\n",
       "   ('emmanuel', -0.41207122570713284),\n",
       "   ('macro', -0.039487737627438534),\n",
       "   ('##n', 0.052480672360112925),\n",
       "   ('lives', 0.10229347071494145),\n",
       "   ('.', -0.014845616401730032),\n",
       "   ('[SEP]', 0.0)]},\n",
       " 'visited': {'label': 'O',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', 0.1666566472443391),\n",
       "   ('visited', 0.7266303424715478),\n",
       "   ('paris', 0.05773183964903933),\n",
       "   ('last', -0.13951299816817211),\n",
       "   ('weekend', 0.3951742413952667),\n",
       "   (',', 0.07842998919432928),\n",
       "   ('where', 0.3093996487484364),\n",
       "   ('emmanuel', -0.2383082011289848),\n",
       "   ('macro', 0.04737151678563919),\n",
       "   ('##n', -0.2811653232241067),\n",
       "   ('lives', 0.10124112882151497),\n",
       "   ('.', -0.1227045800781033),\n",
       "   ('[SEP]', 0.0)]},\n",
       " 'paris': {'label': 'B-LOC',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', -0.014352325471387907),\n",
       "   ('visited', 0.32915222186559123),\n",
       "   ('paris', 0.9086791784795596),\n",
       "   ('last', 0.15181203147624034),\n",
       "   ('weekend', 0.14400210630677038),\n",
       "   (',', 0.01899744327012935),\n",
       "   ('where', -0.039402005463239465),\n",
       "   ('emmanuel', 0.061095284002642025),\n",
       "   ('macro', 0.004192922551105228),\n",
       "   ('##n', 0.09446355513057757),\n",
       "   ('lives', -0.028724312616455003),\n",
       "   ('.', 0.08099007392937585),\n",
       "   ('[SEP]', 0.0)]},\n",
       " 'last': {'label': 'O',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', -0.11692975216720262),\n",
       "   ('visited', 0.07934971102914033),\n",
       "   ('paris', 0.23730594142324693),\n",
       "   ('last', 0.25879242279373943),\n",
       "   ('weekend', 0.6341961014051442),\n",
       "   (',', 0.46116653207513564),\n",
       "   ('where', 0.23399203207207084),\n",
       "   ('emmanuel', -0.12286323001625263),\n",
       "   ('macro', 0.0685506846406235),\n",
       "   ('##n', -0.23253843626436685),\n",
       "   ('lives', 0.19201109918764547),\n",
       "   ('.', -0.27636015607233194),\n",
       "   ('[SEP]', 0.0)]},\n",
       " 'weekend': {'label': 'O',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', -0.033792562210997204),\n",
       "   ('visited', 0.0492260983296448),\n",
       "   ('paris', 0.1556769070246399),\n",
       "   ('last', 0.05740668465740179),\n",
       "   ('weekend', 0.8089283173631592),\n",
       "   (',', 0.24407629582539445),\n",
       "   ('where', 0.33608956458664657),\n",
       "   ('emmanuel', -0.1346558916035443),\n",
       "   ('macro', 0.08464870477107096),\n",
       "   ('##n', -0.17210911438312998),\n",
       "   ('lives', 0.2523869831788622),\n",
       "   ('.', -0.15294189100317238),\n",
       "   ('[SEP]', 0.0)]},\n",
       " ',': {'label': 'O',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', -0.017598309064906406),\n",
       "   ('visited', -0.008882755160827888),\n",
       "   ('paris', 0.0121745960915987),\n",
       "   ('last', -0.0763950896543035),\n",
       "   ('weekend', 0.4423613236708174),\n",
       "   (',', 0.8660889264305044),\n",
       "   ('where', 0.09760917296763205),\n",
       "   ('emmanuel', -0.08848359284827705),\n",
       "   ('macro', 0.052046994930197744),\n",
       "   ('##n', -0.11498731237640712),\n",
       "   ('lives', 0.06940761462669097),\n",
       "   ('.', -0.0986311867131256),\n",
       "   ('[SEP]', 0.0)]},\n",
       " 'where': {'label': 'O',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', 0.029070560904644813),\n",
       "   ('visited', 0.037654296474629534),\n",
       "   ('paris', -0.007230754595621449),\n",
       "   ('last', 0.0036858934423390282),\n",
       "   ('weekend', 0.3736116874430494),\n",
       "   (',', 0.06315325423706226),\n",
       "   ('where', 0.8581912566846205),\n",
       "   ('emmanuel', -0.22937229434021364),\n",
       "   ('macro', 0.014179009509378435),\n",
       "   ('##n', -0.15934323130471037),\n",
       "   ('lives', 0.14802897859675135),\n",
       "   ('.', -0.13224745619448916),\n",
       "   ('[SEP]', 0.0)]},\n",
       " 'emmanuel': {'label': 'B-PER',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', -0.006933030636686712),\n",
       "   ('visited', 0.10396962390436904),\n",
       "   ('paris', 0.14540758744233165),\n",
       "   ('last', 0.08024018944451371),\n",
       "   ('weekend', 0.10687970996804418),\n",
       "   (',', 0.1793198466387937),\n",
       "   ('where', 0.3436407835483767),\n",
       "   ('emmanuel', 0.8774892642652167),\n",
       "   ('macro', 0.03559399361048316),\n",
       "   ('##n', 0.1516315604785551),\n",
       "   ('lives', 0.07056441327498127),\n",
       "   ('.', -0.025820924624605487),\n",
       "   ('[SEP]', 0.0)]},\n",
       " 'macro': {'label': 'I-PER',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', 0.05578067326280157),\n",
       "   ('visited', 0.00857021283406586),\n",
       "   ('paris', 0.16559056506114297),\n",
       "   ('last', 0.08285256685903823),\n",
       "   ('weekend', 0.10468727443796395),\n",
       "   (',', 0.09949509071515888),\n",
       "   ('where', 0.3642458274356929),\n",
       "   ('emmanuel', 0.7449335213978788),\n",
       "   ('macro', 0.3794625659183485),\n",
       "   ('##n', -0.2599031433800762),\n",
       "   ('lives', 0.20563450682196147),\n",
       "   ('.', -0.015607017319486929),\n",
       "   ('[SEP]', 0.0)]},\n",
       " '##n': {'label': 'I-PER',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', 0.025194121717285252),\n",
       "   ('visited', -0.007415022865239864),\n",
       "   ('paris', 0.09478357303107598),\n",
       "   ('last', 0.06927939834474463),\n",
       "   ('weekend', 0.0672008033510708),\n",
       "   (',', 0.08316907214363504),\n",
       "   ('where', 0.3784915854680165),\n",
       "   ('emmanuel', 0.7729352621546081),\n",
       "   ('macro', 0.4148652759139777),\n",
       "   ('##n', -0.20853534512145033),\n",
       "   ('lives', 0.09445057087678274),\n",
       "   ('.', -0.094274985907366),\n",
       "   ('[SEP]', 0.0)]},\n",
       " 'lives': {'label': 'O',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', -0.03344316679723656),\n",
       "   ('visited', -0.04610827849965715),\n",
       "   ('paris', -0.0017249319199365341),\n",
       "   ('last', -0.009504720903982286),\n",
       "   ('weekend', 0.4261308360424482),\n",
       "   (',', 0.4355855229672087),\n",
       "   ('where', 0.4006168996423131),\n",
       "   ('emmanuel', -0.19770155635214395),\n",
       "   ('macro', 0.053362289364445085),\n",
       "   ('##n', -0.1394587404971162),\n",
       "   ('lives', 0.5102111564716015),\n",
       "   ('.', -0.378349738754372),\n",
       "   ('[SEP]', 0.0)]},\n",
       " '.': {'label': 'O',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', -0.05064126284595578),\n",
       "   ('visited', -0.03245005742738379),\n",
       "   ('paris', -0.05286086042866884),\n",
       "   ('last', -0.10029364262979922),\n",
       "   ('weekend', 0.46371302372293943),\n",
       "   (',', 0.5322401130576084),\n",
       "   ('where', 0.397812397904332),\n",
       "   ('emmanuel', -0.2819049109631611),\n",
       "   ('macro', 0.0872671038459618),\n",
       "   ('##n', -0.19308443634276562),\n",
       "   ('lives', 0.42161496104928925),\n",
       "   ('.', 0.15760285381827557),\n",
       "   ('[SEP]', 0.0)]},\n",
       " '[SEP]': {'label': 'B-LOC',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('we', -0.3694351403796742),\n",
       "   ('visited', 0.1699038407402483),\n",
       "   ('paris', 0.5461587414992369),\n",
       "   ('last', 0.0037948102770307517),\n",
       "   ('weekend', 0.1628100955702496),\n",
       "   (',', 0.4513093410909263),\n",
       "   ('where', -0.09577409464161038),\n",
       "   ('emmanuel', 0.48499459835388914),\n",
       "   ('macro', -0.13528905587653023),\n",
       "   ('##n', 0.14362969934754344),\n",
       "   ('lives', -0.05758007024257254),\n",
       "   ('.', -0.13970977266152554),\n",
       "   ('[SEP]', 0.0)]}}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "attributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note the alogirthm used to calcualte attributions are Layer Integreated Gradients to read more about them click [here](https://captum.ai/docs/algorithms)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Visualizing explanations\n",
    "\n",
    "With a single call to the `visualize()` method we get a nice inline display of what inputs are causing the activations to fire that led to classifying each of the tokens into a particular class. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table width: 100%><div style=\"border-top: 1px solid; margin-top: 5px;             padding-top: 5px; display: inline-block\"><b>Legend: </b><span style=\"display: inline-block; width: 10px; height: 10px;                 border: 1px solid; background-color:                 hsl(0, 75%, 60%)\"></span> Negative  <span style=\"display: inline-block; width: 10px; height: 10px;                 border: 1px solid; background-color:                 hsl(0, 75%, 100%)\"></span> Neutral  <span style=\"display: inline-block; width: 10px; height: 10px;                 border: 1px solid; background-color:                 hsl(120, 75%, 50%)\"></span> Positive  </div><tr><th>True Label</th><th>Predicted Label</th><th>Attribution Label</th><th>Attribution Score</th><th>Word Importance</th><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>[CLS]</b></text></td><td><text style=\"padding-right:2em\"><b>1.13</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 90%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 77%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 78%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 80%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 85%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>We</b></text></td><td><text style=\"padding-right:2em\"><b>0.16</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 56%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(0, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(0, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(0, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>visited</b></text></td><td><text style=\"padding-right:2em\"><b>1.15</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 58%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 87%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 84%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>B-LOC</b></text></td><td><text style=\"padding-right:2em\"><b>B-LOC (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>Paris</b></text></td><td><text style=\"padding-right:2em\"><b>1.03</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 51%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>last</b></text></td><td><text style=\"padding-right:2em\"><b>1.46</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 53%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>weekend</b></text></td><td><text style=\"padding-right:2em\"><b>1.97</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 60%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 76%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>,</b></text></td><td><text style=\"padding-right:2em\"><b>2.23</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 91%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 70%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 75%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>where</b></text></td><td><text style=\"padding-right:2em\"><b>1.94</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 89%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 77%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 62%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 88%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>B-PER</b></text></td><td><text style=\"padding-right:2em\"><b>B-PER (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>Emmanuel</b></text></td><td><text style=\"padding-right:2em\"><b>1.17</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 80%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 55%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(0, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>I-PER</b></text></td><td><text style=\"padding-right:2em\"><b>I-PER (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>Mac</b></text></td><td><text style=\"padding-right:2em\"><b>1.27</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 53%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>I-PER</b></text></td><td><text style=\"padding-right:2em\"><b>I-PER (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>##ron</b></text></td><td><text style=\"padding-right:2em\"><b>1.57</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 79%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 62%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 81%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>lives</b></text></td><td><text style=\"padding-right:2em\"><b>1.00</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 91%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(0, 75%, 91%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 54%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>.</b></text></td><td><text style=\"padding-right:2em\"><b>1.96</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 85%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 85%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(0, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 85%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 82%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 65%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (0.99)</b></text></td><td><text style=\"padding-right:2em\"><b>[SEP]</b></text></td><td><text style=\"padding-right:2em\"><b>-1.58</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(0, 75%, 87%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 85%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 91%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 91%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(0, 75%, 76%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(0, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 91%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "html = ner_explainer.visualize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ignore indexes\n",
    "\n",
    "To save computation time, we can indicate a list of token indexes that we want to ignore. The explainer will not compute explanations for these tokens, although attributions of these tokens will be calculated to explain the predictions over other tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "attributions_2 = ner_explainer(sample_text, ignored_indexes=[0, 1, 2, 11, 12, 13, 14])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When calling *visualize()*, we can see how the tokens corresponding to the ignored indexes do not appear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table width: 100%><div style=\"border-top: 1px solid; margin-top: 5px;             padding-top: 5px; display: inline-block\"><b>Legend: </b><span style=\"display: inline-block; width: 10px; height: 10px;                 border: 1px solid; background-color:                 hsl(0, 75%, 60%)\"></span> Negative  <span style=\"display: inline-block; width: 10px; height: 10px;                 border: 1px solid; background-color:                 hsl(0, 75%, 100%)\"></span> Neutral  <span style=\"display: inline-block; width: 10px; height: 10px;                 border: 1px solid; background-color:                 hsl(120, 75%, 50%)\"></span> Positive  </div><tr><th>True Label</th><th>Predicted Label</th><th>Attribution Label</th><th>Attribution Score</th><th>Word Importance</th><tr><td><text style=\"padding-right:2em\"><b>B-LOC</b></text></td><td><text style=\"padding-right:2em\"><b>B-LOC (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>Paris</b></text></td><td><text style=\"padding-right:2em\"><b>1.03</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 51%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>last</b></text></td><td><text style=\"padding-right:2em\"><b>1.46</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 53%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>weekend</b></text></td><td><text style=\"padding-right:2em\"><b>1.97</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 60%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 76%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>,</b></text></td><td><text style=\"padding-right:2em\"><b>2.23</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 91%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 70%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 75%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>where</b></text></td><td><text style=\"padding-right:2em\"><b>1.94</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 89%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 77%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 62%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 88%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>B-PER</b></text></td><td><text style=\"padding-right:2em\"><b>B-PER (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>Emmanuel</b></text></td><td><text style=\"padding-right:2em\"><b>1.17</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 80%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 55%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(0, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>I-PER</b></text></td><td><text style=\"padding-right:2em\"><b>I-PER (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>Mac</b></text></td><td><text style=\"padding-right:2em\"><b>1.27</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 53%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>I-PER</b></text></td><td><text style=\"padding-right:2em\"><b>I-PER (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>##ron</b></text></td><td><text style=\"padding-right:2em\"><b>1.57</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 79%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 62%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 81%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "html = ner_explainer.visualize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ignore labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a similar way, we can also tell the explainer to ignore certain labels, e.g. we might not be interested in seeing the explanations of those tokens that are classified as *'O'*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "attributions_3 = ner_explainer(sample_text, ignored_labels=['O'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Paris': {'label': 'B-LOC',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('We', -0.026176158026416003),\n",
       "   ('visited', 0.1289424214695528),\n",
       "   ('Paris', 0.9820248557100411),\n",
       "   ('last', -0.13107116600328042),\n",
       "   ('weekend', 0.010904143358261256),\n",
       "   (',', 0.011178508562829472),\n",
       "   ('where', 0.023509729193194558),\n",
       "   ('Emmanuel', 0.009830564279598433),\n",
       "   ('Mac', 0.00034993661827931806),\n",
       "   ('##ron', 0.014679480802688051),\n",
       "   ('lives', 0.00523146911109651),\n",
       "   ('.', 9.093164937093907e-05),\n",
       "   ('[SEP]', 0.0)]},\n",
       " 'Emmanuel': {'label': 'B-PER',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('We', -0.02795400955500282),\n",
       "   ('visited', -0.019536101884928644),\n",
       "   ('Paris', 0.016961229547365956),\n",
       "   ('last', 0.02993168457599209),\n",
       "   ('weekend', -0.0005594575778221615),\n",
       "   (',', -0.0028076008557282677),\n",
       "   ('where', 0.4016979194311053),\n",
       "   ('Emmanuel', 0.9076506398659525),\n",
       "   ('Mac', -0.09408993595706144),\n",
       "   ('##ron', -0.04634243847125355),\n",
       "   ('lives', 0.02766440864687646),\n",
       "   ('.', -0.026236800991730185),\n",
       "   ('[SEP]', 0.0)]},\n",
       " 'Mac': {'label': 'I-PER',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('We', -0.023946141938553923),\n",
       "   ('visited', -0.02314137524841115),\n",
       "   ('Paris', 0.016059605323833003),\n",
       "   ('last', 0.0425715380070115),\n",
       "   ('weekend', 0.008064733430597637),\n",
       "   (',', 0.03339827130222196),\n",
       "   ('where', 0.2888303588807151),\n",
       "   ('Emmanuel', 0.954108208748422),\n",
       "   ('Mac', 0.009529915023771622),\n",
       "   ('##ron', 0.0018587632542397066),\n",
       "   ('lives', 0.0063623171046915915),\n",
       "   ('.', -0.041953834660571884),\n",
       "   ('[SEP]', 0.0)]},\n",
       " '##ron': {'label': 'I-PER',\n",
       "  'attribution_scores': [('[CLS]', 0.0),\n",
       "   ('We', -0.13524730066867285),\n",
       "   ('visited', 0.03632829648791544),\n",
       "   ('Paris', 0.16271952183132127),\n",
       "   ('last', 0.004163465393724605),\n",
       "   ('weekend', -0.014037349925373541),\n",
       "   (',', 0.06303365825682841),\n",
       "   ('where', 0.4202547928670278),\n",
       "   ('Emmanuel', 0.7670349435767041),\n",
       "   ('Mac', 0.38679457512942717),\n",
       "   ('##ron', -0.039683691940599335),\n",
       "   ('lives', 0.08338745816005198),\n",
       "   ('.', -0.16318135160442798),\n",
       "   ('[SEP]', 0.0)]}}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "attributions_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table width: 100%><div style=\"border-top: 1px solid; margin-top: 5px;             padding-top: 5px; display: inline-block\"><b>Legend: </b><span style=\"display: inline-block; width: 10px; height: 10px;                 border: 1px solid; background-color:                 hsl(0, 75%, 60%)\"></span> Negative  <span style=\"display: inline-block; width: 10px; height: 10px;                 border: 1px solid; background-color:                 hsl(0, 75%, 100%)\"></span> Neutral  <span style=\"display: inline-block; width: 10px; height: 10px;                 border: 1px solid; background-color:                 hsl(120, 75%, 50%)\"></span> Positive  </div><tr><th>True Label</th><th>Predicted Label</th><th>Attribution Label</th><th>Attribution Score</th><th>Word Importance</th><tr><td><text style=\"padding-right:2em\"><b>B-LOC</b></text></td><td><text style=\"padding-right:2em\"><b>B-LOC (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>Paris</b></text></td><td><text style=\"padding-right:2em\"><b>1.03</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 51%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>B-PER</b></text></td><td><text style=\"padding-right:2em\"><b>B-PER (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>Emmanuel</b></text></td><td><text style=\"padding-right:2em\"><b>1.17</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 80%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 55%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(0, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>I-PER</b></text></td><td><text style=\"padding-right:2em\"><b>I-PER (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>Mac</b></text></td><td><text style=\"padding-right:2em\"><b>1.27</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 53%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>I-PER</b></text></td><td><text style=\"padding-right:2em\"><b>I-PER (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>##ron</b></text></td><td><text style=\"padding-right:2em\"><b>1.57</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 79%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 62%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 81%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "html = ner_explainer.visualize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *ignored_labels* parameter can be combined with the *ignored_indexes* parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "attributions_4 = ner_explainer(sample_text, ignored_indexes=[11, 12, 13, 14], ignored_labels=['B-PER'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table width: 100%><div style=\"border-top: 1px solid; margin-top: 5px;             padding-top: 5px; display: inline-block\"><b>Legend: </b><span style=\"display: inline-block; width: 10px; height: 10px;                 border: 1px solid; background-color:                 hsl(0, 75%, 60%)\"></span> Negative  <span style=\"display: inline-block; width: 10px; height: 10px;                 border: 1px solid; background-color:                 hsl(0, 75%, 100%)\"></span> Neutral  <span style=\"display: inline-block; width: 10px; height: 10px;                 border: 1px solid; background-color:                 hsl(120, 75%, 50%)\"></span> Positive  </div><tr><th>True Label</th><th>Predicted Label</th><th>Attribution Label</th><th>Attribution Score</th><th>Word Importance</th><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>[CLS]</b></text></td><td><text style=\"padding-right:2em\"><b>1.13</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 90%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 77%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 78%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 80%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 85%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>We</b></text></td><td><text style=\"padding-right:2em\"><b>0.16</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 56%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(0, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(0, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(0, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>visited</b></text></td><td><text style=\"padding-right:2em\"><b>1.15</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 58%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 87%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 84%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>B-LOC</b></text></td><td><text style=\"padding-right:2em\"><b>B-LOC (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>Paris</b></text></td><td><text style=\"padding-right:2em\"><b>1.03</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 51%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>last</b></text></td><td><text style=\"padding-right:2em\"><b>1.46</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 53%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>weekend</b></text></td><td><text style=\"padding-right:2em\"><b>1.97</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 60%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 76%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>,</b></text></td><td><text style=\"padding-right:2em\"><b>2.23</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 91%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 70%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 75%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>O</b></text></td><td><text style=\"padding-right:2em\"><b>O (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>where</b></text></td><td><text style=\"padding-right:2em\"><b>1.94</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 89%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 77%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 62%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 88%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(120, 75%, 93%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>I-PER</b></text></td><td><text style=\"padding-right:2em\"><b>I-PER (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>Mac</b></text></td><td><text style=\"padding-right:2em\"><b>1.27</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 98%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 86%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 53%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr><tr><td><text style=\"padding-right:2em\"><b>I-PER</b></text></td><td><text style=\"padding-right:2em\"><b>I-PER (1.00)</b></text></td><td><text style=\"padding-right:2em\"><b>##ron</b></text></td><td><text style=\"padding-right:2em\"><b>1.57</b></text></td><td><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [CLS]                    </font></mark><mark style=\"background-color: hsl(0, 75%, 95%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> We                    </font></mark><mark style=\"background-color: hsl(120, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> visited                    </font></mark><mark style=\"background-color: hsl(120, 75%, 92%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Paris                    </font></mark><mark style=\"background-color: hsl(120, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> last                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> weekend                    </font></mark><mark style=\"background-color: hsl(120, 75%, 97%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ,                    </font></mark><mark style=\"background-color: hsl(120, 75%, 79%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> where                    </font></mark><mark style=\"background-color: hsl(120, 75%, 62%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Emmanuel                    </font></mark><mark style=\"background-color: hsl(120, 75%, 81%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> Mac                    </font></mark><mark style=\"background-color: hsl(0, 75%, 99%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> ##ron                    </font></mark><mark style=\"background-color: hsl(120, 75%, 96%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> lives                    </font></mark><mark style=\"background-color: hsl(0, 75%, 94%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> .                    </font></mark><mark style=\"background-color: hsl(0, 75%, 100%); opacity:1.0;                     line-height:1.75\"><font color=\"black\"> [SEP]                    </font></mark></td><tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "html = ner_explainer.visualize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the tokens are ignored based on the predicted class and its index. The token `Emmanuel` is wrongly predicted as `B-PER`and is therefore ignored along with the tokens in positions `11`, `12`, `13` and `14`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "9c8f96ea35162dbd38aade767cddd21b2379c042e1e2d3cef41825d090a191d5"
  },
  "kernelspec": {
   "display_name": "Python 3.9.5 ('.venv': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}