tutorials/nlp_pipeline/D_information_extraction/02_named_entities.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:purple\"> Information extraction: named entities </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Named Entity Recognition (NER) is a subtask of information extraction that seeks to locate and classify elements in text into pre-defined categories such as the names of persons, organizations, locations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "EstNLTK has multiple NER models for Estonian. \n",
    "The traditional machine learning based model was created by [Tkachenko et al. (2013)](https://aclanthology.org/W13-2412/), and is available through `NerTagger` and `WordLevelNerTagger` components in EstNLTK. This model annotates names from categories `PER`, `LOC`, `ORG`.\n",
    "\n",
    "Neural Bert-based models were created by [Kairit Sirts (2023)](https://openreview.net/pdf?id=4CTnlIc1rhw), and are usable via `EstBERTNERTagger` component in EstNLTK neural. \n",
    "There are 2 neural NER models: the first recognises names from the basic categories (`PER`, `LOC`, `ORG`), and the second recognises names from the extended set of categories: `PER`, `LOC`, `ORG`, `GPE`, `MONEY`, `PERCENT`, `PROD`, `TITLE`, `DATE`, `TIME`, `EVENT`. The first model is also usable via webservice (`NerWebTagger`). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The basic NER model (traditional ML)\n",
    "\n",
    "The traditional machine learning based NER is distributed with the EstNLTK main package. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can apply NER directly via default resolver:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>Layer</h4>\n",
       "\n",
       "\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>layer name</th>\n",
       "      <th>attributes</th>\n",
       "      <th>parent</th>\n",
       "      <th>enveloping</th>\n",
       "      <th>ambiguous</th>\n",
       "      <th>span count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>ner</td>\n",
       "      <td>nertag</td>\n",
       "      <td>None</td>\n",
       "      <td>words</td>\n",
       "      <td>False</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>nertag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>['Eesti']</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Alar', 'Karis']</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Eesti', 'Energia']</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Eesti', 'riigile']</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Layer(name='ner', attributes=('nertag',), spans=SL[EnvelopingSpan(['Eesti'], [{'nertag': 'LOC'}]),\n",
       "EnvelopingSpan(['Alar', 'Karis'], [{'nertag': 'PER'}]),\n",
       "EnvelopingSpan(['Eesti', 'Energia'], [{'nertag': 'ORG'}]),\n",
       "EnvelopingSpan(['Eesti', 'riigile'], [{'nertag': 'LOC'}])])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from estnltk import Text\n",
    "\n",
    "text = Text('Eesti president on Alar Karis. Eesti Energia on Eesti riigile kuuluv energiaettevõte.')\n",
    "\n",
    "text.tag_layer('ner')\n",
    "\n",
    "text.ner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use `enclosing_text` to obtain exact strings corresponding to named entities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Eesti', 'Alar Karis', 'Eesti Energia', 'Eesti riigile']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get named entity strings\n",
    "[named_entity.enclosing_text for named_entity in text.ner]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While NerTagger does not provide lemmatization of names, you can iterate over all words of each named entity, and you can get lemmas for these words from the `morph_analysis` layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eesti ['Eesti']\n",
      "\n",
      "Alar ['Alar']\n",
      "Karis ['Karis']\n",
      "\n",
      "Eesti ['Eesti']\n",
      "Energia ['energia']\n",
      "\n",
      "Eesti ['Eesti']\n",
      "riigile ['riik']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get lemmas of the named entities\n",
    "for named_entity in text.ner:\n",
    "    for word in named_entity:\n",
    "        print(word.text, word.lemma)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that lemmas are provided as a list due to ambiguities in the morph analysis layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage as a tagger \n",
    "\n",
    "Next, we'll see how to use NerTagger as a separate tagger.\n",
    "\n",
    "Loading the tagger:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from estnltk.taggers import NerTagger\n",
    "\n",
    "nertagger = NerTagger()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a text object and add prerequisite layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from estnltk import Text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = Text('''Eesti Vabariik on riik Põhja-Euroopas. Eesti piirneb põhjas üle Soome lahe Soome Vabariigiga.''')\n",
    "text = text.tag_layer(['sentences', 'morph_analysis'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add the NER layer to the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td><div align = \"left\">Eesti Vabariik on riik Põhja-Euroopas. Eesti piirneb põhjas üle Soome lahe Soome Vabariigiga.</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>layer name</th>\n",
       "      <th>attributes</th>\n",
       "      <th>parent</th>\n",
       "      <th>enveloping</th>\n",
       "      <th>ambiguous</th>\n",
       "      <th>span count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>sentences</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>words</td>\n",
       "      <td>False</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>tokens</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>compound_tokens</td>\n",
       "      <td>type, normalized</td>\n",
       "      <td>None</td>\n",
       "      <td>tokens</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>words</td>\n",
       "      <td>normalized_form</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>morph_analysis</td>\n",
       "      <td>normalized_text, lemma, root, root_tokens, ending, clitic, form, partofspeech</td>\n",
       "      <td>words</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ner</td>\n",
       "      <td>nertag</td>\n",
       "      <td>None</td>\n",
       "      <td>words</td>\n",
       "      <td>False</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Text(text='Eesti Vabariik on riik Põhja-Euroopas. Eesti piirneb põhjas üle Soome lahe Soome Vabariigiga.')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nertagger.tag(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The nertag attribute shows the category of each named entity, either \"LOC\" - location, \"PER\" - person or \"ORG\" - organization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>Layer</h4>\n",
       "\n",
       "\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>layer name</th>\n",
       "      <th>attributes</th>\n",
       "      <th>parent</th>\n",
       "      <th>enveloping</th>\n",
       "      <th>ambiguous</th>\n",
       "      <th>span count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>ner</td>\n",
       "      <td>nertag</td>\n",
       "      <td>None</td>\n",
       "      <td>words</td>\n",
       "      <td>False</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>nertag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>['Eesti', 'Vabariik']</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Põhja-Euroopas']</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Eesti']</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Soome', 'lahe']</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Soome', 'Vabariigiga']</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Layer(name='ner', attributes=('nertag',), spans=SL[EnvelopingSpan(['Eesti', 'Vabariik'], [{'nertag': 'LOC'}]),\n",
       "EnvelopingSpan(['Põhja-Euroopas'], [{'nertag': 'LOC'}]),\n",
       "EnvelopingSpan(['Eesti'], [{'nertag': 'LOC'}]),\n",
       "EnvelopingSpan(['Soome', 'lahe'], [{'nertag': 'LOC'}]),\n",
       "EnvelopingSpan(['Soome', 'Vabariigiga'], [{'nertag': 'LOC'}])])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text.ner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For some use cases it might be better to have the output layer with a tag for each word. This can be used for visualizing or making manual changes to the layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from estnltk.taggers import WordLevelNerTagger\n",
    "\n",
    "word_level_ner = WordLevelNerTagger()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td><div align = \"left\">Eesti Vabariik on riik Põhja-Euroopas. Eesti piirneb põhjas üle Soome lahe Soome Vabariigiga.</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>layer name</th>\n",
       "      <th>attributes</th>\n",
       "      <th>parent</th>\n",
       "      <th>enveloping</th>\n",
       "      <th>ambiguous</th>\n",
       "      <th>span count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>sentences</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>words</td>\n",
       "      <td>False</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>tokens</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>compound_tokens</td>\n",
       "      <td>type, normalized</td>\n",
       "      <td>None</td>\n",
       "      <td>tokens</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>words</td>\n",
       "      <td>normalized_form</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>morph_analysis</td>\n",
       "      <td>normalized_text, lemma, root, root_tokens, ending, clitic, form, partofspeech</td>\n",
       "      <td>words</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ner</td>\n",
       "      <td>nertag</td>\n",
       "      <td>None</td>\n",
       "      <td>words</td>\n",
       "      <td>False</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>wordner</td>\n",
       "      <td>nertag</td>\n",
       "      <td>words</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Text(text='Eesti Vabariik on riik Põhja-Euroopas. Eesti piirneb põhjas üle Soome lahe Soome Vabariigiga.')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_level_ner.tag(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the tags are in IOB-format: B- prefix indicates that this token is the beginning of the named entity, I- prefix indicates that this token is inside the named entity. O shows that the token is outside named entities (not part of a named entity)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>Layer</h4>\n",
       "\n",
       "\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>layer name</th>\n",
       "      <th>attributes</th>\n",
       "      <th>parent</th>\n",
       "      <th>enveloping</th>\n",
       "      <th>ambiguous</th>\n",
       "      <th>span count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>wordner</td>\n",
       "      <td>nertag</td>\n",
       "      <td>words</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>nertag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Eesti</td>\n",
       "      <td>B-LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Vabariik</td>\n",
       "      <td>I-LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>on</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>riik</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Põhja-Euroopas</td>\n",
       "      <td>B-LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>.</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Eesti</td>\n",
       "      <td>B-LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>piirneb</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>põhjas</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>üle</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Soome</td>\n",
       "      <td>B-LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>lahe</td>\n",
       "      <td>I-LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Soome</td>\n",
       "      <td>B-LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Vabariigiga</td>\n",
       "      <td>I-LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>.</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Layer(name='wordner', attributes=('nertag',), spans=SL[Span('Eesti', [{'nertag': 'B-LOC'}]),\n",
       "Span('Vabariik', [{'nertag': 'I-LOC'}]),\n",
       "Span('on', [{'nertag': 'O'}]),\n",
       "Span('riik', [{'nertag': 'O'}]),\n",
       "Span('Põhja-Euroopas', [{'nertag': 'B-LOC'}]),\n",
       "Span('.', [{'nertag': 'O'}]),\n",
       "Span('Eesti', [{'nertag': 'B-LOC'}]),\n",
       "Span('piirneb', [{'nertag': 'O'}]),\n",
       "Span('põhjas', [{'nertag': 'O'}]),\n",
       "Span('üle', [{'nertag': 'O'}]),\n",
       "Span('Soome', [{'nertag': 'B-LOC'}]),\n",
       "Span('lahe', [{'nertag': 'I-LOC'}]),\n",
       "Span('Soome', [{'nertag': 'B-LOC'}]),\n",
       "Span('Vabariigiga', [{'nertag': 'I-LOC'}]),\n",
       "Span('.', [{'nertag': 'O'}])])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text.wordner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural NER models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running as a web tagger\n",
    "\n",
    "Easiest way to use neural NER is via TartuNLP's NER web service:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>MultiLayerTagger</h4>\n",
       "NER tagger that uses TartuNLP's NER web service. \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>name</th>\n",
       "      <th>output layers</th>\n",
       "      <th>output mapping</th>\n",
       "      <th>input layers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>NerWebTagger</td>\n",
       "      <td>['webner']</td>\n",
       "      <td>{'webner': ['nertag']}</td>\n",
       "      <td>('words',)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<h4>Configuration</h4>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>custom_words_layer</th>\n",
       "      <td>words</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>url</th>\n",
       "      <td>https://api.tartunlp.ai/bert/ner/v1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>batch_size</th>\n",
       "      <td>4500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "NerWebTagger(input_layers=('words',), output_layers=['webner'], output_layers_to_attributes={'webner': ['nertag']}, custom_words_layer=words, url=https://api.tartunlp.ai/bert/ner/v1, batch_size=4500)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from estnltk import Text\n",
    "from estnltk.web_taggers import NerWebTagger\n",
    "\n",
    "ner_web_tagger = NerWebTagger()\n",
    "ner_web_tagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>Layer</h4>\n",
       "\n",
       "\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>layer name</th>\n",
       "      <th>attributes</th>\n",
       "      <th>parent</th>\n",
       "      <th>enveloping</th>\n",
       "      <th>ambiguous</th>\n",
       "      <th>span count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>webner</td>\n",
       "      <td>nertag</td>\n",
       "      <td>None</td>\n",
       "      <td>words</td>\n",
       "      <td>False</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>nertag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>['Rootsi']</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Gustav', 'II', 'Adolf']</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Nürnbergi']</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Academia', \"Dorpatensis'e\"]</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Layer(name='webner', attributes=('nertag',), spans=SL[EnvelopingSpan(['Rootsi'], [{'nertag': 'LOC'}]),\n",
       "EnvelopingSpan(['Gustav', 'II', 'Adolf'], [{'nertag': 'PER'}]),\n",
       "EnvelopingSpan(['Nürnbergi'], [{'nertag': 'LOC'}]),\n",
       "EnvelopingSpan(['Academia', \"Dorpatensis'e\"], [{'nertag': 'ORG'}])])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = Text('30. juunil 1632. aastal kinnitas Rootsi kuningas Gustav II Adolf oma allkirjaga '+\\\n",
    "\"Nürnbergi all sõjalaagris Academia Dorpatensis'e asutamisüriku, mis lubab meie suurkooli ajalugu \"+\\\n",
    "\"sellest ajast alustada ja auväärseks lugeda.\")\n",
    "# Tag input layer\n",
    "text = text.tag_layer('words')\n",
    "# Tag named entities\n",
    "ner_web_tagger.tag( text )\n",
    "# check results\n",
    "text['webner']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remarks:\n",
    "* This webservice uses the [\"estbertner\"](https://huggingface.co/tartuNLP/EstBERT_NER) model which recognizes names from categories `PER`, `LOC` and `ORG`;\n",
    "* Be aware that using this tagger means that the input data will be processed by the public TartuNLP API. The text will be uploaded and can be read by a third party;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Local installation\n",
    "\n",
    "*In order to use the neural NER models, please install [estnltk_neural](https://github.com/estnltk/estnltk/tree/main/estnltk_neural) (v1.7.2+).*\n",
    "\n",
    "Before applying `EstBERTNERTagger`, you'll need to get the models. Models are not distributed with EstNLTK due to their large size, and need to be downloaded separately:\n",
    "\n",
    "* If you create a new instance of `EstBERTNERTagger` and the models are missing, you'll be prompted with a question asking for a permission to download the \"estbertner\" model;\n",
    "* Alternatively, you can pre-download models manually via `download` function:\n",
    "\n",
    "```python\n",
    "from estnltk import download\n",
    "download(\"estbertner\")\n",
    "download(\"estbertner_v2\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model estbertner\n",
    "\n",
    "The model [estbertner_v1](https://huggingface.co/tartuNLP/EstBERT_NER) is the default model of `EstBERTNERTagger`. \n",
    "It recognizes the following categories of named entities: persons (`PER`), locations (`LOC`) and organizations (`ORG`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>MultiLayerTagger</h4>\n",
       "EstNLTK wrapper for the huggingface EstBERTNER models.\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>name</th>\n",
       "      <th>output layers</th>\n",
       "      <th>output mapping</th>\n",
       "      <th>input layers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>EstBERTNERTagger</td>\n",
       "      <td>['estbertner']</td>\n",
       "      <td>{'estbertner': ['nertag']}</td>\n",
       "      <td>('words',)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<h4>Configuration</h4>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>model_location</th>\n",
       "      <td>C:\\Programmid\\Miniconda3\\envs\\py38_estnltk_neural\\lib\\site-packages\\estnltk-1.7. ..., type: &lt;class 'str'&gt;, length: 169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nlp</th>\n",
       "      <td>&lt;transformers.pipelines.token_classification.TokenClassificationPipeline object  ..., type: &lt;class 'transformers.pipelines.token_classification.TokenClassificationPipeline'&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tokenizer</th>\n",
       "      <td>BertTokenizer(name_or_path='C:\\Programmid\\Miniconda3\\envs\\py38_estnltk_neural\\li ..., type: &lt;class 'transformers.models.bert.tokenization_bert.BertTokenizer'&gt;, length: 50004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>custom_words_layer</th>\n",
       "      <td>words</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>batch_size</th>\n",
       "      <td>1750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>postfix_expand_suffixes</th>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>postfix_concat_same_type_entities</th>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>postfix_remove_infix_matches</th>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "EstBERTNERTagger(input_layers=('words',), output_layers=['estbertner'], output_layers_to_attributes={'estbertner': ['nertag']}, model_location=C:\\Programmid\\Miniconda3\\envs\\py38_estnltk_neural\\lib\\site-packages\\estnltk-1.7. ..., type: <class 'str'>, length: 169, nlp=<transformers.pipelines.token_classification.TokenClassificationPipeline object  ..., type: <class 'transformers.pipelines.token_classification.TokenClassificationPipeline'>, tokenizer=BertTokenizer(name_or_path='C:\\Programmid\\Miniconda3\\envs\\py38_estnltk_neural\\li ..., type: <class 'transformers.models.bert.tokenization_bert.BertTokenizer'>, length: 50004, custom_words_layer=words, batch_size=1750, postfix_expand_suffixes=False, postfix_concat_same_type_entities=False, postfix_remove_infix_matches=False)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from estnltk_neural.taggers import EstBERTNERTagger\n",
    "neural_ner = EstBERTNERTagger()\n",
    "neural_ner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>Layer</h4>\n",
       "\n",
       "\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>layer name</th>\n",
       "      <th>attributes</th>\n",
       "      <th>parent</th>\n",
       "      <th>enveloping</th>\n",
       "      <th>ambiguous</th>\n",
       "      <th>span count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>estbertner</td>\n",
       "      <td>nertag</td>\n",
       "      <td>None</td>\n",
       "      <td>words</td>\n",
       "      <td>False</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>nertag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>['Eesti']</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Alar', 'Karis']</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Eesti', 'Energia']</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Eesti', 'riigile']</td>\n",
       "      <td>LOC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Layer(name='estbertner', attributes=('nertag',), spans=SL[EnvelopingSpan(['Eesti'], [{'nertag': 'LOC'}]),\n",
       "EnvelopingSpan(['Alar', 'Karis'], [{'nertag': 'PER'}]),\n",
       "EnvelopingSpan(['Eesti', 'Energia'], [{'nertag': 'ORG'}]),\n",
       "EnvelopingSpan(['Eesti', 'riigile'], [{'nertag': 'LOC'}])])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from estnltk import Text\n",
    "\n",
    "text = Text('Eesti president on Alar Karis. Eesti Energia on Eesti riigile kuuluv energiaettevõte.')\n",
    "# NER requires words layer\n",
    "text.tag_layer('words')\n",
    "# Add NER layer\n",
    "neural_ner.tag(text)\n",
    "\n",
    "text.estbertner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model estbertner_v2\n",
    "\n",
    "The model [estbertner_v2](https://huggingface.co/tartuNLP/EstBERT_NER_v2) recognizes named entities of 11 categories: `PER`, `LOC`, `ORG`, `GPE`, `MONEY`, `PERCENT`, `PROD`, `TITLE`, `DATE`, `TIME`, `EVENT`. \n",
    "In order to use this model, you first need to download it via EstNLTK's downloader, and then pass it's location to `EstBERTNERTagger` via parameter `model_location`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Resource 'estbertner_v2_from_tartunlp_hf_2022-12-12' has already been downloaded.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the model\n",
    "from estnltk import download, get_resource_paths\n",
    "download(\"estbertner_v2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [],
   "source": [
    "# Initialize tagger with the model\n",
    "model_location = get_resource_paths(\"estbertner_v2\", only_latest=True)\n",
    "neural_ner2 = EstBERTNERTagger(model_location=model_location, output_layer='estbertner2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>Layer</h4>\n",
       "\n",
       "\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>layer name</th>\n",
       "      <th>attributes</th>\n",
       "      <th>parent</th>\n",
       "      <th>enveloping</th>\n",
       "      <th>ambiguous</th>\n",
       "      <th>span count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>estbertner2</td>\n",
       "      <td>nertag</td>\n",
       "      <td>None</td>\n",
       "      <td>words</td>\n",
       "      <td>False</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>nertag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>['Kaia', 'Kanepi']</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['WTA']</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['USA-s']</td>\n",
       "      <td>GPE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Charlestonis']</td>\n",
       "      <td>GPE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['WTA', '500', 'kategooria']</td>\n",
       "      <td>EVENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['poolatarile']</td>\n",
       "      <td>TITLE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Magda', \"Linette'ile\"]</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['WTA']</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Layer(name='estbertner2', attributes=('nertag',), spans=SL[EnvelopingSpan(['Kaia', 'Kanepi'], [{'nertag': 'PER'}]),\n",
       "EnvelopingSpan(['WTA'], [{'nertag': 'ORG'}]),\n",
       "EnvelopingSpan(['USA-s'], [{'nertag': 'GPE'}]),\n",
       "EnvelopingSpan(['Charlestonis'], [{'nertag': 'GPE'}]),\n",
       "EnvelopingSpan(['WTA', '500', 'kategooria'], [{'nertag': 'EVENT'}]),\n",
       "EnvelopingSpan(['poolatarile'], [{'nertag': 'TITLE'}]),\n",
       "EnvelopingSpan(['Magda', \"Linette'ile\"], [{'nertag': 'PER'}]),\n",
       "EnvelopingSpan(['WTA'], [{'nertag': 'ORG'}])])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from estnltk import Text\n",
    "\n",
    "text = Text(\"Kaia Kanepi (WTA 57.) langes USA-s Charlestonis toimuval WTA 500 kategooria tenniseturniiril konkurentsist \"+\\\n",
    "            \"kaheksandikfinaalis, kaotades poolatarile Magda Linette'ile (WTA 64.) 3 : 6, 6 : 4, 2 : 6.\")\n",
    "# NER requires words layer\n",
    "text.tag_layer('words')\n",
    "# Add NER layer\n",
    "neural_ner2.tag(text)\n",
    "\n",
    "text.estbertner2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization of NE layers. Restoring Bert's tokenization\n",
    "\n",
    "By default, detected named entities will envelop around EstNLTK's words layer. \n",
    "However, under the hood, Bert models use a different tokenization and the envelopment around words has been achieved heuristically. \n",
    "In many cases, Bert's tokenizer splits a word into multiple tokens (e.g. `'Charlestonis'` will become tokenized into `['Cha', 'rl', 'est', 'onis']`) and Bert's named entity labels may not cover all tokens of a word (e.g. frequently Estonian case endings will not be covered).\n",
    "The heuristic checks for a minimal (one letter) overlap between a Bert token with a named entity label and a word: if the overlap exists, then the word will obtain corresponding named entity label. \n",
    "If there are multiple Bert tokens with named entity labels covering a word, then the word will obtain the label of the first Bert token.\n",
    "\n",
    "If you want to get the original Bert NE output/tokenization, then use `custom_words_layer=None` to switch off the heuristic. \n",
    "As a result, the tagger will create 2 layers: the layer with the original Bert tokenization (`nertokens`), and the NE layer enveloping around the Bert tokenization layer.\n",
    "Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [],
   "source": [
    "# Switch off mapping NE-s to words layer\n",
    "neural_ner3 = EstBERTNERTagger(model_location=model_location, output_layer='estbertner3', custom_words_layer=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>Layer</h4>\n",
       "\n",
       "\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>layer name</th>\n",
       "      <th>attributes</th>\n",
       "      <th>parent</th>\n",
       "      <th>enveloping</th>\n",
       "      <th>ambiguous</th>\n",
       "      <th>span count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>estbertner3</td>\n",
       "      <td>nertag</td>\n",
       "      <td>None</td>\n",
       "      <td>nertokens</td>\n",
       "      <td>False</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>nertag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>['Kai']</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['a', 'Kanepi']</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['W']</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['USA']</td>\n",
       "      <td>GPE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Cha', 'rl', 'est', 'onis']</td>\n",
       "      <td>GPE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['W', 'TA', '500', 'kategooria']</td>\n",
       "      <td>EVENT</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['poola']</td>\n",
       "      <td>TITLE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['Ma']</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['gda', 'Line', 'tte', \"'\", 'ile']</td>\n",
       "      <td>PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>['W']</td>\n",
       "      <td>ORG</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Layer(name='estbertner3', attributes=('nertag',), spans=SL[EnvelopingSpan(['Kai'], [{'nertag': 'PER'}]),\n",
       "EnvelopingSpan(['a', 'Kanepi'], [{'nertag': 'PER'}]),\n",
       "EnvelopingSpan(['W'], [{'nertag': 'ORG'}]),\n",
       "EnvelopingSpan(['USA'], [{'nertag': 'GPE'}]),\n",
       "EnvelopingSpan(['Cha', 'rl', 'est', 'onis'], [{'nertag': 'GPE'}]),\n",
       "EnvelopingSpan(['W', 'TA', '500', 'kategooria'], [{'nertag': 'EVENT'}]),\n",
       "EnvelopingSpan(['poola'], [{'nertag': 'TITLE'}]),\n",
       "EnvelopingSpan(['Ma'], [{'nertag': 'PER'}]),\n",
       "EnvelopingSpan(['gda', 'Line', 'tte', \"'\", 'ile'], [{'nertag': 'PER'}]),\n",
       "EnvelopingSpan(['W'], [{'nertag': 'ORG'}])])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add NER layer\n",
    "neural_ner3.tag(text)\n",
    "\n",
    "# Examine NER results\n",
    "text.estbertner3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>\n",
       "var elements = document.getElementsByClassName(\"overlapping-span\")\n",
       "for (let i = 0; i < elements.length; i++){\n",
       "    elements.item(i).addEventListener(\"click\",function() {show_conflicting_spans(elements.item(i));})}\n",
       "\n",
       "function show_conflicting_spans(span_element) {\n",
       "    let spantable = document.createElement('div')\n",
       "    spantable.classList.add('tables')\n",
       "\n",
       "    // Prepare the contents of the span table\n",
       "    data = span_element.getAttribute(\"span_info\")\n",
       "    data = data.split(\",\")\n",
       "    var spancontent = '<table>'\n",
       "    for (let row of data) {\n",
       "        spancontent+='<tr><td>'\n",
       "        spancontent+=row\n",
       "        spancontent+='</td></tr>'\n",
       "    }\n",
       "    spancontent += '</table>'\n",
       "    spantable.innerHTML = spancontent\n",
       "    span_element.parentElement.appendChild(spantable)\n",
       "\n",
       "    // Increase the size of the cell so the tables would fit\n",
       "    spantable.parentElement.style.height = Math.max(Number(spantable.parentElement.style.height.substring(0,spantable.parentElement.style.height.length-2)),span_element.offsetTop+90)+ 'px'\n",
       "    // Position the table directly below the corresponding text\n",
       "    spantable.style.left = span_element.getBoundingClientRect().left-spantable.parentElement.parentElement.getBoundingClientRect().left + 'px'\n",
       "    spantable.style.top = span_element.getBoundingClientRect().top-spantable.parentElement.parentElement.getBoundingClientRect().top+20+ 'px'\n",
       "\n",
       "    // Remove the table when clicked on again\n",
       "    spantable.addEventListener('click', function () {\n",
       "        let element = this.parentElement\n",
       "        element.removeChild(this)\n",
       "    })\n",
       "}\n",
       "</script><style>\n",
       ".span {\n",
       "    background-color: yellow;\n",
       "}\n",
       "\n",
       ".overlapping-span {\n",
       "    background-color: red;\n",
       "}\n",
       "\n",
       ".spanline {\n",
       "    background-color: blue;\n",
       "    position: relative;\n",
       "    height: 3px;\n",
       "    margin-left: 0px;\n",
       "}\n",
       "\n",
       ".tables {\n",
       "    position: absolute;\n",
       "    width: fit-content;\n",
       "    width: -moz-fit-content;\n",
       "    border: 1px solid black;\n",
       "}\n",
       "\n",
       ".maintext{0} {\n",
       "    position: relative;\n",
       "}\n",
       "\n",
       ".tables tbody tr:nth-child(even) {\n",
       "    background-color: lightgray;\n",
       "}\n",
       "\n",
       ".tables tbody tr:nth-child(odd) {\n",
       "    background-color: beige;\n",
       "}\n",
       "\n",
       ".tables tbody tr:hover {\n",
       "    background-color: ivory;\n",
       "}\n",
       "</style><span style=background:yellow; \">Kai</span><span style=background:yellow; \">a Kanepi</span> (<span style=background:yellow; \">W</span>TA 57.) langes <span style=background:yellow; \">USA</span>-s <span style=background:yellow; \">Charlestonis</span> toimuval <span style=background:yellow; \">WTA 500 kategooria</span> tenniseturniiril konkurentsist kaheksandikfinaalis, kaotades <span style=background:yellow; \">poola</span>tarile <span style=background:yellow; \">Ma</span><span style=background:yellow; \">gda Linette'ile</span> (<span style=background:yellow; \">W</span>TA 64.) 3 : 6, 6 : 4, 2 : 6."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "text.estbertner3.display()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": [
     "nbval-skip"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>Layer</h4>\n",
       "\n",
       "\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>layer name</th>\n",
       "      <th>attributes</th>\n",
       "      <th>parent</th>\n",
       "      <th>enveloping</th>\n",
       "      <th>ambiguous</th>\n",
       "      <th>span count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>nertokens</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>56</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>Kai</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Kanepi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>(</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>W</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>TA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>langes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>USA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>-</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Cha</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>rl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>est</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>onis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>toimuval</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>W</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>TA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>kategooria</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>tennise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>turniiril</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>konkurentsist</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>kaheksandik</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>finaalis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>kaotades</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>poola</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>tari</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>le</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ma</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>gda</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Line</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>tte</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>ile</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>(</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>W</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>TA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>:</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>:</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>:</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "Layer(name='nertokens', attributes=(), spans=SL[Span('Kai', [{}]),\n",
       "Span('a', [{}]),\n",
       "Span('Kanepi', [{}]),\n",
       "Span('(', [{}]),\n",
       "Span('W', [{}]),\n",
       "Span('TA', [{}]),\n",
       "Span('57', [{}]),\n",
       "Span('.', [{}]),\n",
       "Span(')', [{}]),\n",
       "Span('langes', [{}]),\n",
       "Span('USA', [{}]),\n",
       "Span('-', [{}]),\n",
       "Span('s', [{}]),\n",
       "Span('Cha', [{}]),\n",
       "Span('rl', [{}]),\n",
       "Span('est', [{}]),\n",
       "Span('onis', [{}]),\n",
       "Span('toimuval', [{}]),\n",
       "Span('W', [{}]),\n",
       "Span('TA', [{}]),\n",
       "Span('500', [{}]),\n",
       "Span('kategooria', [{}]),\n",
       "Span('tennise', [{}]),\n",
       "Span('turniiril', [{}]),\n",
       "Span('konkurentsist', [{}]),\n",
       "Span('kaheksandik', [{}]),\n",
       "Span('finaalis', [{}]),\n",
       "Span(',', [{}]),\n",
       "Span('kaotades', [{}]),\n",
       "Span('poola', [{}]),\n",
       "Span('tari', [{}]),\n",
       "Span('le', [{}]),\n",
       "Span('Ma', [{}]),\n",
       "Span('gda', [{}]),\n",
       "Span('Line', [{}]),\n",
       "Span('tte', [{}]),\n",
       "Span(\"'\", [{}]),\n",
       "Span('ile', [{}]),\n",
       "Span('(', [{}]),\n",
       "Span('W', [{}]),\n",
       "Span('TA', [{}]),\n",
       "Span('64', [{}]),\n",
       "Span('.', [{}]),\n",
       "Span(')', [{}]),\n",
       "Span('3', [{}]),\n",
       "Span(':', [{}]),\n",
       "Span('6', [{}]),\n",
       "Span(',', [{}]),\n",
       "Span('6', [{}]),\n",
       "Span(':', [{}]),\n",
       "Span('4', [{}]),\n",
       "Span(',', [{}]),\n",
       "Span('2', [{}]),\n",
       "Span(':', [{}]),\n",
       "Span('6', [{}]),\n",
       "Span('.', [{}])])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Examine the original Bert's tokenization\n",
    "text.nertokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}