Acquire_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extract Lemmatization from JSON\n",
    "The code in this notebook will the parse [ORACC](http://oracc.org) `JSON` file of the Ur III corpus to extract lemmatization data. \n",
    "\n",
    "The output contains text IDs, line IDs, lemmas, and (potentially) other data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import zipfile\n",
    "import json\n",
    "import requests\n",
    "from tqdm.auto import tqdm\n",
    "import os\n",
    "import errno"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0 Create Directories, if Necessary\n",
    "The two directories needed for this script are `jsonzip` and `output`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "for d in ['jsonzip', 'output']:\n",
    "    try:\n",
    "        os.mkdir(d)\n",
    "    except OSError as exc:\n",
    "        if exc.errno !=errno.EEXIST:\n",
    "            raise\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Input Project Name\n",
    "In order to download the data we need to know the [ORACC](http://oracc.org) project name of the Ur III data. This project name corresponds to the last part of the URL; the URL is http://oracc.org/epsd2/admin/ur3 and thus the project name is 'epsd2/admin/u3adm'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "p = ['epsd2/admin/ur3']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download the ZIP files.\n",
    "Download the zipped JSON files from the [ORACC](http://oracc.org) server. If the file is already on your local computer and has not been updated on the server, you may skip this cell (downloading can take quite some time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving http://build-oracc.museum.upenn.edu/json/epsd2-admin-ur3.zip as jsonzip/epsd2-admin-ur3.zip\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6607299b20ff481aa356e3ec71a45ec7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=1, bar_style='info', description='epsd2/admin/ur3', max=1, style=ProgressStyl…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "CHUNK = 16 * 1024\n",
    "for project in p:\n",
    "    proj = project.replace('/', '-')\n",
    "    url = \"http://build-oracc.museum.upenn.edu/json/\" + proj + \".zip\"\n",
    "    file = 'jsonzip/' + proj + '.zip'\n",
    "    with requests.get(url, stream=True) as r:\n",
    "        if r.status_code == 200:\n",
    "            tqdm.write('Saving ' + url + ' as ' + file)\n",
    "            with open(file, 'wb') as f:\n",
    "                for c in tqdm(r.iter_content(chunk_size=CHUNK), desc = project):\n",
    "                    f.write(c)\n",
    "        else:\n",
    "            tqdm.write(\"WARNING: \" + url + \" does not exist.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a name=\"head21\"></a>2.1 The `parsejson()` function\n",
    "The `parsejson()` function will \"dig into\" the `json` file (transformed into a dictionary) until it finds the relevant data. The `json` file consists of a hierarchy of `cdl` nodes; only the lowest nodes contain lemmatization data. The function goes down this hierarchy by calling itself when another `cdl` node is encountered. For more information about the data hierarchy in the [ORACC](http://oracc.org) `json` files, see [ORACC Open Data](http://oracc.org/doc/opendata/index.html).\n",
    "\n",
    "The argument of the `parsejson()` function is a `JSON` object, essentially a Python dictionary that initially contains the entire contents of the original JSON file. The code takes the key `cdl` the value of which is a list of `JSON` dictionaries. Iterating through these dictionaries, if a dictionary contains another `cdl` node, the function calls itself with this lower-level dictionary as argument. This way the function digs deeper and deeper into the `JSON` tree, until it does not encounter a `cdl` key anymore. Here we are at the level of individual words. The code checks for a key `f`, if it exists the value of that key (another dictionary) is appended to the list `lemm_l`. The list `lemm_l`, which is initiated outside of the function proper, will become a list of dictionaries, where each dictionary represents a single word.\n",
    "\n",
    "The variable `id_text` consists of a project abbreviation, such as `blms` or `cams/gkab` plus a text ID, in the format `cams/gkab/P338616` or `dcclt/Q000039`. The `id_text` is a global variable that is defined each time `parsejson()` is called in the main process. Therefore, it can be accessed from within the function and is added to the lemmatization data of every word.\n",
    "\n",
    "The variable `ftype` is used to denote if a node is a year name (`yn`) or not. \n",
    "\n",
    "The field `word_id` consists of three parts, namely a text ID, line ID, and word ID, in the format `Q000039.76.2` meaning: the second word in line 76 of text object `Q000039`. Note that `76` is not a line number strictly speaking but an object reference within the text object. Things like horizontal rulings, columns, and breaks also get object references. The `word_id` field allows us to put lines, breaks, and horizontal drawings together in the proper order.\n",
    "\n",
    "The field `label` is a human-legible label that refers to a line or another part of the text; it may look like `o i 23` (obverse column 1 line 23) or `r v 23'` (reverse column 5 line 23 prime). The `label` field is used in online [ORACC](http://oracc.org) editions to indicate line numbers.\n",
    "\n",
    "The fields `extent`, `scope`, and `state` give metatextual data about the condition of the object; they capture the number of broken lines or columns and similar information. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parsejson(text):\n",
    "    for JSONobject in text[\"cdl\"]:\n",
    "        if \"cdl\" in JSONobject: \n",
    "            parsejson(JSONobject)\n",
    "        if \"label\" in JSONobject:\n",
    "            meta_d[\"label\"] = JSONobject['label']\n",
    "        if \"f\" in JSONobject:\n",
    "            lemma = JSONobject[\"f\"]\n",
    "            if \"ftype\" in JSONobject:\n",
    "                lemma['ftype'] = JSONobject['ftype']\n",
    "            lemma[\"id_word\"] = JSONobject[\"ref\"]\n",
    "            lemma['label'] = meta_d[\"label\"]\n",
    "            lemma[\"id_text\"] = meta_d[\"id_text\"]\n",
    "            lemm_l.append(lemma)\n",
    "        if \"strict\" in JSONobject and JSONobject[\"strict\"] == \"1\":\n",
    "            lemma = {key: JSONobject[key] for key in dollar_keys}\n",
    "            lemma[\"id_word\"] = JSONobject[\"ref\"]\n",
    "            lemma[\"id_text\"] = meta_d[\"id_text\"]\n",
    "            lemm_l.append(lemma)\n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Call the `parsejson()` function for every `JSON` file\n",
    "The code in this cell will iterate through the list of projects entered above (1.1). For each project the `JSON` zip file is located in the directory `jsonzip`, named PROJECT.zip. The `zip` file contains a directory that is called `corpusjson` that contains a JSON file for every text that is available in that corpus. The files are called after their text IDs in the pattern `P######.json` (or `Q######.json` or `X######.json`).\n",
    "\n",
    "The function `namelist()` of the `zipfile` package is used to create a list of the names of all the files in the ZIP. From this list we select all the file names in the `corpusjson` directory with extension `.json` (this way we exclude the name of the directory itself). \n",
    "\n",
    "Each of these files is read from the `zip` file and loaded with the command `json.loads()`, which transforms the string into a proper JSON object. \n",
    "\n",
    "This JSON object (essentially a Python dictionary), which is called `data_json` is now sent to the `parsejson()` function. The function adds lemmata to the `lemm_l` list. In the end, `lemm_l` will contain as many list elements as there are words in all the texts in the projects requested.\n",
    "\n",
    "The dictionary `meta_d` is created to hold temporary information. The value of the key `id_text` is updated in the main process every time a new JSON file is opened and send to the `parsejson()` function. The `parsejson()` function itself will change values or add new keys, depending on the information found while iterating through the JSON file. When a new lemma row is created, `parsejon()` will supply data such as `id_text`, `label` and (potentially) other information from `meta_d`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d42a1d1d055439a8e653864d1cd8b49",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='epsd2/admin/ur3', max=71561, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "lemm_l = []\n",
    "meta_d = {\"label\": None, \"id_text\": None}\n",
    "dollar_keys = [\"extent\", \"scope\", \"state\"]\n",
    "for project in p:\n",
    "    file = \"jsonzip/\" + project.replace(\"/\", \"-\") + \".zip\"\n",
    "    try:\n",
    "        z = zipfile.ZipFile(file)       # create a Zipfile object\n",
    "    except:\n",
    "        print(file + \" does not exist or is not a proper ZIP file\")\n",
    "        continue\n",
    "    files = z.namelist()     # list of all the files in the ZIP\n",
    "    files = [name for name in files if \"corpusjson\" in name and name[-5:] == '.json']                                                                                                  #that holds all the P, Q, and X numbers.\n",
    "    for filename in tqdm(files, desc = project):       #iterate over the file names\n",
    "        id_text = project + filename[-13:-5] # id_text is, for instance, blms/P414332\n",
    "        meta_d[\"id_text\"] = id_text\n",
    "        try:\n",
    "            st = z.read(filename).decode('utf-8')         #read and decode the json file of one particular text\n",
    "            data_json = json.loads(st)                # make it into a json object (essentially a dictionary)\n",
    "            parsejson(data_json)               # and send to the parsejson() function\n",
    "        except:\n",
    "            print(id_text + ' is not available or not complete')\n",
    "    z.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 Data Structuring\n",
    "### 3.1 Transform the Data into a DataFrame\n",
    "The list `lemm_l` is transformed into a `pandas` dataframe for further manipulation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lang</th>\n",
       "      <th>form</th>\n",
       "      <th>delim</th>\n",
       "      <th>gdl</th>\n",
       "      <th>pos</th>\n",
       "      <th>id_word</th>\n",
       "      <th>label</th>\n",
       "      <th>id_text</th>\n",
       "      <th>cf</th>\n",
       "      <th>gw</th>\n",
       "      <th>...</th>\n",
       "      <th>epos</th>\n",
       "      <th>base</th>\n",
       "      <th>morph</th>\n",
       "      <th>cont</th>\n",
       "      <th>extent</th>\n",
       "      <th>scope</th>\n",
       "      <th>state</th>\n",
       "      <th>aform</th>\n",
       "      <th>stem</th>\n",
       "      <th>norm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>sux</td>\n",
       "      <td>1(geš₂)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '1(geš₂)', 'id': 'P113959....</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.1</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>sux</td>\n",
       "      <td>1(u)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.2...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.2</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>sux</td>\n",
       "      <td>6(aš)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '6(aš)', 'id': 'P113959.3....</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.3</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>sux</td>\n",
       "      <td>4(barig)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '4(barig)', 'id': 'P113959...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.4</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>sux</td>\n",
       "      <td>3(ban₂)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '3(ban₂)', 'id': 'P113959....</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.5</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>sux</td>\n",
       "      <td>4(diš)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '4(diš)', 'id': 'P113959.3...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.6</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>sux</td>\n",
       "      <td>sila₃</td>\n",
       "      <td></td>\n",
       "      <td>[{'v': 'sila₃', 'id': 'P113959.3.7.0'}]</td>\n",
       "      <td>N</td>\n",
       "      <td>P113959.3.7</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>sila</td>\n",
       "      <td>unit</td>\n",
       "      <td>...</td>\n",
       "      <td>N</td>\n",
       "      <td>sila₃</td>\n",
       "      <td>~</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>sux</td>\n",
       "      <td>1(u)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.8...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.8</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>sux</td>\n",
       "      <td>gin₂</td>\n",
       "      <td></td>\n",
       "      <td>[{'v': 'gin₂', 'id': 'P113959.3.9.0'}]</td>\n",
       "      <td>N</td>\n",
       "      <td>P113959.3.9</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>giŋ</td>\n",
       "      <td>unit</td>\n",
       "      <td>...</td>\n",
       "      <td>N</td>\n",
       "      <td>giŋ₄</td>\n",
       "      <td>~</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>sux</td>\n",
       "      <td>še</td>\n",
       "      <td></td>\n",
       "      <td>[{'v': 'še', 'id': 'P113959.3.10.0'}]</td>\n",
       "      <td>N</td>\n",
       "      <td>P113959.3.10</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>še</td>\n",
       "      <td>barley</td>\n",
       "      <td>...</td>\n",
       "      <td>N</td>\n",
       "      <td>še</td>\n",
       "      <td>~</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  lang      form delim                                                gdl pos  \\\n",
       "0  sux   1(geš₂)        [{'n': 'n', 'form': '1(geš₂)', 'id': 'P113959....   n   \n",
       "1  sux      1(u)        [{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.2...   n   \n",
       "2  sux     6(aš)        [{'n': 'n', 'form': '6(aš)', 'id': 'P113959.3....   n   \n",
       "3  sux  4(barig)        [{'n': 'n', 'form': '4(barig)', 'id': 'P113959...   n   \n",
       "4  sux   3(ban₂)        [{'n': 'n', 'form': '3(ban₂)', 'id': 'P113959....   n   \n",
       "5  sux    4(diš)        [{'n': 'n', 'form': '4(diš)', 'id': 'P113959.3...   n   \n",
       "6  sux     sila₃                  [{'v': 'sila₃', 'id': 'P113959.3.7.0'}]   N   \n",
       "7  sux      1(u)        [{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.8...   n   \n",
       "8  sux      gin₂                   [{'v': 'gin₂', 'id': 'P113959.3.9.0'}]   N   \n",
       "9  sux        še                    [{'v': 'še', 'id': 'P113959.3.10.0'}]   N   \n",
       "\n",
       "        id_word label                  id_text    cf      gw  ... epos   base  \\\n",
       "0   P113959.3.1   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "1   P113959.3.2   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "2   P113959.3.3   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "3   P113959.3.4   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "4   P113959.3.5   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "5   P113959.3.6   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "6   P113959.3.7   o 1  epsd2/admin/ur3/P113959  sila    unit  ...    N  sila₃   \n",
       "7   P113959.3.8   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "8   P113959.3.9   o 1  epsd2/admin/ur3/P113959   giŋ    unit  ...    N   giŋ₄   \n",
       "9  P113959.3.10   o 1  epsd2/admin/ur3/P113959    še  barley  ...    N     še   \n",
       "\n",
       "  morph cont extent scope state aform stem norm  \n",
       "0                                                \n",
       "1                                                \n",
       "2                                                \n",
       "3                                                \n",
       "4                                                \n",
       "5                                                \n",
       "6     ~                                          \n",
       "7                                                \n",
       "8     ~                                          \n",
       "9     ~                                          \n",
       "\n",
       "[10 rows x 22 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words_df = pd.DataFrame(lemm_l)\n",
    "words_df = words_df.fillna('')   # replace NaN (Not a Number) with empty string\n",
    "words_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['lang', 'form', 'delim', 'gdl', 'pos', 'id_word', 'label', 'id_text',\n",
       "       'cf', 'gw', 'sense', 'norm0', 'epos', 'base', 'morph', 'cont', 'extent',\n",
       "       'scope', 'state', 'aform', 'stem', 'norm'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Remove Spaces and Commas from Guide Word and Sense\n",
    "Spaces and commas in Guide Word and Sense may cause trouble in computational methods in tokenization, or when saved in Comma Separated Values format. All spaces and commas are replaced by hyphens and nothing (empty string), respectively.\n",
    "\n",
    "By default the `replace()` function in `pandas` will match the entire string (that is, \"lugal\" matches \"lugal\" but there is no match between \"l\" and \"lugal\"). In order to match partial strings the parameter `regex` must be set to `True`.\n",
    "\n",
    "The `replace()` function takes a nested dictionary as argument. The top-level keys identify the columns on which the `replace()` function should operate (in this case 'gw' and 'sense'). The value of each key is another dictionary with the search string as key and the replace string as value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "findreplace = {' ' : '-', ',' : ''}\n",
    "words_df = words_df.replace({'gw' : findreplace, 'sense' : findreplace}, regex=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The columns in the resulting DataFrame correspond to the elements of a full [ORACC](http://oracc.org) signature, plus information about text, line, and word ids:\n",
    "* base (Sumerian only)\n",
    "* cf (Citation Form)\n",
    "* cont (continuation of the base; Sumerian only)\n",
    "* epos (Effective Part of Speech)\n",
    "* form (transliteration, omitting all flags such as indication of breakage)\n",
    "* frag (transliteration; including flags)\n",
    "* gdl_utf8 (cuneiform)\n",
    "* gw (Guide Word: main or first translation in standard dictionary)\n",
    "* id_text (six-digit P, Q, or X number)\n",
    "* id_word (word ID in the format Text_ID.Line_ID.Word_ID)\n",
    "* label (traditional line number in the form o ii 2' (obverse column 2 line 2'), etc.)\n",
    "* lang (language code, including sux, sux-x-emegir, sux-x-emesal, akk, akk-x-stdbab, etc)\n",
    "* morph (Morphology; Sumerian only)\n",
    "* norm (Normalization: Akkadian)\n",
    "* norm0 (Normalization: Sumerian)\n",
    "* pos (Part of Speech)\n",
    "* sense (contextual meaning)\n",
    "* sig (full ORACC signature)\n",
    "\n",
    "Not all data elements (columns) are available for all words. Sumerian words never have a `norm`, Akkadian words do not have `norm0`, `base`, `cont`, or `morph`. Most data elements are only present when the word is lemmatized; only `lang`, `form`, `id_word`, and `id_text` should always be there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['lang', 'form', 'delim', 'gdl', 'pos', 'id_word', 'label', 'id_text',\n",
       "       'cf', 'gw', 'sense', 'norm0', 'epos', 'base', 'morph', 'cont', 'extent',\n",
       "       'scope', 'state', 'aform', 'stem', 'norm'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lang</th>\n",
       "      <th>form</th>\n",
       "      <th>delim</th>\n",
       "      <th>gdl</th>\n",
       "      <th>pos</th>\n",
       "      <th>id_word</th>\n",
       "      <th>label</th>\n",
       "      <th>id_text</th>\n",
       "      <th>cf</th>\n",
       "      <th>gw</th>\n",
       "      <th>...</th>\n",
       "      <th>epos</th>\n",
       "      <th>base</th>\n",
       "      <th>morph</th>\n",
       "      <th>cont</th>\n",
       "      <th>extent</th>\n",
       "      <th>scope</th>\n",
       "      <th>state</th>\n",
       "      <th>aform</th>\n",
       "      <th>stem</th>\n",
       "      <th>norm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>0 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [lang, form, delim, gdl, pos, id_word, label, id_text, cf, gw, sense, norm0, epos, base, morph, cont, extent, scope, state, aform, stem, norm]\n",
       "Index: []\n",
       "\n",
       "[0 rows x 22 columns]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words_df[words_df['norm'].str.contains('yn')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Line ID\n",
    "The DataFrame currently has a word-by-word data representation. We will add to each word a field `id_line` that will make it possible to reconstruct lines. This newly created field `id_line` is different from a traditional line number (found in the field \"label\") in two ways. First, id_line is an integer, so that lines are sorted correctly. Second, `id_line` is assigned to words, but also to gaps and horizontal drawings on the tablet. The field `id_line` will allow us to keep all these elements in their proper order.\n",
    "\n",
    "The field \"id_line\" is created by splitting the field \"id_word\" into (two or) three elements. The format of \"id_word\" is IDtext.line.word. The middle part, id_line, is selected and its data type is changed from string to integer. Rows that represent gaps in the text or horizontal drawings have an \"id_word\" in the format IDtext.line (consisting of only two elements), but are treated in exactly the same way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lang</th>\n",
       "      <th>form</th>\n",
       "      <th>delim</th>\n",
       "      <th>gdl</th>\n",
       "      <th>pos</th>\n",
       "      <th>id_word</th>\n",
       "      <th>label</th>\n",
       "      <th>id_text</th>\n",
       "      <th>cf</th>\n",
       "      <th>gw</th>\n",
       "      <th>...</th>\n",
       "      <th>base</th>\n",
       "      <th>morph</th>\n",
       "      <th>cont</th>\n",
       "      <th>extent</th>\n",
       "      <th>scope</th>\n",
       "      <th>state</th>\n",
       "      <th>aform</th>\n",
       "      <th>stem</th>\n",
       "      <th>norm</th>\n",
       "      <th>id_line</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>sux</td>\n",
       "      <td>1(geš₂)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '1(geš₂)', 'id': 'P113959....</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.1</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>sux</td>\n",
       "      <td>1(u)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.2...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.2</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>sux</td>\n",
       "      <td>6(aš)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '6(aš)', 'id': 'P113959.3....</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.3</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>sux</td>\n",
       "      <td>4(barig)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '4(barig)', 'id': 'P113959...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.4</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>sux</td>\n",
       "      <td>3(ban₂)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '3(ban₂)', 'id': 'P113959....</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.5</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>sux</td>\n",
       "      <td>4(diš)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '4(diš)', 'id': 'P113959.3...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.6</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>sux</td>\n",
       "      <td>sila₃</td>\n",
       "      <td></td>\n",
       "      <td>[{'v': 'sila₃', 'id': 'P113959.3.7.0'}]</td>\n",
       "      <td>N</td>\n",
       "      <td>P113959.3.7</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>sila</td>\n",
       "      <td>unit</td>\n",
       "      <td>...</td>\n",
       "      <td>sila₃</td>\n",
       "      <td>~</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>sux</td>\n",
       "      <td>1(u)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.8...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.8</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>sux</td>\n",
       "      <td>gin₂</td>\n",
       "      <td></td>\n",
       "      <td>[{'v': 'gin₂', 'id': 'P113959.3.9.0'}]</td>\n",
       "      <td>N</td>\n",
       "      <td>P113959.3.9</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>giŋ</td>\n",
       "      <td>unit</td>\n",
       "      <td>...</td>\n",
       "      <td>giŋ₄</td>\n",
       "      <td>~</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>sux</td>\n",
       "      <td>še</td>\n",
       "      <td></td>\n",
       "      <td>[{'v': 'še', 'id': 'P113959.3.10.0'}]</td>\n",
       "      <td>N</td>\n",
       "      <td>P113959.3.10</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>še</td>\n",
       "      <td>barley</td>\n",
       "      <td>...</td>\n",
       "      <td>še</td>\n",
       "      <td>~</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  lang      form delim                                                gdl pos  \\\n",
       "0  sux   1(geš₂)        [{'n': 'n', 'form': '1(geš₂)', 'id': 'P113959....   n   \n",
       "1  sux      1(u)        [{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.2...   n   \n",
       "2  sux     6(aš)        [{'n': 'n', 'form': '6(aš)', 'id': 'P113959.3....   n   \n",
       "3  sux  4(barig)        [{'n': 'n', 'form': '4(barig)', 'id': 'P113959...   n   \n",
       "4  sux   3(ban₂)        [{'n': 'n', 'form': '3(ban₂)', 'id': 'P113959....   n   \n",
       "5  sux    4(diš)        [{'n': 'n', 'form': '4(diš)', 'id': 'P113959.3...   n   \n",
       "6  sux     sila₃                  [{'v': 'sila₃', 'id': 'P113959.3.7.0'}]   N   \n",
       "7  sux      1(u)        [{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.8...   n   \n",
       "8  sux      gin₂                   [{'v': 'gin₂', 'id': 'P113959.3.9.0'}]   N   \n",
       "9  sux        še                    [{'v': 'še', 'id': 'P113959.3.10.0'}]   N   \n",
       "\n",
       "        id_word label                  id_text    cf      gw  ...   base  \\\n",
       "0   P113959.3.1   o 1  epsd2/admin/ur3/P113959                ...          \n",
       "1   P113959.3.2   o 1  epsd2/admin/ur3/P113959                ...          \n",
       "2   P113959.3.3   o 1  epsd2/admin/ur3/P113959                ...          \n",
       "3   P113959.3.4   o 1  epsd2/admin/ur3/P113959                ...          \n",
       "4   P113959.3.5   o 1  epsd2/admin/ur3/P113959                ...          \n",
       "5   P113959.3.6   o 1  epsd2/admin/ur3/P113959                ...          \n",
       "6   P113959.3.7   o 1  epsd2/admin/ur3/P113959  sila    unit  ...  sila₃   \n",
       "7   P113959.3.8   o 1  epsd2/admin/ur3/P113959                ...          \n",
       "8   P113959.3.9   o 1  epsd2/admin/ur3/P113959   giŋ    unit  ...   giŋ₄   \n",
       "9  P113959.3.10   o 1  epsd2/admin/ur3/P113959    še  barley  ...     še   \n",
       "\n",
       "  morph cont extent scope state aform stem norm id_line  \n",
       "0                                                     3  \n",
       "1                                                     3  \n",
       "2                                                     3  \n",
       "3                                                     3  \n",
       "4                                                     3  \n",
       "5                                                     3  \n",
       "6     ~                                               3  \n",
       "7                                                     3  \n",
       "8     ~                                               3  \n",
       "9     ~                                               3  \n",
       "\n",
       "[10 rows x 23 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words_df['id_line'] = [int(wordid.split('.')[1]) for wordid in words_df['id_word']]\n",
    "words_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['lang', 'form', 'delim', 'gdl', 'pos', 'id_word', 'label', 'id_text',\n",
       "       'cf', 'gw', 'sense', 'norm0', 'epos', 'base', 'morph', 'cont', 'extent',\n",
       "       'scope', 'state', 'aform', 'stem', 'norm'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lang</th>\n",
       "      <th>form</th>\n",
       "      <th>delim</th>\n",
       "      <th>gdl</th>\n",
       "      <th>pos</th>\n",
       "      <th>id_word</th>\n",
       "      <th>label</th>\n",
       "      <th>id_text</th>\n",
       "      <th>cf</th>\n",
       "      <th>gw</th>\n",
       "      <th>...</th>\n",
       "      <th>epos</th>\n",
       "      <th>base</th>\n",
       "      <th>morph</th>\n",
       "      <th>cont</th>\n",
       "      <th>extent</th>\n",
       "      <th>scope</th>\n",
       "      <th>state</th>\n",
       "      <th>aform</th>\n",
       "      <th>stem</th>\n",
       "      <th>norm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>sux</td>\n",
       "      <td>1(geš₂)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '1(geš₂)', 'id': 'P113959....</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.1</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>sux</td>\n",
       "      <td>1(u)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.2...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.2</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>sux</td>\n",
       "      <td>6(aš)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '6(aš)', 'id': 'P113959.3....</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.3</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>sux</td>\n",
       "      <td>4(barig)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '4(barig)', 'id': 'P113959...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.4</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>sux</td>\n",
       "      <td>3(ban₂)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '3(ban₂)', 'id': 'P113959....</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.5</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>sux</td>\n",
       "      <td>4(diš)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '4(diš)', 'id': 'P113959.3...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.6</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>sux</td>\n",
       "      <td>sila₃</td>\n",
       "      <td></td>\n",
       "      <td>[{'v': 'sila₃', 'id': 'P113959.3.7.0'}]</td>\n",
       "      <td>N</td>\n",
       "      <td>P113959.3.7</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>sila</td>\n",
       "      <td>unit</td>\n",
       "      <td>...</td>\n",
       "      <td>N</td>\n",
       "      <td>sila₃</td>\n",
       "      <td>~</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>sux</td>\n",
       "      <td>1(u)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.8...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.8</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>sux</td>\n",
       "      <td>gin₂</td>\n",
       "      <td></td>\n",
       "      <td>[{'v': 'gin₂', 'id': 'P113959.3.9.0'}]</td>\n",
       "      <td>N</td>\n",
       "      <td>P113959.3.9</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>giŋ</td>\n",
       "      <td>unit</td>\n",
       "      <td>...</td>\n",
       "      <td>N</td>\n",
       "      <td>giŋ₄</td>\n",
       "      <td>~</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>sux</td>\n",
       "      <td>še</td>\n",
       "      <td></td>\n",
       "      <td>[{'v': 'še', 'id': 'P113959.3.10.0'}]</td>\n",
       "      <td>N</td>\n",
       "      <td>P113959.3.10</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>še</td>\n",
       "      <td>barley</td>\n",
       "      <td>...</td>\n",
       "      <td>N</td>\n",
       "      <td>še</td>\n",
       "      <td>~</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  lang      form delim                                                gdl pos  \\\n",
       "0  sux   1(geš₂)        [{'n': 'n', 'form': '1(geš₂)', 'id': 'P113959....   n   \n",
       "1  sux      1(u)        [{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.2...   n   \n",
       "2  sux     6(aš)        [{'n': 'n', 'form': '6(aš)', 'id': 'P113959.3....   n   \n",
       "3  sux  4(barig)        [{'n': 'n', 'form': '4(barig)', 'id': 'P113959...   n   \n",
       "4  sux   3(ban₂)        [{'n': 'n', 'form': '3(ban₂)', 'id': 'P113959....   n   \n",
       "5  sux    4(diš)        [{'n': 'n', 'form': '4(diš)', 'id': 'P113959.3...   n   \n",
       "6  sux     sila₃                  [{'v': 'sila₃', 'id': 'P113959.3.7.0'}]   N   \n",
       "7  sux      1(u)        [{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.8...   n   \n",
       "8  sux      gin₂                   [{'v': 'gin₂', 'id': 'P113959.3.9.0'}]   N   \n",
       "9  sux        še                    [{'v': 'še', 'id': 'P113959.3.10.0'}]   N   \n",
       "\n",
       "        id_word label                  id_text    cf      gw  ... epos   base  \\\n",
       "0   P113959.3.1   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "1   P113959.3.2   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "2   P113959.3.3   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "3   P113959.3.4   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "4   P113959.3.5   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "5   P113959.3.6   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "6   P113959.3.7   o 1  epsd2/admin/ur3/P113959  sila    unit  ...    N  sila₃   \n",
       "7   P113959.3.8   o 1  epsd2/admin/ur3/P113959                ...               \n",
       "8   P113959.3.9   o 1  epsd2/admin/ur3/P113959   giŋ    unit  ...    N   giŋ₄   \n",
       "9  P113959.3.10   o 1  epsd2/admin/ur3/P113959    še  barley  ...    N     še   \n",
       "\n",
       "  morph cont extent scope state aform stem norm  \n",
       "0                                                \n",
       "1                                                \n",
       "2                                                \n",
       "3                                                \n",
       "4                                                \n",
       "5                                                \n",
       "6     ~                                          \n",
       "7                                                \n",
       "8     ~                                          \n",
       "9     ~                                          \n",
       "\n",
       "[10 rows x 22 columns]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_df = words_df[words_df[\"gdl\"].notnull()]\n",
    "filtered_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['lang', 'form', 'delim', 'gdl', 'pos', 'id_word', 'label', 'id_text',\n",
       "       'cf', 'gw', 'sense', 'norm0', 'epos', 'base', 'morph', 'cont', 'extent',\n",
       "       'scope', 'state', 'aform', 'stem', 'norm'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lang</th>\n",
       "      <th>form</th>\n",
       "      <th>delim</th>\n",
       "      <th>gdl</th>\n",
       "      <th>pos</th>\n",
       "      <th>id_word</th>\n",
       "      <th>label</th>\n",
       "      <th>id_text</th>\n",
       "      <th>cf</th>\n",
       "      <th>gw</th>\n",
       "      <th>...</th>\n",
       "      <th>epos</th>\n",
       "      <th>base</th>\n",
       "      <th>morph</th>\n",
       "      <th>cont</th>\n",
       "      <th>extent</th>\n",
       "      <th>scope</th>\n",
       "      <th>state</th>\n",
       "      <th>aform</th>\n",
       "      <th>stem</th>\n",
       "      <th>norm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>0 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [lang, form, delim, gdl, pos, id_word, label, id_text, cf, gw, sense, norm0, epos, base, morph, cont, extent, scope, state, aform, stem, norm]\n",
       "Index: []\n",
       "\n",
       "[0 rows x 22 columns]"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_df[filtered_df['norm'].str.contains('YN', na = False)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 Create Lemma Column\n",
    "A lemma, [ORACC](http://oracc.org) style, combines Citation Form, GuideWord and POS into a unique reference to one particular lemma in a standard dictionary, as in `lugal[king]N` (Sumerian) or `šarru[king]N`. \n",
    "\n",
    "Proper Nouns (proper names, geographical names, etc.) are a special case, because they currently receive a Part of Speech, but not a Citation Form or Guide Word.\n",
    "\n",
    "Usually, not all words in a text are lemmatized, because a word may be (partly) broken and/or unknown. Unlemmatized and unlemmatizable words will receive a place-holder lemmatization that consists of the transliteration of the word (instead of the Citation Form), with `NA` as GuideWord and `NA` as POS, as in `i-bu-x[NA]NA`. Note that `NA` is a string. \n",
    "\n",
    "Finally, rows representing horizontal rulings, blank lines, or broken lines have data in the fields 'state', 'scope', and 'extent' (for instance 'state' = broken, 'scope' = line, and 'extent' = 3, to indicate three broken lines). We can use this to prevent scripts to 'jump over' such breaks when looking for key words after (or before) a proper noun. We distinguish between physical breaks and logical breaks (such as horizontal rulings or seal impressions). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lang</th>\n",
       "      <th>form</th>\n",
       "      <th>delim</th>\n",
       "      <th>gdl</th>\n",
       "      <th>pos</th>\n",
       "      <th>id_word</th>\n",
       "      <th>label</th>\n",
       "      <th>id_text</th>\n",
       "      <th>cf</th>\n",
       "      <th>gw</th>\n",
       "      <th>...</th>\n",
       "      <th>morph</th>\n",
       "      <th>cont</th>\n",
       "      <th>extent</th>\n",
       "      <th>scope</th>\n",
       "      <th>state</th>\n",
       "      <th>aform</th>\n",
       "      <th>stem</th>\n",
       "      <th>norm</th>\n",
       "      <th>id_line</th>\n",
       "      <th>lemma</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>sux</td>\n",
       "      <td>1(geš₂)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '1(geš₂)', 'id': 'P113959....</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.1</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "      <td>1(geš₂)[]NU</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>sux</td>\n",
       "      <td>1(u)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.2...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.2</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "      <td>1(u)[]NU</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>sux</td>\n",
       "      <td>6(aš)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '6(aš)', 'id': 'P113959.3....</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.3</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "      <td>6(aš)[]NU</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>sux</td>\n",
       "      <td>4(barig)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '4(barig)', 'id': 'P113959...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.4</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "      <td>4(barig)[]NU</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>sux</td>\n",
       "      <td>3(ban₂)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '3(ban₂)', 'id': 'P113959....</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.5</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "      <td>3(ban₂)[]NU</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>sux</td>\n",
       "      <td>4(diš)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '4(diš)', 'id': 'P113959.3...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.6</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "      <td>4(diš)[]NU</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>sux</td>\n",
       "      <td>sila₃</td>\n",
       "      <td></td>\n",
       "      <td>[{'v': 'sila₃', 'id': 'P113959.3.7.0'}]</td>\n",
       "      <td>N</td>\n",
       "      <td>P113959.3.7</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>sila</td>\n",
       "      <td>unit</td>\n",
       "      <td>...</td>\n",
       "      <td>~</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "      <td>sila[unit]N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>sux</td>\n",
       "      <td>1(u)</td>\n",
       "      <td></td>\n",
       "      <td>[{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.8...</td>\n",
       "      <td>n</td>\n",
       "      <td>P113959.3.8</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "      <td>1(u)[]NU</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>sux</td>\n",
       "      <td>gin₂</td>\n",
       "      <td></td>\n",
       "      <td>[{'v': 'gin₂', 'id': 'P113959.3.9.0'}]</td>\n",
       "      <td>N</td>\n",
       "      <td>P113959.3.9</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>giŋ</td>\n",
       "      <td>unit</td>\n",
       "      <td>...</td>\n",
       "      <td>~</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "      <td>giŋ[unit]N</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>sux</td>\n",
       "      <td>še</td>\n",
       "      <td></td>\n",
       "      <td>[{'v': 'še', 'id': 'P113959.3.10.0'}]</td>\n",
       "      <td>N</td>\n",
       "      <td>P113959.3.10</td>\n",
       "      <td>o 1</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>še</td>\n",
       "      <td>barley</td>\n",
       "      <td>...</td>\n",
       "      <td>~</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>3</td>\n",
       "      <td>še[barley]N</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  lang      form delim                                                gdl pos  \\\n",
       "0  sux   1(geš₂)        [{'n': 'n', 'form': '1(geš₂)', 'id': 'P113959....   n   \n",
       "1  sux      1(u)        [{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.2...   n   \n",
       "2  sux     6(aš)        [{'n': 'n', 'form': '6(aš)', 'id': 'P113959.3....   n   \n",
       "3  sux  4(barig)        [{'n': 'n', 'form': '4(barig)', 'id': 'P113959...   n   \n",
       "4  sux   3(ban₂)        [{'n': 'n', 'form': '3(ban₂)', 'id': 'P113959....   n   \n",
       "5  sux    4(diš)        [{'n': 'n', 'form': '4(diš)', 'id': 'P113959.3...   n   \n",
       "6  sux     sila₃                  [{'v': 'sila₃', 'id': 'P113959.3.7.0'}]   N   \n",
       "7  sux      1(u)        [{'n': 'n', 'form': '1(u)', 'id': 'P113959.3.8...   n   \n",
       "8  sux      gin₂                   [{'v': 'gin₂', 'id': 'P113959.3.9.0'}]   N   \n",
       "9  sux        še                    [{'v': 'še', 'id': 'P113959.3.10.0'}]   N   \n",
       "\n",
       "        id_word label                  id_text    cf      gw  ... morph cont  \\\n",
       "0   P113959.3.1   o 1  epsd2/admin/ur3/P113959                ...              \n",
       "1   P113959.3.2   o 1  epsd2/admin/ur3/P113959                ...              \n",
       "2   P113959.3.3   o 1  epsd2/admin/ur3/P113959                ...              \n",
       "3   P113959.3.4   o 1  epsd2/admin/ur3/P113959                ...              \n",
       "4   P113959.3.5   o 1  epsd2/admin/ur3/P113959                ...              \n",
       "5   P113959.3.6   o 1  epsd2/admin/ur3/P113959                ...              \n",
       "6   P113959.3.7   o 1  epsd2/admin/ur3/P113959  sila    unit  ...     ~        \n",
       "7   P113959.3.8   o 1  epsd2/admin/ur3/P113959                ...              \n",
       "8   P113959.3.9   o 1  epsd2/admin/ur3/P113959   giŋ    unit  ...     ~        \n",
       "9  P113959.3.10   o 1  epsd2/admin/ur3/P113959    še  barley  ...     ~        \n",
       "\n",
       "  extent scope state aform stem norm id_line         lemma  \n",
       "0                                          3   1(geš₂)[]NU  \n",
       "1                                          3      1(u)[]NU  \n",
       "2                                          3     6(aš)[]NU  \n",
       "3                                          3  4(barig)[]NU  \n",
       "4                                          3   3(ban₂)[]NU  \n",
       "5                                          3    4(diš)[]NU  \n",
       "6                                          3   sila[unit]N  \n",
       "7                                          3      1(u)[]NU  \n",
       "8                                          3    giŋ[unit]N  \n",
       "9                                          3   še[barley]N  \n",
       "\n",
       "[10 rows x 24 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "proper_nouns = ['FN', 'PN', 'DN', 'AN', 'WN', 'ON', 'TN', 'MN', 'CN', 'GN']\n",
    "physical_break = ['illegible', 'traces', 'missing', 'effaced']\n",
    "logical_break = ['other', 'blank', 'ruling']\n",
    "words_df['lemma'] = words_df[\"cf\"] + '[' + words_df[\"gw\"] + ']' + words_df[\"pos\"]\n",
    "words_df.loc[words_df[\"cf\"] == \"\" , 'lemma'] = words_df['form'] + '[NA]NA'\n",
    "words_df.loc[words_df[\"pos\"] == \"n\" , 'lemma'] = words_df['form'] + '[]NU'\n",
    "words_df.loc[words_df[\"pos\"].isin(proper_nouns) , 'lemma'] = words_df['form'] + '[]' + words_df['pos']\n",
    "words_df.loc[words_df[\"state\"].isin(logical_break), 'lemma'] = \"break_logical\"\n",
    "words_df.loc[words_df[\"state\"].isin(physical_break), 'lemma'] = \"break_physical\"\n",
    "words_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5 Select Relevant Columns\n",
    "Now we can select only the field 'lemma' and the fields that indicate the ID of the word, the line, or the document, plus the field 'label', which indicates the physical position of the line on the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lemma</th>\n",
       "      <th>id_text</th>\n",
       "      <th>id_line</th>\n",
       "      <th>id_word</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1(geš₂)[]NU</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.1</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1(u)[]NU</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.2</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>6(aš)[]NU</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.3</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4(barig)[]NU</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.4</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3(ban₂)[]NU</td>\n",
       "      <td>epsd2/admin/ur3/P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.5</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>95</td>\n",
       "      <td>nibru{ki}[]GN</td>\n",
       "      <td>epsd2/admin/ur3/P125538</td>\n",
       "      <td>10</td>\n",
       "      <td>P125538.10.3</td>\n",
       "      <td>r 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>96</td>\n",
       "      <td>zabardab[official]N</td>\n",
       "      <td>epsd2/admin/ur3/P125538</td>\n",
       "      <td>11</td>\n",
       "      <td>P125538.11.1</td>\n",
       "      <td>r 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>97</td>\n",
       "      <td>maškim[administrator]N</td>\n",
       "      <td>epsd2/admin/ur3/P125538</td>\n",
       "      <td>11</td>\n",
       "      <td>P125538.11.2</td>\n",
       "      <td>r 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>98</td>\n",
       "      <td>ud[sun]N</td>\n",
       "      <td>epsd2/admin/ur3/P125538</td>\n",
       "      <td>12</td>\n",
       "      <td>P125538.12.1</td>\n",
       "      <td>r 3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>99</td>\n",
       "      <td>2(u)[]NU</td>\n",
       "      <td>epsd2/admin/ur3/P125538</td>\n",
       "      <td>12</td>\n",
       "      <td>P125538.12.2</td>\n",
       "      <td>r 3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     lemma                  id_text  id_line       id_word  \\\n",
       "0              1(geš₂)[]NU  epsd2/admin/ur3/P113959        3   P113959.3.1   \n",
       "1                 1(u)[]NU  epsd2/admin/ur3/P113959        3   P113959.3.2   \n",
       "2                6(aš)[]NU  epsd2/admin/ur3/P113959        3   P113959.3.3   \n",
       "3             4(barig)[]NU  epsd2/admin/ur3/P113959        3   P113959.3.4   \n",
       "4              3(ban₂)[]NU  epsd2/admin/ur3/P113959        3   P113959.3.5   \n",
       "..                     ...                      ...      ...           ...   \n",
       "95           nibru{ki}[]GN  epsd2/admin/ur3/P125538       10  P125538.10.3   \n",
       "96     zabardab[official]N  epsd2/admin/ur3/P125538       11  P125538.11.1   \n",
       "97  maškim[administrator]N  epsd2/admin/ur3/P125538       11  P125538.11.2   \n",
       "98                ud[sun]N  epsd2/admin/ur3/P125538       12  P125538.12.1   \n",
       "99                2(u)[]NU  epsd2/admin/ur3/P125538       12  P125538.12.2   \n",
       "\n",
       "   label  \n",
       "0    o 1  \n",
       "1    o 1  \n",
       "2    o 1  \n",
       "3    o 1  \n",
       "4    o 1  \n",
       "..   ...  \n",
       "95   r 1  \n",
       "96   r 2  \n",
       "97   r 2  \n",
       "98   r 3  \n",
       "99   r 3  \n",
       "\n",
       "[100 rows x 5 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = ['lemma', 'id_text', 'id_line', 'id_word', 'label']\n",
    "words_df = words_df[cols].copy()\n",
    "words_df.head(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can simplify the 'id_text' column, because all documents derive from the same project (epsd2/admnin/u3adm)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lemma</th>\n",
       "      <th>id_text</th>\n",
       "      <th>id_line</th>\n",
       "      <th>id_word</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1(geš₂)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.1</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1(u)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.2</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>6(aš)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.3</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4(barig)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.4</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3(ban₂)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.5</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>4(diš)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.6</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>sila[unit]N</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.7</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1(u)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.8</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>giŋ[unit]N</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.9</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>še[barley]N</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.10</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>gur[unit]N</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.11</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>8(aš)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>4</td>\n",
       "      <td>P113959.4.1</td>\n",
       "      <td>o 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2(barig)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>4</td>\n",
       "      <td>P113959.4.2</td>\n",
       "      <td>o 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>ziznumun[seed]N</td>\n",
       "      <td>P113959</td>\n",
       "      <td>4</td>\n",
       "      <td>P113959.4.3</td>\n",
       "      <td>o 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>gur[unit]N</td>\n",
       "      <td>P113959</td>\n",
       "      <td>4</td>\n",
       "      <td>P113959.4.4</td>\n",
       "      <td>o 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>2(barig)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>5</td>\n",
       "      <td>P113959.5.1</td>\n",
       "      <td>o 3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>3(ban₂)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>5</td>\n",
       "      <td>P113959.5.2</td>\n",
       "      <td>o 3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>kibnumun[wheat]N</td>\n",
       "      <td>P113959</td>\n",
       "      <td>5</td>\n",
       "      <td>P113959.5.3</td>\n",
       "      <td>o 3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>šenumun[seed]N</td>\n",
       "      <td>P113959</td>\n",
       "      <td>6</td>\n",
       "      <td>P113959.6.1</td>\n",
       "      <td>o 4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>murgu[fodder]N</td>\n",
       "      <td>P113959</td>\n",
       "      <td>6</td>\n",
       "      <td>P113959.6.2</td>\n",
       "      <td>o 4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               lemma  id_text  id_line       id_word label\n",
       "0        1(geš₂)[]NU  P113959        3   P113959.3.1   o 1\n",
       "1           1(u)[]NU  P113959        3   P113959.3.2   o 1\n",
       "2          6(aš)[]NU  P113959        3   P113959.3.3   o 1\n",
       "3       4(barig)[]NU  P113959        3   P113959.3.4   o 1\n",
       "4        3(ban₂)[]NU  P113959        3   P113959.3.5   o 1\n",
       "5         4(diš)[]NU  P113959        3   P113959.3.6   o 1\n",
       "6        sila[unit]N  P113959        3   P113959.3.7   o 1\n",
       "7           1(u)[]NU  P113959        3   P113959.3.8   o 1\n",
       "8         giŋ[unit]N  P113959        3   P113959.3.9   o 1\n",
       "9        še[barley]N  P113959        3  P113959.3.10   o 1\n",
       "10        gur[unit]N  P113959        3  P113959.3.11   o 1\n",
       "11         8(aš)[]NU  P113959        4   P113959.4.1   o 2\n",
       "12      2(barig)[]NU  P113959        4   P113959.4.2   o 2\n",
       "13   ziznumun[seed]N  P113959        4   P113959.4.3   o 2\n",
       "14        gur[unit]N  P113959        4   P113959.4.4   o 2\n",
       "15      2(barig)[]NU  P113959        5   P113959.5.1   o 3\n",
       "16       3(ban₂)[]NU  P113959        5   P113959.5.2   o 3\n",
       "17  kibnumun[wheat]N  P113959        5   P113959.5.3   o 3\n",
       "18    šenumun[seed]N  P113959        6   P113959.6.1   o 4\n",
       "19    murgu[fodder]N  P113959        6   P113959.6.2   o 4"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words_df['id_text'] = [tid[-7:] for tid in words_df['id_text']]\n",
    "words_df.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lemma</th>\n",
       "      <th>id_text</th>\n",
       "      <th>id_line</th>\n",
       "      <th>id_word</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1(geš₂)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.1</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1(u)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.2</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>6(aš)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.3</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4(barig)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.4</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3(ban₂)[]NU</td>\n",
       "      <td>P113959</td>\n",
       "      <td>3</td>\n",
       "      <td>P113959.3.5</td>\n",
       "      <td>o 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3993</td>\n",
       "      <td>kišib[seal]N</td>\n",
       "      <td>P111437</td>\n",
       "      <td>9</td>\n",
       "      <td>P111437.9.1</td>\n",
       "      <td>r 5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3994</td>\n",
       "      <td>ur-{d}ba-ba₆-ka[]PN</td>\n",
       "      <td>P111437</td>\n",
       "      <td>9</td>\n",
       "      <td>P111437.9.2</td>\n",
       "      <td>r 5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3997</td>\n",
       "      <td>4(u)[]NU</td>\n",
       "      <td>P200789</td>\n",
       "      <td>4</td>\n",
       "      <td>P200789.4.1</td>\n",
       "      <td>o i 10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3998</td>\n",
       "      <td>6(diš)[]NU</td>\n",
       "      <td>P200789</td>\n",
       "      <td>4</td>\n",
       "      <td>P200789.4.2</td>\n",
       "      <td>o i 10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3999</td>\n",
       "      <td>1/2(diš)[]NU</td>\n",
       "      <td>P200789</td>\n",
       "      <td>4</td>\n",
       "      <td>P200789.4.3</td>\n",
       "      <td>o i 10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3577 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                    lemma  id_text  id_line      id_word   label\n",
       "0             1(geš₂)[]NU  P113959        3  P113959.3.1     o 1\n",
       "1                1(u)[]NU  P113959        3  P113959.3.2     o 1\n",
       "2               6(aš)[]NU  P113959        3  P113959.3.3     o 1\n",
       "3            4(barig)[]NU  P113959        3  P113959.3.4     o 1\n",
       "4             3(ban₂)[]NU  P113959        3  P113959.3.5     o 1\n",
       "...                   ...      ...      ...          ...     ...\n",
       "3993         kišib[seal]N  P111437        9  P111437.9.1     r 5\n",
       "3994  ur-{d}ba-ba₆-ka[]PN  P111437        9  P111437.9.2     r 5\n",
       "3997             4(u)[]NU  P200789        4  P200789.4.1  o i 10\n",
       "3998           6(diš)[]NU  P200789        4  P200789.4.2  o i 10\n",
       "3999         1/2(diš)[]NU  P200789        4  P200789.4.3  o i 10\n",
       "\n",
       "[3577 rows x 5 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = words_df.head(4000)\n",
    "test[test['lemma'].str.contains('N')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6 Save Results in CSV file or in Pickle\n",
    "The output file is called `parsed.csv` and is placed in the directory `output`. In most computers, `csv` files open automatically in Excel. This program does not deal well with `utf-8` encoding (files in `utf-8` need to be imported; see the instructions [here](https://www.itg.ias.edu/content/how-import-csv-file-uses-utf-8-character-encoding-0). If you intend to use the file in Excel, change `encoding ='utf-8'` to `encoding='utf-16'`. For usage in computational text analysis applications `utf-8` is usually preferred. \n",
    "\n",
    "The Pandas function `to_pickle()` writes a binary file that can be opened in a later phase of the project with the `read_pickle()` command and will reproduce exactly the same DataFrame with the same data structure. The resulting file cannot be used in other programs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "savefile =  'parsed.csv'\n",
    "with open('output/' + savefile, 'w', encoding=\"utf-8\") as w:\n",
    "    words_df.to_csv(w, index=False)\n",
    "pickled = \"output/parsed.p\"\n",
    "words_df.to_pickle(pickled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "widgets": {
   "state": {
    "1ab0b1b79a3a44338a62bf030ec6e1cf": {
     "views": [
      {
       "cell_index": 11
      }
     ]
    },
    "2a77bd83bf804c21b2db18a0537abbf4": {
     "views": [
      {
       "cell_index": 7
      }
     ]
    },
    "7bb2968f39ca4dbcbf636b457d10da77": {
     "views": [
      {
       "cell_index": 7
      }
     ]
    },
    "9a4924badbaa4aa893e0ae18d65b0df4": {
     "views": [
      {
       "cell_index": 11
      }
     ]
    },
    "a8c9c60f02e340a980e0a74f2cd4d437": {
     "views": [
      {
       "cell_index": 11
      }
     ]
    },
    "b7bad46fe1144cff8a64fbb5987c6f58": {
     "views": [
      {
       "cell_index": 7
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}