egyptiandatasetgenerator.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "interpreter": {
      "hash": "83355747db63703de1c2ac4de6fccd06216fd8d2c661a8e64bf54486265eb7b3"
    },
    "kernelspec": {
      "display_name": "Python 3.9.0 64-bit",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": ""
    },
    "metadata": {
      "interpreter": {
        "hash": "83355747db63703de1c2ac4de6fccd06216fd8d2c661a8e64bf54486265eb7b3"
      }
    },
    "colab": {
      "name": "egyptiandatasetgenerator.ipynb",
      "provenance": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Ol8vueleSnS"
      },
      "source": [
        "# Dataset Generation for Egyptian-English Neural Machine Translation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MEogyl7GeSnV"
      },
      "source": [
        "# Generate the aligned corpora"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R4zOO-IReSnW"
      },
      "source": [
        "We will use 3 types of files to build our dataset for semi-supervised machine translation.\n",
        "- .txt files containing aligned examples from Egyptian grammars\n",
        "- A series of files compiled by Mark-Jan Nederhof for his hieroglyphic display and input program PhilologEg\n",
        "- A monolingual corpus taken from [a site dedicated to Egyptian transliterations](http://www.egyptomaniak.gr/Egyptian%20Texts.htm)\n",
        "\n",
        "The first two file types are separately preprocessed to account for formatting differences, and then merged to form a small aligned corpus of around 8,000 sentences. The monolingual corpus adds an additional 50,000 sentences to improve translation accuracy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U4gRd_vCeSnW"
      },
      "source": [
        "## 1. Parse .txt Files from Grammars"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KtrmuxIaelXQ",
        "outputId": "40ce5539-aac6-4094-deb4-ce8035677a53"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2PlA1xOieymq",
        "outputId": "bcf8a89a-8dac-4c6d-c7d0-147d1225d3fe"
      },
      "source": [
        "%cd /content/drive/MyDrive/EgyptianTranslation"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/drive/MyDrive/EgyptianTranslation\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "le2bRfxJeSnW"
      },
      "source": [
        "try:\n",
        "    import pandas as pd\n",
        "except:\n",
        "    !pip install pandas\n",
        "    import pandas as pd\n",
        "import string\n",
        "pd.set_option('display.width', None)\n",
        "pd.set_option('display.max_colwidth', None)"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "_cell_guid": "d859ecf5-0f5e-4ee5-8d15-9a038a872df1",
        "_uuid": "8c8ea0de-d536-47e3-bcba-4daaffd48668",
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:37.488198Z",
          "iopub.status.busy": "2021-05-30T06:44:37.487847Z",
          "iopub.status.idle": "2021-05-30T06:44:37.491841Z",
          "shell.execute_reply": "2021-05-30T06:44:37.491185Z",
          "shell.execute_reply.started": "2021-05-30T06:44:37.48817Z"
        },
        "trusted": true,
        "id": "y6UC5_DHeSnX"
      },
      "source": [
        "import re\n",
        "txt_files = ['https://mjn.host.cs.st-andrews.ac.uk/egyptian/grammars/Allen.txt',\n",
        "             'https://mjn.host.cs.st-andrews.ac.uk/egyptian/grammars/Callender.txt',\n",
        "             'https://mjn.host.cs.st-andrews.ac.uk/egyptian/grammars/Gardiner.txt',\n",
        "             'https://mjn.host.cs.st-andrews.ac.uk/egyptian/grammars/Loprieno.txt',\n",
        "             'https://mjn.host.cs.st-andrews.ac.uk/egyptian/grammars/Ockinga.txt']"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6vjP5rIpeSnY"
      },
      "source": [
        "### Imports each .txt file and converts it to a series of AlignedSentences\n",
        "- Split imported files into paragraphs\n",
        "- Filters paragraphs by whether they contain both \"transliteration\" and \"translation\" keys\n",
        "- Constructs dictionaries with only the necessary keys\n",
        "- Constructs AlignedSentence objects from `{ '%tr' : \"...\", '%al' : \"...\" }` dictionaries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:37.493484Z",
          "iopub.status.busy": "2021-05-30T06:44:37.493207Z",
          "iopub.status.idle": "2021-05-30T06:44:37.512992Z",
          "shell.execute_reply": "2021-05-30T06:44:37.511947Z",
          "shell.execute_reply.started": "2021-05-30T06:44:37.493458Z"
        },
        "trusted": true,
        "id": "yOYNNHpSeSnY"
      },
      "source": [
        "class AlignedSentence(object):\n",
        "    def __init__(self, translit, translate, signs = None, from_file = None):\n",
        "        self.translit = translit\n",
        "        self.translate = translate \n",
        "        self.signs = signs\n",
        "        self.from_file = from_file\n",
        "        \n",
        "    def __hash__(self):\n",
        "        return hash((self.translit, self.translate))\n",
        "\n",
        "    def __eq__(self, other):\n",
        "        if not isinstance(other, type(self)): return NotImplemented\n",
        "        return self.translit == other.translit and self.translate == other.translate\n",
        "    \n",
        "    def __str__(self):\n",
        "        return \"Translit: {}, Translate: {}\".format(self.translit, self.translate)\n",
        "    \n",
        "    def str(self):\n",
        "        return \"Translit: {}, Translate: {}\".format(self.translit, self.translate)\n",
        "\n",
        "    def as_dict(self):\n",
        "        return {'Transliteration': self.translit, 'Translation': self.translate, 'File Name': self.from_file}"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "_cell_guid": "06d150af-7114-45e5-a942-28dfa44b1f7d",
        "_uuid": "ee3d2992-0e4f-4385-bb3f-ee37300b3848",
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:37.515263Z",
          "iopub.status.busy": "2021-05-30T06:44:37.514929Z",
          "iopub.status.idle": "2021-05-30T06:44:37.528578Z",
          "shell.execute_reply": "2021-05-30T06:44:37.527701Z",
          "shell.execute_reply.started": "2021-05-30T06:44:37.515234Z"
        },
        "trusted": true,
        "id": "Hb_c2LaHeSnY"
      },
      "source": [
        "import requests\n",
        "import os\n",
        "from os.path import exists\n",
        "\n",
        "def parse_text_file(file):\n",
        "    file_name = \"./texts/grammar-texts/\" + file.split('/')[-1]\n",
        "    if not exists(file_name):\n",
        "        res = requests.get(file)\n",
        "        with open(file_name, 'wb') as to_write:\n",
        "            to_write.write(res.content)\n",
        "            \n",
        "    with open(file_name, 'r') as opened_file:\n",
        "        all_text = opened_file.read()\n",
        "        split_para = all_text.split('\\n\\n')\n",
        "        with_tr_al = [item for item in split_para if '%al' in item and '%tr' in item]\n",
        "        return [parse_paragraph(item, file_name) for item in with_tr_al]\n",
        "            \n",
        "def parse_paragraph(paragraph, file_name):\n",
        "    split_by_header = paragraph.split('\\n')\n",
        "    merged = []\n",
        "    for i in range(len(split_by_header)):\n",
        "        if not split_by_header[i].startswith('%'):\n",
        "            merged[-1] += \" \" + split_by_header[i]\n",
        "        else:\n",
        "            merged.append(split_by_header[i])\n",
        "    as_dict = {item[:3].strip() : item[3:].strip() for item in merged if item.startswith('%al') or item.startswith('%tr')}\n",
        "    return AlignedSentence(as_dict['%al'], as_dict['%tr'], from_file = file_name)"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:37.530595Z",
          "iopub.status.busy": "2021-05-30T06:44:37.530196Z",
          "iopub.status.idle": "2021-05-30T06:44:37.54819Z",
          "shell.execute_reply": "2021-05-30T06:44:37.54743Z",
          "shell.execute_reply.started": "2021-05-30T06:44:37.530566Z"
        },
        "trusted": true,
        "id": "QBpDHHr4eSnZ"
      },
      "source": [
        "parsed = []\n",
        "for tf in txt_files:\n",
        "    parsed_f = parse_text_file(tf)\n",
        "    parsed.extend(parsed_f)"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "_kg_hide-input": false,
        "_kg_hide-output": false,
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:37.54951Z",
          "iopub.status.busy": "2021-05-30T06:44:37.549181Z",
          "iopub.status.idle": "2021-05-30T06:44:37.559558Z",
          "shell.execute_reply": "2021-05-30T06:44:37.558756Z",
          "shell.execute_reply.started": "2021-05-30T06:44:37.549474Z"
        },
        "trusted": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kC_iuOJseSnZ",
        "outputId": "38385428-b2f4-483f-e5d2-78c37396d349"
      },
      "source": [
        "for item in parsed[:10]:\n",
        "    print(item.translate)\n",
        "print(len(parsed))"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "I am one who says what is good and repeats what is loved.\n",
            "His Majesty appointed me scribe of the cadaster, and His Majesty showed me great favor.\n",
            "When it was read to me, I placed myself on my stomach...\n",
            "[You are the rudder of the entire land,] and how Egypt sails depends on what you command.\n",
            "[Eloquence is more hidden than the emerald,] although it is found with slave-girls at the mill-stones.\n",
            "behold, I am before thee.\n",
            "I placed myself on my belly.\n",
            "overseer of the house, i.e. steward.\n",
            "possessor of veneration, venerable.\n",
            "the king of Egypt.\n",
            "137\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "roK4UfaDeSna"
      },
      "source": [
        "### Preprocess Translation Fields\n",
        "\n",
        "- Keep the contents of <em\\> and <i\\> but remove other html tags' contents. Remove all tags themselves.\n",
        "- Remove square brackets & their contents\n",
        "- Remove parentheses (and contents where they explain an existing noun)\n",
        "- Remove all \"lit.\" and \"i.e.\" explanation clauses\n",
        "- Standardizes \"favour\" with \"favor\"\n",
        "- Standardize ellipsis length to 3\n",
        "- Standardize spacing by removing multiple contiguous spaces\n",
        "- Add spaces before unremoved punctuation (\"the cat's toy\" becomes \"the cat 's toy\")\n",
        "- Convert translation to lowercase"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:37.561345Z",
          "iopub.status.busy": "2021-05-30T06:44:37.560937Z",
          "iopub.status.idle": "2021-05-30T06:44:37.576492Z",
          "shell.execute_reply": "2021-05-30T06:44:37.575334Z",
          "shell.execute_reply.started": "2021-05-30T06:44:37.561305Z"
        },
        "trusted": true,
        "id": "KphQBaB8eSna"
      },
      "source": [
        "def preprocess_translation(parsed):\n",
        "    with_brackets = [item for item in parsed if '<' in item.translate or '>' in item.translate]\n",
        "\n",
        "    for item in with_brackets:\n",
        "        item.translate = re.sub(r\"\\<em\\>(.+)\\<\\/em\\>\", \"\\\\1\", item.translate)\n",
        "        item.translate = re.sub(r\"\\<i\\>(.+)\\<\\/i\\>\", \"\\\\1\", item.translate)\n",
        "        item.translate = re.sub(r\"\\<.+\\>(.+)\\<\\/.+\\>\", \"\", item.translate)\n",
        "        \n",
        "    with_lit = [item for item in parsed if 'lit' in item.translate.lower()]\n",
        "    for item in with_lit:\n",
        "        item2 = re.sub(r\"\\(\\s*\\)\", \"\", item.translate)\n",
        "        item3 = re.sub(r\"\\(lit\\.(.+?)\\)\", \"\", item2)\n",
        "        item4 = re.sub(r\"(,|\\.|\\!|\\?)\\s*(l|L)it\\..+(,|\\.|\\!|\\?)$\", \"\\\\3\", item3)\n",
        "        item4 = re.sub(r\"(,|\\.|\\!|\\?)\\s*(l|L)it\\..+$\", \"\", item3)\n",
        "        item.translate = item4\n",
        "    \n",
        "    with_paren = [item for item in parsed if '(' in item.translate or ')' in item.translate]\n",
        "    for item in with_paren:\n",
        "        item2 = re.sub(r\"(it) \\(.+?\\)\", \"\\\\1\", item.translate)\n",
        "        item3 = re.sub(r\"\\(=.+?\\)\", \"\", item2)\n",
        "        item4 = item3.replace(')', \"\").replace('(', '')\n",
        "        item.translate = item4\n",
        "\n",
        "    with_square = [item for item in parsed if '[' in item.translate or ']' in item.translate]\n",
        "    for item in with_square:\n",
        "        item.translate = re.sub(r\"\\[.+?\\]\\s*\", \"\", item.translate)\n",
        "        \n",
        "    with_ie = [item for item in parsed if 'i.e' in item.translate.lower()]\n",
        "    for item in with_ie:\n",
        "        item.translate = re.sub(\"(,|\\.) (i|I)\\.e\\..+$\", \".\", item.translate)\n",
        "\n",
        "    for item in parsed:\n",
        "        item.translate = item.translate.replace('favor', 'favour')\n",
        "        item.translate = ' '.join(item.translate.split())\n",
        "        item.translate = re.sub(r\"\\.{4,}\", r\"...\", item.translate)\n",
        "        item.translate = item.translate.replace('[...]', '...').replace('.', ' .').replace(' . . .', ' [...]').replace(':', '').replace(';', '')\n",
        "        item.translate = item.translate.replace(',', '').replace('???', '...').replace('?', '').replace(\"'\", \"\").replace('\"', \"\")\n",
        "        item.translate = re.sub(r'([^\\.\\.\\.\\]])\\]', \"\\\\1\", item.translate)\n",
        "        item.translate = re.sub(r'\\[([^\\.\\.\\.\\]])', \"\\\\1\", item.translate)\n",
        "        item.translate = ' '.join([x for x in item.translate.split() if x != '.' and len(x)])\n",
        "        item.translate = item.translate.lower()"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:37.578506Z",
          "iopub.status.busy": "2021-05-30T06:44:37.578078Z",
          "iopub.status.idle": "2021-05-30T06:44:37.592366Z",
          "shell.execute_reply": "2021-05-30T06:44:37.591461Z",
          "shell.execute_reply.started": "2021-05-30T06:44:37.578465Z"
        },
        "trusted": true,
        "id": "AZy0AUrqeSnb"
      },
      "source": [
        "preprocess_translation(parsed)"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dBEpdb0veSnb"
      },
      "source": [
        "### Preprocess Transliteration Fields:\n",
        "- For `<no>Replace __ with __</no>` the two words are exchanged in the transliteration and the tag is removed\n",
        "- `i` is standardized to `j` to match the longer corpora\n",
        "- Remove `^`, which only serves as artificial capitalization\n",
        "- Remove `(?)`\n",
        "- Standardize ellipse length\n",
        "- Remove `*` as it represents a character unwritten elsewhere\n",
        "- Standardize `:` and `=` to `.`\n",
        "- Remove parentheses\n",
        "- Add a space before any verb-endings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:37.594006Z",
          "iopub.status.busy": "2021-05-30T06:44:37.593589Z",
          "iopub.status.idle": "2021-05-30T06:44:37.606778Z",
          "shell.execute_reply": "2021-05-30T06:44:37.606139Z",
          "shell.execute_reply.started": "2021-05-30T06:44:37.593962Z"
        },
        "trusted": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tH4aXCx-eSnc",
        "outputId": "fe158d19-97df-4847-b806-8109f7077720"
      },
      "source": [
        "for item in parsed[:20]:\n",
        "    print(item.translit)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "jnk Dd nfrt wHm mrrt\n",
            "rdj wj Hm .f r sXA nj tmA Hsj wj Hm .f r aAt wrt\n",
            "Sdj .n .tw .f n .j rdj .n .j wj Hr Xt .j\n",
            "sqdd tA xft wjD .k\n",
            "jw gmj .tw .s m a Hjmwt Hr bnwt\n",
            "mk wj m bAH .k\n",
            "rdj .n .j wj Hr Xt .j\n",
            "jmj r pr\n",
            "nb jmAx\n",
            "nsw n kmt\n",
            "wrw nw AbDw\n",
            "mk wj r nHm aA .k , sxtj , Hr wnm .f Sma .j\n",
            "HD .n .j , wn hrw\n",
            "pA pw wsjr\n",
            "m xt jAw n .k jmj\n",
            "nn js n sbj Hr Hm .f\n",
            "nj sw mH 30\n",
            "ntk nbw\n",
            "mk tw m njwt , nn HqA Hwt .s\n",
            "jw nA m sbAjt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:37.617972Z",
          "iopub.status.busy": "2021-05-30T06:44:37.617668Z",
          "iopub.status.idle": "2021-05-30T06:44:37.632543Z",
          "shell.execute_reply": "2021-05-30T06:44:37.631538Z",
          "shell.execute_reply.started": "2021-05-30T06:44:37.617945Z"
        },
        "trusted": true,
        "id": "A8LW2f1KeSnc"
      },
      "source": [
        "def preprocess_translit(parsed):\n",
        "    with_no_tag = [item for item in parsed if \"<no>\" in item.translit]\n",
        "    for item in with_no_tag:\n",
        "        matches = list(re.finditer(r'\\<\\/no\\>(.+?)\\<no\\>', item.translit))\n",
        "        replaced_miswritten = item.translit.replace(matches[0].groups()[0], matches[1].groups()[0])\n",
        "        without_tags = re.sub(r'\\<no\\>(.+)\\<\\/no\\>', '', replaced_miswritten)\n",
        "        item.translit = without_tags.strip()\n",
        "        \n",
        "    with_j = [item for item in parsed if 'i' in item.translit or 'y' in item.translit]\n",
        "    for item in with_j:\n",
        "        item.translit = item.translit.replace('i', 'j').replace('y', 'j')\n",
        "        \n",
        "    with_caret = [item for item in parsed if '^' in item.translit]\n",
        "    for item in with_caret:\n",
        "        item.translit = item.translit.replace('^', '')\n",
        "        \n",
        "    with_qm = [item for item in parsed if '(?)' in item.translit]\n",
        "    for item in with_qm:\n",
        "        item.translit = item.translit.replace('(?)', \"\")\n",
        "        \n",
        "    with_ask = [item for item in parsed if \"*\" in item.translit]\n",
        "    for item in with_ask:\n",
        "        item.translit = item.translit.replace('.*', '').replace('*', \"\")\n",
        "        \n",
        "    with_colon = [item for item in parsed if ':' in item.translit]\n",
        "    for item in with_colon:\n",
        "        item.translit = item.translit.replace(': ', ' ').replace(':', '.')\n",
        "        \n",
        "    with_eq = [item for item in parsed if '=' in item.translit]\n",
        "    for item in with_eq:\n",
        "        item.translit = item.translit.replace('=', '.')\n",
        "        \n",
        "    with_hyphen = [item for item in parsed if '-' in item.translit]\n",
        "    for item in with_hyphen:\n",
        "        item.translit = item.translit.replace('-', ' ')\n",
        "\n",
        "    for item in parsed:\n",
        "        item.translit = re.sub(r\"\\.{4,}\", r\"...\", item.translit)\n",
        "        \n",
        "    for item in parsed:\n",
        "        item.translit = item.translit.replace('.', ' .').replace(' . . .', '[...]')\n",
        "        item.translit = item.translit.replace('(', '').replace(')', '').replace(',', ' ,')\n",
        "        item.translit = re.sub(r'([^\\.\\.\\.\\]])\\]', '\\\\1', item.translit)\n",
        "        item.translit = re.sub(r'\\[([^\\.\\.\\.\\]])', '\\\\1', item.translit)"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "id": "QZh9AtZTeSnc"
      },
      "source": [
        "preprocess_translit(parsed)"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "7bCLtz7bgFcS",
        "outputId": "6261e21f-d714-4ef1-9a52-18f8ccc005ae"
      },
      "source": [
        "grammar_df = pd.DataFrame([x.as_dict() for x in parsed])\n",
        "grammar_df.head()"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Transliteration</th>\n",
              "      <th>Translation</th>\n",
              "      <th>File Name</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>jnk Dd nfrt wHm mrrt</td>\n",
              "      <td>i am one who says what is good and repeats what is loved</td>\n",
              "      <td>./texts/grammar-texts/Allen.txt</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>rdj wj Hm .f r sXA nj tmA Hsj wj Hm .f r aAt wrt</td>\n",
              "      <td>his majesty appointed me scribe of the cadaster and his majesty showed me great favour</td>\n",
              "      <td>./texts/grammar-texts/Callender.txt</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Sdj .n .tw .f n .j rdj .n .j wj Hr Xt .j</td>\n",
              "      <td>when it was read to me i placed myself on my stomach [...]</td>\n",
              "      <td>./texts/grammar-texts/Callender.txt</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>sqdd tA xft wjD .k</td>\n",
              "      <td>and how egypt sails depends on what you command</td>\n",
              "      <td>./texts/grammar-texts/Callender.txt</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>jw gmj .tw .s m a Hjmwt Hr bnwt</td>\n",
              "      <td>although it is found with slave-girls at the mill-stones</td>\n",
              "      <td>./texts/grammar-texts/Callender.txt</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                    Transliteration  ...                            File Name\n",
              "0                              jnk Dd nfrt wHm mrrt  ...      ./texts/grammar-texts/Allen.txt\n",
              "1  rdj wj Hm .f r sXA nj tmA Hsj wj Hm .f r aAt wrt  ...  ./texts/grammar-texts/Callender.txt\n",
              "2          Sdj .n .tw .f n .j rdj .n .j wj Hr Xt .j  ...  ./texts/grammar-texts/Callender.txt\n",
              "3                                sqdd tA xft wjD .k  ...  ./texts/grammar-texts/Callender.txt\n",
              "4                   jw gmj .tw .s m a Hjmwt Hr bnwt  ...  ./texts/grammar-texts/Callender.txt\n",
              "\n",
              "[5 rows x 3 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CoQ18L5MeSnd"
      },
      "source": [
        "## 2. Import Nederhof files"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qE0p-GCEeSnd"
      },
      "source": [
        "- Eliminates prepending header\n",
        "- Splits files into paragraph chunks, each of which represents a sentence, in the following format:\n",
        "\n",
        "    [Sign Data]<br/> \n",
        "    ,<br/> \n",
        "    [Translit Data]<br/> \n",
        "    ;<br/> \n",
        "    [Translation data]<br/> \n",
        "\n",
        "- Parses the given data into AlignedSentence objects"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:37.669665Z",
          "iopub.status.busy": "2021-05-30T06:44:37.669406Z",
          "iopub.status.idle": "2021-05-30T06:44:37.678516Z",
          "shell.execute_reply": "2021-05-30T06:44:37.677316Z",
          "shell.execute_reply.started": "2021-05-30T06:44:37.669639Z"
        },
        "trusted": true,
        "id": "eqygjoEAeSnd"
      },
      "source": [
        "import re\n",
        "def parse_nederhof_file(file_name, file_path):\n",
        "    # Open file\n",
        "    with open(file_path, 'r') as current_file:\n",
        "        file_text = current_file.read()\n",
        "        \n",
        "    # Remove file header\n",
        "    section_break = \"###\"\n",
        "    matches = list(re.finditer(section_break, file_text))\n",
        "    assert len(matches) == 2\n",
        "    without_header = file_text[matches[1].span()[1] + 1:].strip()\n",
        "    \n",
        "    # Split paragraphs and return AlignedSententence list\n",
        "    asList = []\n",
        "    paras = without_header.split('\\n\\n')\n",
        "    for para in paras:\n",
        "        comma_idx = para.index('\\n,\\n') if '\\n,\\n' in para else -1\n",
        "        semicolon_idx = para.index('\\n;') if '\\n;' in para else -1\n",
        "        sign = para[:comma_idx] if comma_idx != -1 else None\n",
        "        translit = para[comma_idx + 3:semicolon_idx] if comma_idx != -1 else para[:semicolon_idx]\n",
        "        translate = para[semicolon_idx + 3:]\n",
        "        if 'jw jr.tw n=j' in para:\n",
        "            translate = 'they made for me'\n",
        "        asList.append(AlignedSentence(translit, translate, sign, file_name))\n",
        "    return asList"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:37.681026Z",
          "iopub.status.busy": "2021-05-30T06:44:37.68063Z",
          "iopub.status.idle": "2021-05-30T06:44:37.986515Z",
          "shell.execute_reply": "2021-05-30T06:44:37.985789Z",
          "shell.execute_reply.started": "2021-05-30T06:44:37.680986Z"
        },
        "trusted": true,
        "id": "D615N3wdeSnd"
      },
      "source": [
        "import os\n",
        "\n",
        "nederhof_path = './texts/nederhof-texts'\n",
        "all_parsed = []\n",
        "for item in os.listdir(nederhof_path):\n",
        "    parsed_file = parse_nederhof_file(item, '{}/{}'.format(nederhof_path, item))\n",
        "    all_parsed.extend(parsed_file)"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oB0ZNQsTeSne"
      },
      "source": [
        "### Preprocess Nederhof Translation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i1Krcfn6eSne"
      },
      "source": [
        "- Remove mid-sentence line breaks\n",
        "- Removes brackets containing english notes and their contents\n",
        "- Removes numbers marking chapters and line numbers\n",
        "- Removes `<al>` tags while leaving contents intact\n",
        "- Removes improperly parsed quotes `&quot;`\n",
        "- Adds a space before kept punctuation\n",
        "- Converts to lowercase"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:43.870427Z",
          "iopub.status.busy": "2021-05-30T06:44:43.870079Z",
          "iopub.status.idle": "2021-05-30T06:44:43.876347Z",
          "shell.execute_reply": "2021-05-30T06:44:43.875192Z",
          "shell.execute_reply.started": "2021-05-30T06:44:43.870397Z"
        },
        "trusted": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WZ1jWIa5eSne",
        "outputId": "e2c4f891-a7b5-400b-8218-3d1aab108036"
      },
      "source": [
        "for item in all_parsed[:10]:\n",
        "    print(item.translate)"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "overseer of works of djeserkare justified peniati\n",
            "overseer of works of aakheperkare peniati\n",
            "overseer of works of aakheperenre peniati\n",
            "the king of upper and lower egypt menkheperre given life\n",
            "the good goddess maatkare who lives again\n",
            "overseer of works in the temple of amun peniati justified\n",
            "[...] then the majesty of the king of upper and lower egypt cheops justified said\n",
            "let an offering be made of a thousand loaves of bread\n",
            "a hundred jars of beer\n",
            "one ox and two balls of incense\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:37.996332Z",
          "iopub.status.busy": "2021-05-30T06:44:37.995767Z",
          "iopub.status.idle": "2021-05-30T06:44:38.07616Z",
          "shell.execute_reply": "2021-05-30T06:44:38.075213Z",
          "shell.execute_reply.started": "2021-05-30T06:44:37.996289Z"
        },
        "trusted": true,
        "id": "y7sLNdYYeSne"
      },
      "source": [
        "with_brackets = [item for item in all_parsed if '<' in item.translate or '>' in item.translate]\n",
        "for item in with_brackets:\n",
        "    item.translate = item.translate.replace('\\n', ' ')\n",
        "    item.translate = re.sub('(<note>.+?<\\/note>)', '', item.translate)\n",
        "    item.translate = re.sub('^.+\\<\\/note\\>', '', item.translate)\n",
        "    item.translate = re.sub('\\<\\@?\\d+[a-z]?\\>?\\s?', \"\", item.translate)\n",
        "    item.translate = re.sub('\\<[A-Za-z]\\d+\\>\\s?', '', item.translate)\n",
        "    item.translate = re.sub('\\<\\d+(:|-)\\d+\\>', '', item.translate)\n",
        "    item.translate = re.sub('\\<\"[A-Za-z]+\"\\>', '', item.translate)\n",
        "    item.translate = re.sub('(:|-|,|.)\\d+\\>\\s?', '', item.translate)\n",
        "    item.translate = item.translate.replace('<al>', '').replace('</al>', '')\n",
        "    item.translate = re.sub(',?\\d*\\^pre>(.+?),?\\d*\\^post>', '\\\\1', item.translate)\n",
        "    item.translate = re.sub(\"<I{1,2}\", \"\", item.translate)\n",
        "    item.translate = item.translate.replace('b>', '')"
      ],
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T06:44:38.077907Z",
          "iopub.status.busy": "2021-05-30T06:44:38.077497Z",
          "iopub.status.idle": "2021-05-30T06:44:38.097829Z",
          "shell.execute_reply": "2021-05-30T06:44:38.096799Z",
          "shell.execute_reply.started": "2021-05-30T06:44:38.077867Z"
        },
        "trusted": true,
        "id": "Cdn-F9NUeSne"
      },
      "source": [
        "with_amph = [item for item in all_parsed if '&' in item.translate]\n",
        "for item in with_amph:\n",
        "    item.translate = item.translate.replace('&quot;', '')\n",
        "    \n",
        "for item in all_parsed:\n",
        "    item.translate = item.translate.replace('\\r\\n', ' ').replace('\\n', ' ')\n",
        "    item.translate = item.translate.replace('.', ' .').replace('[ . . .]', '[...]').replace('l .p .h .', 'l.p.h.').replace('!', '')\n",
        "    item.translate = item.translate.replace(',', '').replace('???','...').replace('?', '').replace(\"'\", \"\").replace('(', '').replace(')', '').replace('{', '').replace('}', '').replace('-', ' ').replace(':', \"\").replace(';', '')\n",
        "    item.translate = re.sub(r'([^\\.\\.\\.\\]])\\]', \"\\\\1\", item.translate)\n",
        "    item.translate = re.sub(r'\\[([^\\.\\.\\.\\]])', \"\\\\1\", item.translate)\n",
        "    item.translate = ' '.join([x for x in item.translate.split() if x != '.' and len(x)])\n",
        "    item.translate = item.translate.lower()"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nA67oDzteSne"
      },
      "source": [
        "### Preprocess Nederhof Transliteration"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "erSLGr9TeSnf"
      },
      "source": [
        "- Removes mid-sentence line breaks\n",
        "- Removes HTML tags\n",
        "- Removes artificial capitalization via caret\n",
        "- Standardizes `=` to `.`\n",
        "- Removes parentheses\n",
        "- Adds a space before kept punctuation\n",
        "- Removes hyphenation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "trusted": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rG5-fy6ZeSnf",
        "outputId": "acecb184-86aa-43b7-953f-f3a3e0936f7a"
      },
      "source": [
        "for item in all_parsed[:10]:\n",
        "    print(item.translate)"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "overseer of works of djeserkare justified peniati\n",
            "overseer of works of aakheperkare peniati\n",
            "overseer of works of aakheperenre peniati\n",
            "the king of upper and lower egypt menkheperre given life\n",
            "the good goddess maatkare who lives again\n",
            "overseer of works in the temple of amun peniati justified\n",
            "[...] then the majesty of the king of upper and lower egypt cheops justified said\n",
            "let an offering be made of a thousand loaves of bread\n",
            "a hundred jars of beer\n",
            "one ox and two balls of incense\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T00:07:49.723984Z",
          "iopub.status.busy": "2021-05-30T00:07:49.723707Z",
          "iopub.status.idle": "2021-05-30T00:07:49.820957Z",
          "shell.execute_reply": "2021-05-30T00:07:49.82017Z",
          "shell.execute_reply.started": "2021-05-30T00:07:49.723957Z"
        },
        "tags": [
          "outputPrepend"
        ],
        "trusted": true,
        "id": "ltfMmbCIeSnf"
      },
      "source": [
        "with_brackets = [item for item in all_parsed if '<' in item.translit or '>' in item.translit]\n",
        "\n",
        "for item in with_brackets:\n",
        "    item.translit = item.translit.replace('\\n', ' ')\n",
        "    item.translit = re.sub('(\\<note\\>.+?\\<\\/note\\>)', '', item.translit)\n",
        "    item.translit = re.sub('^.+\\<\\/note\\>', '', item.translit)\n",
        "    item.translit = item.translit.replace('<no>\"\"</no>', '').replace('<no> </no>', '')\n",
        "    item.translit = re.sub('\\<\\@?\\d+[a-z]?\\>?\\s?', \"\", item.translit)\n",
        "    item.translit = re.sub('\\<[A-Za-z]\\d+\\>\\s?', '', item.translit)\n",
        "    item.translit = re.sub('\\<\\d+(:|-)\\d+\\>', '', item.translit)\n",
        "    item.translit = re.sub('\\<\"([A-Za-z]+|\\d+\\s)\"\\>\\s?', '', item.translit)\n",
        "    item.translit = re.sub('(:|-|,|.)\\d+\\>\\s?', '', item.translit)\n",
        "    item.translit = item.translit.replace('<al>', '').replace('</al>', '')\n",
        "    item.translit = re.sub(',?\\d*\\^pre>(.+?),?\\d*\\^post>', '\\\\1', item.translit)\n",
        "    item.translit = re.sub(\"<I{1,2}\", \"\", item.translit)\n",
        "    item.translit = re.sub('[a-z]>\\s?', '', item.translit)"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T00:07:49.823616Z",
          "iopub.status.busy": "2021-05-30T00:07:49.823028Z",
          "iopub.status.idle": "2021-05-30T00:07:49.832284Z",
          "shell.execute_reply": "2021-05-30T00:07:49.831429Z",
          "shell.execute_reply.started": "2021-05-30T00:07:49.82357Z"
        },
        "trusted": true,
        "id": "In0xrB7beSnf"
      },
      "source": [
        "with_caret = [item for item in all_parsed if '^' in item.translit]\n",
        "for item in with_caret:\n",
        "    item.translit = item.translit.replace('^', '')"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T00:07:49.834275Z",
          "iopub.status.busy": "2021-05-30T00:07:49.833849Z",
          "iopub.status.idle": "2021-05-30T00:07:49.860085Z",
          "shell.execute_reply": "2021-05-30T00:07:49.859176Z",
          "shell.execute_reply.started": "2021-05-30T00:07:49.834235Z"
        },
        "trusted": true,
        "id": "THGJSaGLeSnf"
      },
      "source": [
        "with_eq = [item for item in all_parsed if '=' in item.translit]\n",
        "for item in with_eq:\n",
        "    item.translit = item.translit.replace('=', '.')\n",
        "\n",
        "for item in all_parsed:\n",
        "    item.translit = item.translit.replace('\\n', ' ')\n",
        "    item.translit = item.translit.replace('.', ' .').replace('a .w .s .', 'a.w.s.').replace('[ . . .]', '[...]')\n",
        "    item.translit = item.translit.replace('(', '').replace(')', '').replace('{', '').replace('}', '')\n",
        "    item.translit = item.translit.replace('-', ' ')\n",
        "    item.translit = re.sub(r'([^\\.\\.\\.\\]])\\]', \"\\\\1\", item.translit)\n",
        "    item.translit = re.sub(r'\\[([^\\.\\.\\.\\]])', \"\\\\1\", item.translit)"
      ],
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ma7OXD3seSnf"
      },
      "source": [
        "## Combine Aligned Corpora and Remove Empty Entries"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UeiqWbhEeSng"
      },
      "source": [
        "Combines the .txt and Nederhof aligned corpora and removes entries containing only spaces / numbers / ellipses."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T00:07:49.86731Z",
          "iopub.status.busy": "2021-05-30T00:07:49.866889Z",
          "iopub.status.idle": "2021-05-30T00:07:50.400864Z",
          "shell.execute_reply": "2021-05-30T00:07:50.398926Z",
          "shell.execute_reply.started": "2021-05-30T00:07:49.86728Z"
        },
        "trusted": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V3gA_W5-eSng",
        "outputId": "7c426e8c-3353-4caa-954d-9abf81f36fd5"
      },
      "source": [
        "empty = [item for item in all_parsed if item.translit.isspace() or item.translit == '' or item.translate.isspace() or item.translate == '']\n",
        "\n",
        "all_parsed = [item for item in all_parsed if item not in empty]\n",
        "print(len(all_parsed))"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "8300\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T00:07:49.861646Z",
          "iopub.status.busy": "2021-05-30T00:07:49.861304Z",
          "iopub.status.idle": "2021-05-30T00:07:49.865808Z",
          "shell.execute_reply": "2021-05-30T00:07:49.864676Z",
          "shell.execute_reply.started": "2021-05-30T00:07:49.861615Z"
        },
        "trusted": true,
        "id": "YBOVEcGPeSng"
      },
      "source": [
        "try:\n",
        "    all_parsed.extend(parsed)\n",
        "except:\n",
        "    pass"
      ],
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vAR2ngY1eSng",
        "outputId": "7c474f5e-41a0-42b9-efda-c71039726185"
      },
      "source": [
        "aligned_lst = [[item.translit, item.translate, item.from_file] for item in all_parsed if not all([ch.isspace() or ch.isnumeric() or ch in string.punctuation for ch in item.translate]) and not all([ch.isspace() or ch.isnumeric() or ch in string.punctuation for ch in item.translit]) and len(item.translate) and len(item.translit) and '[...]' not in item.translit and '[...]' not in item.translate]\n",
        "aligned_df = pd.DataFrame(data=aligned_lst, columns=['Transliteration', 'Translation', 'File Name'])\n",
        "aligned_df.shape"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(8176, 3)"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "488I4ldDeSnh",
        "outputId": "2b9aea7f-577c-40c8-b7b2-6465d2cd062c"
      },
      "source": [
        "pt_df = pd.read_csv(\"./pyramidtext-parsing/preprocessed.csv\", index_col=0)\n",
        "pt_df['File Name'] = \"Pyramid Texts\"\n",
        "aligned_df = pd.concat([aligned_df,pt_df])\n",
        "aligned_df.shape"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(12938, 3)"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jPGeWC7beSnh"
      },
      "source": [
        "def fix_cell(content):\n",
        "    stripped = content.strip()\n",
        "    without_return = stripped.strip('\\n')\n",
        "    return without_return\n",
        "aligned_df = aligned_df.applymap(fix_cell)"
      ],
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a1koXCaqhRPg",
        "outputId": "3bd9331e-f198-4b6a-c136-e6ba8a941b11"
      },
      "source": [
        "val_counts = aligned_df['File Name'].value_counts()\n",
        "print(val_counts)\n",
        "#for item in val_counts.to_string().split('\\n'):\n",
        "#  print(item)\n",
        "\n",
        "val_set = ['WestcarTr.txt', 'GebelBarkalTuthmosisIIITr.txt', 'RestorationTutankhamunTr.txt', 'HymnSunTr.txt'] # 1039\n",
        "test_set = ['DisputeTr.txt', 'HatshepsutPuntTr.txt', 'LoyalistDETr.txt', 'DestructionMankindTr.txt'] # 660\n",
        "train_set = [x for x in aligned_df['File Name'].unique() if x not in val_set and x not in test_set]"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pyramid Texts                         4762\n",
            "SinuheTrB.txt                          609\n",
            "PeasantTrB1.txt                        546\n",
            "PtahhotepDETr.txt                      494\n",
            "WestcarTr.txt                          472\n",
            "                                      ... \n",
            "./texts/grammar-texts/Ockinga.txt        1\n",
            "urkIV-015Tr.txt                          1\n",
            "./texts/grammar-texts/Allen.txt          1\n",
            "SinuheTrG.txt                            1\n",
            "./texts/grammar-texts/Loprieno.txt       1\n",
            "Name: File Name, Length: 104, dtype: int64\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "irfQcI9Wkti7",
        "outputId": "111bf6be-f902-45bc-c93e-75aa5ffefca8"
      },
      "source": [
        ""
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "96\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E9gfrUC7eSnh"
      },
      "source": [
        "aligned_df['Transliteration'].to_csv('./compiled_corpora/aligned.egy.csv', index=False)\n",
        "aligned_df['Translation'].to_csv('./compiled_corpora/aligned.eng.csv', index=False)"
      ],
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hSyTVzOReSnh"
      },
      "source": [
        "import numpy as np\n",
        "def train_validate_test_split(df):\n",
        "    test = df[df['File Name'].isin(test_set)]\n",
        "    val = df[df['File Name'].isin(val_set)]\n",
        "    train = df[df['File Name'].isin(train_set)]\n",
        "    print(\"Train len: {}, Valid len: {}, Test len: {}\".format(train.shape, val.shape, test.shape))\n",
        "    return train, val, test"
      ],
      "execution_count": 57,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LKumz9L2eSnh",
        "outputId": "bccca983-7d9b-4b0d-9079-762c90dfe782"
      },
      "source": [
        "train, val, test = train_validate_test_split(aligned_df)\n",
        "for k, v in {'train': train, 'val': val, 'test': test}.items():\n",
        "    v['Transliteration'].to_csv('./compiled_corpora/aligned_{}.egy.csv'.format(k), index=False, header=False)\n",
        "    v['Translation'].to_csv('./compiled_corpora/aligned_{}.eng.csv'.format(k), index=False, header=False)"
      ],
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train len: (11239, 3), Valid len: (1039, 3), Test len: (660, 3)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rzNzEKvmeSni"
      },
      "source": [
        "## Web-scrape single-language corpus"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.status.busy": "2021-05-30T00:07:50.402122Z",
          "iopub.status.idle": "2021-05-30T00:07:50.402918Z"
        },
        "trusted": true,
        "id": "0UVvVvsWeSni"
      },
      "source": [
        "try:\n",
        "    from bs4 import BeautifulSoup\n",
        "except:\n",
        "    !pip install beautifulsoup4\n",
        "    from bs4 import BeautifulSoup"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.status.busy": "2021-05-30T00:07:50.404142Z",
          "iopub.status.idle": "2021-05-30T00:07:50.40493Z"
        },
        "trusted": true,
        "id": "CF00NfLveSni"
      },
      "source": [
        "import requests\n",
        "\n",
        "links = ['http://www.egyptomaniak.gr/Pyramid%20Texts%202-3-4_N.html',\n",
        "         'http://www.egyptomaniak.gr/Old%20Kingdom%20Texts_N.html',\n",
        "         'http://www.egyptomaniak.gr/New%20Kingdom%20Texts%20I_N.html',\n",
        "         'http://www.egyptomaniak.gr/New%20Kingdom%20Texts%20II_N.html',\n",
        "         'http://www.egyptomaniak.gr/Book%20of%20the%20Dead_N.html',\n",
        "         'http://www.egyptomaniak.gr/Various%20Texts_N/Pyramid%20Texts%201_N.html',\n",
        "         'http://www.egyptomaniak.gr/Various%20Texts_N/Coffin%20Texts%20vol%20I_N.html',\n",
        "         'http://www.egyptomaniak.gr/Various%20Texts_N/Coffin%20Texts%20vol%20II_N.html',\n",
        "         'http://www.egyptomaniak.gr/Various%20Texts_N/Coffin%20Texts%20vol%20III_N.html',\n",
        "         'http://www.egyptomaniak.gr/Various%20Texts_N/Libyan%20anarchy_N.html',\n",
        "         'http://www.egyptomaniak.gr/Various%20Texts_N/Late%20Period%20Texts_N.html',\n",
        "         'http://www.egyptomaniak.gr/Various%20Texts_N/Middle%20Kingdom%20Texts/Middle%20Kingdom%20Texts_N.html',\n",
        "         'http://www.egyptomaniak.gr/Various%20Texts_N/Various%20Texts%202_N.html']\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.status.busy": "2021-05-30T00:07:50.406173Z",
          "iopub.status.idle": "2021-05-30T00:07:50.406952Z"
        },
        "trusted": true,
        "id": "PTdmTQ4-eSni"
      },
      "source": [
        "import unicodedata\n",
        "def get_link_lines(link):\n",
        "    file_name = 'texts/monolingual-texts/' + link.split('/')[-1]\n",
        "    if not exists(file_name):\n",
        "        page = requests.get(link)\n",
        "        with open(file_name, 'wb') as to_write:\n",
        "                to_write.write(page.content)\n",
        "        content = page.content\n",
        "    else:\n",
        "        with open(file_name, 'r') as to_read:\n",
        "            content = to_read.read()\n",
        "    soup = BeautifulSoup(content, 'html.parser')\n",
        "    paras = soup.select('p')\n",
        "    tag_txt = [unicodedata.normalize('NFKD', para.text).replace('\\r\\n', ' ').replace('\\n', ' ').strip() for para in paras if len(para.select('span')) > 0 and all(['style' in item.attrs and 'Times' in item.attrs['style'] and not item.text.isspace() for item in para.select('span')])]\n",
        "    ans = [item for item in tag_txt if item != \"\" and not item.isspace() and not item.isnumeric() and not item.startswith('Warning')]\n",
        "    print('Link: {}, Length: {}'.format(link, len(ans)))\n",
        "    print(ans[:10])\n",
        "    print('\\n')\n",
        "    return ans"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.status.busy": "2021-05-30T00:07:50.40815Z",
          "iopub.status.idle": "2021-05-30T00:07:50.408936Z"
        },
        "trusted": true,
        "id": "ws3j3Hc6eSni"
      },
      "source": [
        "nonempty = []\n",
        "sm = 0\n",
        "for link in links:\n",
        "    print(\"Working on \" + link)\n",
        "    lines = get_link_lines(link)\n",
        "    sm += len(lines)\n",
        "    nonempty.extend(lines)\n",
        "print('Total unaligned corpus length before processing: {}'.format(sm))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ky4jQELIeSnj"
      },
      "source": [
        "#### Standardize alphabetization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.status.busy": "2021-05-30T00:07:50.410136Z",
          "iopub.status.idle": "2021-05-30T00:07:50.410963Z"
        },
        "trusted": true,
        "id": "fB1ELFJXeSnj"
      },
      "source": [
        "to_swap = {'ḫ': 'x', 'ḥ': 'H', 'ȝ': 'A', 'ḳ': 'q', 'ỉ': 'i', 'ḏ': 'D', 'š': 'S', \"ʿ\": 'a', 'ẖ':'X', 'ṯ':'T'}\n",
        "times_swapped = {'ḫ': 0, 'ḥ': 0, 'ȝ': 0, 'ḳ': 0, 'ỉ': 0, 'ḏ': 0, 'š': 0, \"ʿ\": 0, 'ẖ': 0, 'ṯ': 0}\n",
        "def standardize_translit(inp):\n",
        "    current = []\n",
        "    alphabetized = []\n",
        "    skip = []\n",
        "    for item in inp:\n",
        "        temp = item\n",
        "        for k in to_swap.keys():\n",
        "            temp = re.sub(k, to_swap[k], temp)\n",
        "        alphabetized.append(temp)\n",
        "    return alphabetized"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.status.busy": "2021-05-30T00:07:50.41215Z",
          "iopub.status.idle": "2021-05-30T00:07:50.412909Z"
        },
        "trusted": true,
        "id": "0xyLXsuEeSnj"
      },
      "source": [
        "alphabetized = standardize_translit(nonempty)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bHK_v5Z0eSnj"
      },
      "source": [
        "### Preprocess transliteration"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T00:07:55.33515Z",
          "iopub.status.busy": "2021-05-30T00:07:55.334687Z",
          "iopub.status.idle": "2021-05-30T00:07:55.614064Z",
          "shell.execute_reply": "2021-05-30T00:07:55.613078Z",
          "shell.execute_reply.started": "2021-05-30T00:07:55.335108Z"
        },
        "trusted": true,
        "id": "zwhiQtYeeSnj"
      },
      "source": [
        "alphabetized = [item for item in alphabetized if not all([ch in string.punctuation or ch.isspace() for ch in item])]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T00:07:55.616067Z",
          "iopub.status.busy": "2021-05-30T00:07:55.615768Z",
          "iopub.status.idle": "2021-05-30T00:07:55.692783Z",
          "shell.execute_reply": "2021-05-30T00:07:55.691778Z",
          "shell.execute_reply.started": "2021-05-30T00:07:55.616038Z"
        },
        "trusted": true,
        "id": "yCwM5B-yeSnj"
      },
      "source": [
        "for i in range(len(alphabetized)):\n",
        "    if re.match(r'(\\d+|1A4)\\)\\s?',alphabetized[i]):\n",
        "        alphabetized[i] = re.sub(r'(\\d+|1A4)\\)\\s?', '', alphabetized[i])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T00:07:55.694894Z",
          "iopub.status.busy": "2021-05-30T00:07:55.694505Z",
          "iopub.status.idle": "2021-05-30T00:07:55.711068Z",
          "shell.execute_reply": "2021-05-30T00:07:55.709958Z",
          "shell.execute_reply.started": "2021-05-30T00:07:55.694848Z"
        },
        "trusted": true,
        "id": "vA1tz9seeSnj",
        "outputId": "6b675b34-7f8b-49b1-8b34-4445fa2ac820"
      },
      "source": [
        "alphabetized = [item for item in alphabetized if not 'Chapter' in item and not 'Sethe' in item]\n",
        "len(alphabetized)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "50617"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T00:07:55.713172Z",
          "iopub.status.busy": "2021-05-30T00:07:55.71264Z",
          "iopub.status.idle": "2021-05-30T00:07:55.733081Z",
          "shell.execute_reply": "2021-05-30T00:07:55.732229Z",
          "shell.execute_reply.started": "2021-05-30T00:07:55.71313Z"
        },
        "trusted": true,
        "id": "XF2OjsUOeSnk",
        "outputId": "6a6e448a-5bd6-4b65-ce3d-6cadfed6e635"
      },
      "source": [
        "df = pd.DataFrame(alphabetized, columns=['Sentences'])\n",
        "df['Sentences'] = df['Sentences'].astype('string')\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                    Sentences\n",
              "0    wn aA.wy (sAt) n Hr sn aA.wy SAbwt n stS\n",
              "1  pna.k n.f m xnti-inb-f swA.n N Hr.Tn m itm\n",
              "2                   N pw xaii-tAw Hri-ib ngAw\n",
              "3              Dd-mdw wab.n N Hna ra m S-iArw\n",
              "4      Hr sin.f iwf.k N DHwti sin.f rd.wy.k N"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Sentences</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>wn aA.wy (sAt) n Hr sn aA.wy SAbwt n stS</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>pna.k n.f m xnti-inb-f swA.n N Hr.Tn m itm</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>N pw xaii-tAw Hri-ib ngAw</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Dd-mdw wab.n N Hna ra m S-iArw</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Hr sin.f iwf.k N DHwti sin.f rd.wy.k N</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T00:07:55.734999Z",
          "iopub.status.busy": "2021-05-30T00:07:55.734242Z",
          "iopub.status.idle": "2021-05-30T00:07:55.741549Z",
          "shell.execute_reply": "2021-05-30T00:07:55.740651Z",
          "shell.execute_reply.started": "2021-05-30T00:07:55.734961Z"
        },
        "trusted": true,
        "id": "mg-h8WPIeSnk",
        "outputId": "05275e17-9b22-4527-8676-5c918b1c957b"
      },
      "source": [
        "for item in df['Sentences'][:10]:\n",
        "    print(item)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "wn aA.wy (sAt) n Hr sn aA.wy SAbwt n stS\n",
            "pna.k n.f m xnti-inb-f swA.n N Hr.Tn m itm\n",
            "N pw xaii-tAw Hri-ib ngAw\n",
            "Dd-mdw wab.n N Hna ra m S-iArw\n",
            "Hr sin.f iwf.k N DHwti sin.f rd.wy.k N\n",
            "Sw fA N ir Hrt nwt imi a.T n N\n",
            "Dd-mdw inD-Hr.k iri-aA n Hr ... arrwt nt wsir\n",
            "i.Dd mii rn n N ... n Hr\n",
            "i.n.f Xr psg smA(?) r smA.f pw\n",
            "mr ir.f tp-Abdw nqm ir(.f) tp-smdwt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T00:07:55.742949Z",
          "iopub.status.busy": "2021-05-30T00:07:55.742691Z",
          "iopub.status.idle": "2021-05-30T00:07:56.260586Z",
          "shell.execute_reply": "2021-05-30T00:07:56.259748Z",
          "shell.execute_reply.started": "2021-05-30T00:07:55.742924Z"
        },
        "trusted": true,
        "id": "z2sO4hNDeSnk"
      },
      "source": [
        "df['Sentences'] = df['Sentences'].str.replace('\\{', '', regex=True)\n",
        "df['Sentences'] = df['Sentences'].str.replace('\\}', '', regex=True)\n",
        "df['Sentences'] = df['Sentences'].str.replace('\\.\\.\\.', '[...]', regex=True)\n",
        "df['Sentences'] = df['Sentences'].str.replace('\\(\\.\\.\\.\\)', '[...]', regex=True)\n",
        "df['Sentences'] = df['Sentences'].str.replace('\\(\\s?\\?\\)', '', regex=True)\n",
        "df['Sentences'] = df['Sentences'].str.replace('\\.', ' .', regex=True)\n",
        "df['Sentences'] = df['Sentences'].str.replace('a .w .s .', 'a.w.s.', regex=True)\n",
        "df['Sentences'] = df['Sentences'].str.replace('\\[\\s\\.\\s\\.\\s\\.\\]', '[...]', regex=True)\n",
        "df['Sentences'] = df['Sentences'].str.replace('\\(', '', regex=True)\n",
        "df['Sentences'] = df['Sentences'].str.replace('\\)', '', regex=True)\n",
        "df['Sentences'] = df['Sentences'].str.replace('\\-', ' ', regex=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0TzRtHIPeSnk"
      },
      "source": [
        "### Remove any monolingual entry comprised solely of punctuation, spaces and/or numbers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T00:07:56.262814Z",
          "iopub.status.busy": "2021-05-30T00:07:56.262349Z",
          "iopub.status.idle": "2021-05-30T00:07:56.361495Z",
          "shell.execute_reply": "2021-05-30T00:07:56.360684Z",
          "shell.execute_reply.started": "2021-05-30T00:07:56.262763Z"
        },
        "trusted": true,
        "id": "5Yb_TGRgeSnl"
      },
      "source": [
        "broken = [item for item in df['Sentences'] if all([ch.isspace() or ch.isnumeric() or ch in string.punctuation for ch in item]) or not len(item)]\n",
        "cond = df['Sentences'].isin(broken)\n",
        "df.drop(df[cond].index, inplace = True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TD7PvbyTeSnl"
      },
      "source": [
        "### Save monolingual corpus"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "execution": {
          "iopub.execute_input": "2021-05-30T00:08:18.381058Z",
          "iopub.status.busy": "2021-05-30T00:08:18.380702Z",
          "iopub.status.idle": "2021-05-30T00:08:18.392237Z",
          "shell.execute_reply": "2021-05-30T00:08:18.391343Z",
          "shell.execute_reply.started": "2021-05-30T00:08:18.381025Z"
        },
        "trusted": true,
        "id": "GINnc28eeSnl",
        "outputId": "636a92cc-725a-40e4-da2f-f64553473fa7"
      },
      "source": [
        "print(df.shape)\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(50456, 1)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                        Sentences\n",
              "0        wn aA .wy sAt n Hr sn aA .wy SAbwt n stS\n",
              "1  pna .k n .f m xnti inb f swA .n N Hr .Tn m itm\n",
              "2                       N pw xaii tAw Hri ib ngAw\n",
              "3                 Dd mdw wab .n N Hna ra m S iArw\n",
              "4     Hr sin .f iwf .k N DHwti sin .f rd .wy .k N"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Sentences</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>wn aA .wy sAt n Hr sn aA .wy SAbwt n stS</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>pna .k n .f m xnti inb f swA .n N Hr .Tn m itm</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>N pw xaii tAw Hri ib ngAw</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Dd mdw wab .n N Hna ra m S iArw</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Hr sin .f iwf .k N DHwti sin .f rd .wy .k N</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ]
          },
          "metadata": {},
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v56e3TBfeSnl"
      },
      "source": [
        "df.to_csv('./compiled_corpora/egyptian_monolingual.csv', index=False)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}