session7/Session07_CollateXInputFormats.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Session 7 - Collate with CollateX: Input Formats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Summary \n",
    "\n",
    "- Plain text\n",
    "- JSON (simple)\n",
    "- JSON (tokenized)\n",
    "  - jupyter trick: displaying JSON\n",
    "  - jupyter trick: saving an HTML alignment table\n",
    "- XML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In this exercise, we will look at the possible input formats for CollateX.\n",
    "\n",
    "As before, delete the outputs to start with a fresh notebook (Kernel > Restart & Clear Outputs).\n",
    "\n",
    "Again, we import `collatex` and a new module called `json` that we will need for one of the input formats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collatex import *\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Input 1 - Plain text\n",
    "\n",
    "We have seen already how to use a plain text input in the previous sessions [5a](https://github.com/automaticCollationLausanne2020/Materials/blob/98b873a89b9bdb0c152f2ac7f06a899a71fda120/session5/Session05_PlainTextCollation.ipynb) and [5b](https://github.com/automaticCollationLausanne2020/Materials/blob/master/session5/Session05b_collateFiles.ipynb).\n",
    "\n",
    "**Plain text** means that it is a **string** data type: a string is the most basic way to represent text (hence the name \"plain\").\n",
    "\n",
    "*Coming soon*: we will add an example in the ancillary folder for opening a `docx` document!\n",
    "\n",
    "But it is possible to have a more complex representation of text. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Input 2 - JSON (simple) \n",
    "\n",
    "You have seen what JSON is in [session 6](https://github.com/automaticCollationLausanne2020/Materials/blob/master/session6/collation-outputs.ipynb), so you should not be too suprised by this input format!\n",
    "\n",
    "There are two possible JSON inputs. The first one is a simple version that is very similar to the plain text collation, except that all the witnesses and their sigils are gathered together in one single variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "json_input = \"\"\"{\n",
    "  \"witnesses\" : [\n",
    "    {\n",
    "      \"id\" : \"UK\",\n",
    "      \"content\" : \"George reckons he had a bogey-flavoured one once.\"\n",
    "    },\n",
    "    {\n",
    "      \"id\" : \"US\",\n",
    "      \"content\" : \"George reckons he had a booger-flavored one once.\"\n",
    "    },\n",
    "    {\n",
    "      \"id\" : \"Film\",\n",
    "      \"content\" : \"George sweared he got a bogey-flavoured one once.\"\n",
    "    }\n",
    "  ]\n",
    "}\"\"\" # three \"\"\" is for a string spanning multiple lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The `json_input` variable is a string, but it contains `{}` and `[]` which mark JSON's objects/arrays and Python's dictionaries/lists. \n",
    "\n",
    "It is very straighforward to convert this string into a dictionary that can be processed in Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "json_input:  <class 'str'>\n",
      "witnesses:  <class 'dict'>\n"
     ]
    }
   ],
   "source": [
    "# we are using the loads() method\n",
    "witnesses_simple = json.loads(json_input)\n",
    "\n",
    "# we can check that we have the correct data type\n",
    "print(\"json_input: \", type(json_input))\n",
    "print(\"witnesses: \", type(witnesses_simple))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Plain text vs JSON\n",
    "\n",
    "CollateX has a slightly different behavior between the two types of input.\n",
    "\n",
    "With plain text input, we need to bring together the witnesses which are each stored into a different file. We do that thanks to the 'Collation()' instance. This is a way to gather the witnesses before we pass them on to the 'collate()' function. \n",
    "\n",
    "But in json, we already have our witnesses together into a single file, so that we don't need to bring them together through a 'Collation()' instance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Since we already have all our witnesses gathered into a single variable, we do not need to create a Collation instance with the line:\n",
    "\n",
    "`collation = Collation()`\n",
    "\n",
    "We can directly collate the witnesses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+--------+---------+----+-----+---+--------+---+-----------+-----------+\n",
      "| UK   | George | reckons | he | had | a | bogey  | - | flavoured | one once. |\n",
      "| US   | George | reckons | he | had | a | booger | - | flavored  | one once. |\n",
      "| Film | George | sweared | he | got | a | bogey  | - | flavoured | one once. |\n",
      "+------+--------+---------+----+-----+---+--------+---+-----------+-----------+\n"
     ]
    }
   ],
   "source": [
    "# of course we could have more options (output, layout, segmentation, indent)\n",
    "result = collate(witnesses_simple)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Input 3 - JSON (tokenized)\n",
    "\n",
    "For this input format, the witnesses must be divided into tokens first, before we can collate (see the [Gothenburg model](https://automaticcollationlausanne2020.github.io/session4a.html) about tokenization).\n",
    "\n",
    "In JSON, the tokens are like a python dictionary, they have properties expressed as key/value pairs and are surrounded by curly braces (see [session 2](https://github.com/automaticCollationLausanne2020/Materials/blob/master/session2/Session02_python_introduction.ipynb))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Tokens have the following properties:\n",
    "- **t** : a mandatory property with the actual text of the transcribed document (technically, it is called a 'reading')\n",
    "- **n** : an optional normalized form of the reading, which can help CollateX with the alignment (we will look at the normalization in a few minutes during [session 7b](https://github.com/automaticCollationLausanne2020/Materials/blob/master/session7/Session07b_Normalization.ipynb)).\n",
    "- Anything that you feel like including! For instance part-of-speech tagging, or the location of the reading in the document, an editorial note or comment...\n",
    "\n",
    "The last point is an aspect of CollateX that makes it very advantageous: the ability to pass more information along a token. It can be particularly helpful for the [normalization](https://github.com/automaticCollationLausanne2020/Materials/blob/master/session7/Session07b_Normalization.ipynb) phase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "json_sample = {\"witnesses\": [ # we have a list of witnesses\n",
    "    {\"id\": \"A\", # each witness has a name\n",
    "    \"tokens\": [ # and a list of tokens\n",
    "        {\"t\":\"Bladorthin,\"}, # mandatory 't'\n",
    "        {\"t\":\"Dwarves,\", \"n\":\"dwarves,\"}, # optional 'n'\n",
    "        {\"t\":\"and\", \"pos\":\"conjunction\"}, # and whatever you want\n",
    "        {\"t\":\"Mr\"},\n",
    "        {\"t\":\"Baggins.\"} # last element: no comma after\n",
    "        ]\n",
    "    }, \n",
    "    #... here you would have the other witnesses\n",
    "]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If this looks scary, you can take some time to have a good look at it again. It is a very complex object, made of several nested lists and dictionaries:\n",
    "\n",
    "1. dictionary: the whole object is in curly braces `{}`\n",
    "2. the dictionary has one key called 'witnesses'\n",
    "3. the value of 'witnesses' is a list of dictionaries!\n",
    "4. each witness is a dictionary with two keys ('id' and 'tokens')\n",
    "5. the 'tokens' key is a list of dictionaries again!\n",
    "6. each token has at least one key/value ('t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Reading a JSON file\n",
    "Ususally, we don't want to write an entire JSON file ourselves in Python. For now we are going to read it from a file with the `load()` method.\n",
    "\n",
    "**Attention!** There are two json methods:\n",
    "\n",
    "- `json.loads()` is for \"load string\", it converts a string into a dictionary, as we have seen with the simple JSON input\n",
    "- `json.load()` is for reading JSON from a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# this is how to read a json file\n",
    "with open(\"../data/Catullus/catullus-tokenized-OGBodmer47.json\", \"r\") as file:\n",
    "    witnesses = json.load(file) # note that we are not using read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# let us look at the file we just read: what properties can token have here?\n",
    "witnesses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Collating JSON\n",
    "\n",
    "Again, since we have already our witnesses in a variable, we do not need to create a `Collation()` instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# the collate function does not return anything when the output is 'html',\n",
    "# therefore we don't need a variable to store the result\n",
    "collate(witnesses, output=\"html\", layout=\"vertical\", segmentation=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**A couple of remarks**\n",
    "\n",
    "Do you see weird characters in the collation result? If this ever happen to you, the most likely explanation is a file encoding issue. Go back to the opening function for the json file and add `, encoding='utf-8'` after `r`, and collate again. That's better..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You also see that the text is completely concatenated in some cells of the table. This is a result of the JSON input: there are no whitespaces to separate the tokens, as there was in plain text. Go back to the `collate()`function and add `, segmentation=False` to the arguments.\n",
    "\n",
    "Now your collation result should be readable. That is nice! but what happened to our tokens properties, the **n**, **locus**, and **note** that we could see in the JSON file?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "CollateX does not keep them for most outputs: if you try to modify the output in the next code cell, you will see that `json` is the only one where you can find them again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "with open(\"../data/Catullus/catullus-tokenized-OGBodmer47.json\", \"r\", encoding=\"utf-8\") as file:\n",
    "    witnesses = json.load(file)\n",
    "\n",
    "result = collate(witnesses, output=\"json\", layout=\"vertical\", segmentation=False)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Jupyter trick: Displaying JSON \n",
    "\n",
    "\n",
    "The JSON result of CollateX is not very easy to read. It is actually a string, a sequence of characters. The computer does not interpret the brackets as indications of the structure of the result. \n",
    "\n",
    "So, if we want a better display of the JSON result (like what we saw when we opened the input file), we need to follow a few steps:\n",
    "1. convert the string to a dictionary object\n",
    "2. \"prettify\" the dictionary\n",
    "3. print the pretty dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "type(result) # => it is a string. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# To display it nicely, it would be better to have a json object\n",
    "\n",
    "dictionary = json.loads(result) # we convert the string to json\n",
    "type(dictionary) # => now we have a dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# we can \"prettify\" json with json.dumps()\n",
    "pretty_dict = json.dumps(dictionary, indent=1) \n",
    "print(pretty_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# and of course we can save it as a file\n",
    "with open('collatex_json_result.json', 'w', encoding='utf-8') as file:\n",
    "    file.write(pretty_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Jupyter trick: saving your html table \n",
    "\n",
    "As we have seen, the HTML table and the graph output are displayed directly in Jupyter, but you cannot save them in a variable or in a file. That is sometimes inconvenient, since you may not always want to open a notebook to see your results.\n",
    "\n",
    "There is a Jupyter trick, a magic command, that will capture the output of the cell (what you see after `out [xx]:`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%%capture result \n",
    "\n",
    "# %%capture is a jupyter magic command! \n",
    "# It must be always the first line in a cell\n",
    "\n",
    "collate(witnesses, output=\"html\", layout=\"vertical\", segmentation=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# since we captured the previous cell's output, it did not appear\n",
    "# if we want to see the result we have to display it in a new cell with the method show()\n",
    "result.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# we save the html part of the output in a variable\n",
    "# this is tricky! You just have to believe that it will work for now\n",
    "# Attention: it could stop working later with new versions of Jupyter\n",
    "html = result.outputs[0].data[\"text/html\"]\n",
    "html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# and finally we can save our table in a file for later\n",
    "with open('collatex_html_result.html', 'w', encoding=\"utf-8\") as file:\n",
    "    file.write(html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, we only save the HTML, not the CSS, which is what makes alternate lines darker, and highlights the rows of the table when you hover over it with your mouse.\n",
    "\n",
    "This can be fixed by adding [CSS code](https://www.w3schools.com/css/css_table.asp) to your HTML file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Input 4 - XML TEI?\n",
    "\n",
    "The [TEI Guidelines](https://www.tei-c.org/release/doc/tei-p5-doc/en/html/index.html) is the standard format for transcription encoding. It is very likely that your transcriptions will use XML TEI, as plain text is rarely suitable to record the text of a document in its complexity.\n",
    "\n",
    "Here is a short video to watch later, if you want to know what TEI is and why it is a good idea to use it: https://www.youtube.com/watch?v=VvSQ530gxPM\n",
    "\n",
    "Unfortunately, XML is not an input option for the Python version of CollateX."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Why no XML input\n",
    "\n",
    "If we try to collate XML directly we will have problems: CollateX cannot make the difference between the tags and the text! We need to remove the tags, but there are two problems with that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "1. There can be many different ways of encoding the same in the TEI: it is a flexible format that can be adapted to the needs of researchers depending on their sources, and their particular needs. Therefore it is difficult for CollateX to anticipate all possible TEI input formats! \n",
    "\n",
    "2. Second, there can be several layers of texts encoded in a single file. For Catullus, we have two separate files for the first hand text and the second hand corrections. But you could very well have both in a single file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Consider the following example from codex [Bodmer 47](https://www.e-codices.unifr.ch/fr/fmb/cb-0047/1v/):\n",
    "\n",
    "<img src=\"../ancillary/catullus2_secondHandCorr.jpg\" width=\"75%\"/>\n",
    "\n",
    "You could have this encoding:\n",
    "\n",
    "`<subst><del>Credo</del><add>Corde</add></subst> ut...`\n",
    "\n",
    "If you simply remove the tags, your text will be \"CredoCorde ut...\" which does not make much sense! You should have either *Credo* or *Corde* but not both..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What can I do?\n",
    "\n",
    "If you need to collate from XML encoded files, there will be different solutions depending on the complexity of your encoding and what you are trying to achieve. \n",
    "\n",
    "Therefore it is important from the beginning of your project, if possible even before you start transcribing, to think about the **purpose** of your collation project, and **what information you will need later on for your analysis.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If you have a relatively simple encoding, and you don't need more than simple tokens **t**, you can extract a plain text version by removing the tags (using XSLT or even regular expressions) fairly easily. There is an example script in the ancillary folder, [tei2txt.xsl](https://github.com/automaticCollationLausanne2020/Materials/blob/master/ancillary/tei2txt.xsl)\n",
    "\n",
    "If you need to include some normalization forms for a better collation, or other kind of notes (e.g. part of speech tagging, etc), then it will be worth converting your XML to JSON."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the conversion from XML to JSON, here are a few examples online:\n",
    "\n",
    "- [example 1](https://github.com/enury/phd-automated-collation/blob/master/XSLT/witnesses-to-json.xsl) (Elisa Nury, it is the transformation I have used for the Catullus example and the Calpurnius App)\n",
    "- [example 2](https://gitlab.huma-num.fr/mgillelevenson/tei_collator/-/blob/master/xsl/pre_alignement/transformation_json.xsl) (Matthias Gille [Levenson](http://perso.ens-lyon.fr/matthias.gille-levenson/accueil.html), PhD student, Lyon)\n",
    "- [example 3](https://github.com/CondorCompPhil/falcon/blob/656a1ed8988bf8b3f60f31c3445af89cb97fba61/falcon/collation.py#L8-L68) (Jean-Baptiste [Camps](http://www.chartes.psl.eu/fr/jean-baptiste-camps), for the [*Falcon*](https://github.com/CondorCompPhil/falcon) project)\n",
    "\n",
    "However, you should note that these XSLT transformations are really tailored for a particular TEI encoding, and a specific JSON input. They are not meant to be directly reused for another project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other options?\n",
    "\n",
    "1. HyperCollate? https://github.com/HuygensING/hyper-collate\n",
    "2. CollateX java version has limited support for XML - https://collatex.net/doc/#xml-input\n",
    "3. Another collation tool..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Recap and Exercise \n",
    " There are three input formats for CollateX: plain text, simple JSON and tokenized JSON.\n",
    " \n",
    "Text files with plain text witnesses are read from files with `read()`. Plain text witnesses must be added to a `Collation()` instance before collating.\n",
    " \n",
    "JSON inputs are loaded either from a string with `json.loads()` or from a file with `json.load()`. You do not need to add JSON inputs to a `Collation()` instance.\n",
    " \n",
    "The tokenized JSON input can include additional information about tokens. These additional informations can be accessed again from the JSON output.\n",
    " \n",
    "Both plain text and simple JSON let you get the same outputs. In the case of tokenized JSON, you may need to use the option `segmentation=False`.\n",
    " \n",
    " **Exercise:** start the notebook again and collate both simple JSON and tokenized JSON. When you feel confident, create your own code cell and try to do the following again:\n",
    " - load the Catullus tokenized JSON \n",
    " - collate\n",
    " - try various combinations of options (output, layout, segmentation, indent)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Diaporama",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "rise": {
   "scroll": true,
   "theme": "solarized",
   "transition": "slide"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}