session6/collation-outputs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Collation outputs\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Introduction\n",
    "- In practice\n",
    "    - Table: plain text & HTML\n",
    "- An interlude: the alignment parameters\n",
    "- An interlude: the JSON format\n",
    "- Continuing the practice\n",
    "    - Table: JSON\n",
    "    - Table: CSV and TSV\n",
    "    - Table: XML and XML/TEI\n",
    "    - Graph: SVG\n",
    "- Summary: supplementary output parameters\n",
    "- Summary: output types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this tutorial we will be trying different outputs for our collation, this is, we will try different graphical representations, formats and visualizations of the result.\n",
    "\n",
    "The visualization of the collation result is an open discussion: several possibilities have been suggested and used and new ones are always being proposed. When the output of the collation is a printed format, such as a book, it is rare to see anything different from the traditional critical apparatus. Now that output formats are more frequently digital (or at least have a digital component), collation tools tend to offer more than one visualization option, like it is the case for CollateX. The different visualizations are not incompatible; on the contrary, they can be complementary, highlighting different aspects of the result and suitable for different users or different stages of the workflow.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "\n",
    "When we introduced CollateX (session 4), we saw a few examples of the alignment table and the graph. The alignment table, in use since the 1960's, is the equivalent of the matrix of bioinformatic for sequence alignment (for example, strings of DNA). In contrast, the graph is meant to represent the fluidity of the text and its variation. The idea of a graph-oriented model for expressing textual variance has been originally developed by Desmond Schmidt [(2008)](http://multiversiondocs.blogspot.it/2008/03/whats-multi-version-document.html). You can refer to [this video](https://vimeo.com/114242362), for a presentation on *Apparatus vs. Graph – an Interface as Scholarly Argument* by Tara Andrews and Joris van Zundert.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "CollateX has two main ways to present the collation result: as a **table** (with many different formatting options) and as a **graph**:\n",
    "- table formats\n",
    "    - plain text table (no need to specify the output)\n",
    "    - HTML table (`output='**html**'`)\n",
    "    - HTML vertical table with colors (`output='**html2**'`)\n",
    "    - JSON (`output='**json**'`)\n",
    "    - XML (`output='**xml**'`)\n",
    "    - XML/TEI (`output='**tei**'`)\n",
    "- graph format\n",
    "    - SVG (`output='**svg**'`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## In practice: output parameters\n",
    "\n",
    "Even though we have already encountered some of these outputs, it is worth going through them one more time focussing on the part of the code that needs to change to produce the different formats. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "### Table: plain text\n",
    "\n",
    "In this tutorial we will use a sentence taken from the British and US editions of _Harry Potter and the Philosopher’s Stone_ by J.K. Rowling and the same quote from the film adaptation.\n",
    "\n",
    "Let’s start with the most simple output, for which we don't need to specify any output format (note that you can name the variable containing the output anything you like, but in this tutorial we call it `table` or `graph`)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "\n",
    "In the code cell below the lines starting with a hash (`#`) are comments and are not executed. They are there in this instance to help you remember what the different parts of the code do. You do not need to use them in your notebook (although sometimes it is helpful to add comments to your code so you remember what things do)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#import the collatex library\n",
    "from collatex import *\n",
    "#create an instance of the CollateX engine\n",
    "collation = Collation()\n",
    "#add witnesses to the CollateX instance\n",
    "collation.add_plain_witness( \"UK\", \"George reckons he had a bogey-flavoured one once.\")\n",
    "collation.add_plain_witness( \"US\", \"George reckons he had a booger-flavored one once.\" )\n",
    "collation.add_plain_witness( \"Film\", \"George sweared he got a bogey-flavoured one once.\" )\n",
    "#collate the witnesses and store the result in a variable called 'table'\n",
    "#as we have not specified an output this will be stored in plain text\n",
    "table = collate(collation)\n",
    "#print the collation result\n",
    "print(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Specially when there are many witnesses, you might want to create a vertical table instead. For that, you need to add the parameter `layout=\"vertical\"` to the `collate()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "table = collate(collation, layout=\"vertical\")\n",
    "print(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Table: HTML\n",
    "\n",
    "Now let's try a different output. This time we still want a table format but instead of it being in plain text we would like it exported in HTML (the markup language used for web pages). To achieve this all you need to do is add the keyword `output` to the `collate` command and give it the value `html`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "table = collate(collation, output=\"html\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "CollateX Python supports two HTML output methods, `html` and `html2`. Unlike the plain text table, which must be printed with a `print()` statement, both HTML formats automatically write their output to the screen inside the Jupyter Notebook interface. These output formats are intended for use only inside Jupyter Notebook, and CollateX Python currently does not expose a method to save them for use elsewhere (but refer to the following session to see how to save the results of the collation).\n",
    "\n",
    "By default the HTML method, like the ASCII table method, creates a horizontal table. You can create a vertical table adding again the `layout` parameter.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "table = collate(collation, output=\"html\", layout=\"vertical\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The `html2` method produces only vertical output (the layout parameter is ignored) and the output is colorized, which makes it easier to distinguish zones with variation (red background) and those without (cyan)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "table = collate(collation, output=\"html2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## An interlude: aligment parameters\n",
    "\n",
    "The first argument to the `collate()` function is the name of the Collation object. The `output`, `layout`, and `indent` parameters control the shape of the output, and are discussed below. There are also two parameters that control the way the alignment is performed or rendered independently of output format: `segmentation` and `near_match`. Both take Boolean values (True or False)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The segmentation parameter\n",
    "\n",
    "The segmentation parameter determines whether each token is output separately (`False`) or whether adjacent tokens that agree in, whether they include variation or not, are merged into the same output node or cell (`True`). The default is `True`, so `collate(collation)`, using the same sample produces:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<TABLE border=\"1\" style=\"border: 1px solid #000000; border-collapse: collapse;\" cellpadding=\"4\">\n",
       " <TR>\n",
       "  <TH>UK</TH>\n",
       "  <TH>US</TH>\n",
       "  <TH>Film</TH>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"00FFFF\">George</TD>\n",
       "  <TD bgcolor=\"00FFFF\">George</TD>\n",
       "  <TD bgcolor=\"00FFFF\">George</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"FF0000\">reckons</TD>\n",
       "  <TD bgcolor=\"FF0000\">reckons</TD>\n",
       "  <TD bgcolor=\"FF0000\">sweared</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"00FFFF\">he</TD>\n",
       "  <TD bgcolor=\"00FFFF\">he</TD>\n",
       "  <TD bgcolor=\"00FFFF\">he</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"FF0000\">had</TD>\n",
       "  <TD bgcolor=\"FF0000\">had</TD>\n",
       "  <TD bgcolor=\"FF0000\">got</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"00FFFF\">a</TD>\n",
       "  <TD bgcolor=\"00FFFF\">a</TD>\n",
       "  <TD bgcolor=\"00FFFF\">a</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"FF0000\">bogey</TD>\n",
       "  <TD bgcolor=\"FF0000\">booger</TD>\n",
       "  <TD bgcolor=\"FF0000\">bogey</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"00FFFF\">-</TD>\n",
       "  <TD bgcolor=\"00FFFF\">-</TD>\n",
       "  <TD bgcolor=\"00FFFF\">-</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"FF0000\">flavoured</TD>\n",
       "  <TD bgcolor=\"FF0000\">flavored</TD>\n",
       "  <TD bgcolor=\"FF0000\">flavoured</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"00FFFF\">one once.</TD>\n",
       "  <TD bgcolor=\"00FFFF\">one once.</TD>\n",
       "  <TD bgcolor=\"00FFFF\">one once.</TD>\n",
       " </TR>\n",
       "</TABLE>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#import the collatex library\n",
    "from collatex import *\n",
    "#create an instance of the CollateX engine\n",
    "collation = Collation()\n",
    "#add witnesses to the CollateX instance\n",
    "collation.add_plain_witness( \"UK\", \"George reckons he had a bogey-flavoured one once.\")\n",
    "collation.add_plain_witness( \"US\", \"George reckons he had a booger-flavored one once.\" )\n",
    "collation.add_plain_witness( \"Film\", \"George sweared he got a bogey-flavoured one once.\" )\n",
    "#collate the witnesses and store the result in a variable called 'table'\n",
    "#as we have not specified an output this will be stored in plain text\n",
    "table = collate(collation)\n",
    "#print the collation result\n",
    "table = collate(collation, output=\"html2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "while `segmentation=False` produces:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<TABLE border=\"1\" style=\"border: 1px solid #000000; border-collapse: collapse;\" cellpadding=\"4\">\n",
       " <TR>\n",
       "  <TH>UK</TH>\n",
       "  <TH>US</TH>\n",
       "  <TH>Film</TH>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"00FFFF\">George</TD>\n",
       "  <TD bgcolor=\"00FFFF\">George</TD>\n",
       "  <TD bgcolor=\"00FFFF\">George</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"FF0000\">reckons</TD>\n",
       "  <TD bgcolor=\"FF0000\">reckons</TD>\n",
       "  <TD bgcolor=\"FF0000\">sweared</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"00FFFF\">he</TD>\n",
       "  <TD bgcolor=\"00FFFF\">he</TD>\n",
       "  <TD bgcolor=\"00FFFF\">he</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"FF0000\">had</TD>\n",
       "  <TD bgcolor=\"FF0000\">had</TD>\n",
       "  <TD bgcolor=\"FF0000\">got</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"00FFFF\">a</TD>\n",
       "  <TD bgcolor=\"00FFFF\">a</TD>\n",
       "  <TD bgcolor=\"00FFFF\">a</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"FF0000\">bogey</TD>\n",
       "  <TD bgcolor=\"FF0000\">booger</TD>\n",
       "  <TD bgcolor=\"FF0000\">bogey</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"00FFFF\">-</TD>\n",
       "  <TD bgcolor=\"00FFFF\">-</TD>\n",
       "  <TD bgcolor=\"00FFFF\">-</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"FF0000\">flavoured</TD>\n",
       "  <TD bgcolor=\"FF0000\">flavored</TD>\n",
       "  <TD bgcolor=\"FF0000\">flavoured</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"00FFFF\">one</TD>\n",
       "  <TD bgcolor=\"00FFFF\">one</TD>\n",
       "  <TD bgcolor=\"00FFFF\">one</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"00FFFF\">once</TD>\n",
       "  <TD bgcolor=\"00FFFF\">once</TD>\n",
       "  <TD bgcolor=\"00FFFF\">once</TD>\n",
       " </TR>\n",
       " <TR>\n",
       "  <TD bgcolor=\"00FFFF\">.</TD>\n",
       "  <TD bgcolor=\"00FFFF\">.</TD>\n",
       "  <TD bgcolor=\"00FFFF\">.</TD>\n",
       " </TR>\n",
       "</TABLE>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "table = collate(collation, segmentation=False, output=\"html2\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Spoiler alert: \n",
    "\n",
    "There is `near_match` parameter that will be discussed in session 7b."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## An interlude: the JSON format\n",
    "\n",
    "One of the CollateX ouput formats is [JavaScript Object Notation (JSON)](https://www.json.org/json-en.html). JSON is a syntax for storing and exchanging data. It is a lightweight data-interchange format and due to its “self-describing” nature, it is easy to understand.\n",
    "\n",
    "**JSON data** is written as **name/value pairs**. A name/value pair consists of a field name (in double quotes), followed by a colon, followed by a value:\n",
    "\n",
    "``\"firstName\": \"Virginia\"``\n",
    "\n",
    "**JSON objects** are written inside curly braces. A object can contain multiple name/values pairs:\n",
    "\n",
    "``{\"firstName\": \"Virginia\", \"lastName\": \"Woolf\"}``\n",
    "\n",
    "**JSON arrays** (ordered lists) are written inside square brackets. Arrays can contain objects.\n",
    "\n",
    "``\"authors\":[\n",
    "    {\"firstName\":\"Virginia\", \"lastName\":\"Woolf\"},\n",
    "    {\"firstName\":\"Toni\", \"lastName\":\"Morrison\"},\n",
    "    {\"firstName\":\"Maya\", \"lastName\":\"Angelou\"}\n",
    "]``\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "JSON is a very flexible format that can be used either as the input or the output of CollateX. A JSON file as a CollateX input consists of one root object that contains the data of the transcription. This object requires a property called `witnesses` which encloses the different versions we want to collate. The value of this property is an array (a list) of objects in turn, with each object representing a version. The order of the array elements determines the order in which they are processed by the alignment algorithm.\n",
    "\n",
    "Each object in the witnesses array must have a unique identifier in the required property `id`. This identifier will be used in the output to reference a particular version. Besides the identifier each object must describe the content of the version. The content can either be specified as a string property named `content`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "``{\n",
    "  \"witnesses\" : [\n",
    "    {\n",
    "      \"id\" : \"UK\",\n",
    "      \"content\" : \"George reckons he had a bogey-flavoured one once.\"\n",
    "    },\n",
    "    {\n",
    "      \"id\" : \"US\",\n",
    "      \"content\" : \"George reckons he had a booger-flavored one once.\"\n",
    "    },\n",
    "    {\n",
    "      \"id\" : \"Film\",\n",
    "      \"content\" : \"George sweared he got a bogey-flavoured one once.\"\n",
    "    }\n",
    "  ]\n",
    "}``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "Another option is to provide the content of versions in tokenized (and optionally normalized) form:\n",
    "\n",
    "``{\n",
    "    \"witnesses\": [\n",
    "        {\n",
    "            \"id\": \"UK\",\n",
    "            \"tokens\": [\n",
    "                {\"t\": \"George\"},\n",
    "                {\"t\": \"reckons\"},\n",
    "                {\"t\": \"he\"},\n",
    "                {\"t\": \"had\"},\n",
    "                {\"t\": \"a\"},\n",
    "                {\"t\": \"bogey\", \"n\": \"booger\"},\n",
    "                {\"t\": \"-\"},\n",
    "                {\"t\": \"flavoured\"},\n",
    "                {\"t\": \"one\"},\n",
    "                {\"t\": \"once\"},\n",
    "                {\"t\": \".\"}\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"id\": \"US\",\n",
    "            \"tokens\": [\n",
    "                {\"t\": \"George\"},\n",
    "                {\"t\": \"reckons\"},\n",
    "                {\"t\": \"he\"},\n",
    "                {\"t\": \"had\"},\n",
    "                {\"t\": \"a\"},\n",
    "                {\"t\": \"booger\"},\n",
    "                {\"t\": \"-\"},\n",
    "                {\"t\": \"flavored\", \"n\": \"flavoured\"},\n",
    "                {\"t\": \"one\"},\n",
    "                {\"t\": \"once\"},\n",
    "                {\"t\": \".\"}\n",
    "            ]\n",
    "        },\n",
    "                {\n",
    "            \"id\": \"Film\",\n",
    "            \"tokens\": [\n",
    "                {\"t\": \"George\"},\n",
    "                {\"t\": \"swears\"},\n",
    "                {\"t\": \"he\"},\n",
    "                {\"t\": \"had\"},\n",
    "                {\"t\": \"a\"},\n",
    "                {\"t\": \"bogey\", \"n\": \"booger\"},\n",
    "                {\"t\": \"-\"},\n",
    "                {\"t\": \"flavoured\"},\n",
    "                {\"t\": \"one\"},\n",
    "                {\"t\": \"once\"},\n",
    "                {\"t\": \".\"}\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "}``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "When there is the `n` property, it is importance to note that the alignment is done taking into consideration the contents of this property and not those of `t`. In the ecdotic context, we can consider that `t` contains the diplomatic transcription and `n` the normalized form."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Continuing the practise\n",
    "\n",
    "### Table: JSON\n",
    "\n",
    "In order to produce JSON as output, we need to specify *json* as the output format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "table = collate(collation, output='json')\n",
    "print(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s present the JSON output indented to facilitate its legibility:\n",
    "\n",
    "\n",
    "``{\n",
    "    \"table\": [\n",
    "        [\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"UK\",\n",
    "                    \"_token_array_position\": 0,\n",
    "                    \"n\": \"George\",\n",
    "                    \"t\": \"George \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"UK\",\n",
    "                    \"_token_array_position\": 1,\n",
    "                    \"n\": \"reckons\",\n",
    "                    \"t\": \"reckons \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"UK\",\n",
    "                    \"_token_array_position\": 2,\n",
    "                    \"n\": \"he\",\n",
    "                    \"t\": \"he \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"UK\",\n",
    "                    \"_token_array_position\": 3,\n",
    "                    \"n\": \"had\",\n",
    "                    \"t\": \"had \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"UK\",\n",
    "                    \"_token_array_position\": 4,\n",
    "                    \"n\": \"a\",\n",
    "                    \"t\": \"a \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"UK\",\n",
    "                    \"_token_array_position\": 5,\n",
    "                    \"n\": \"bogey\",\n",
    "                    \"t\": \"bogey\"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"UK\",\n",
    "                    \"_token_array_position\": 6,\n",
    "                    \"n\": \"-\",\n",
    "                    \"t\": \"-\"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"UK\",\n",
    "                    \"_token_array_position\": 7,\n",
    "                    \"n\": \"flavoured\",\n",
    "                    \"t\": \"flavoured \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"UK\",\n",
    "                    \"_token_array_position\": 8,\n",
    "                    \"n\": \"one\",\n",
    "                    \"t\": \"one \"\n",
    "                },\n",
    "                {\n",
    "                    \"_sigil\": \"UK\",\n",
    "                    \"_token_array_position\": 9,\n",
    "                    \"n\": \"once\",\n",
    "                    \"t\": \"once\"\n",
    "                },\n",
    "                {\n",
    "                    \"_sigil\": \"UK\",\n",
    "                    \"_token_array_position\": 10,\n",
    "                    \"n\": \".\",\n",
    "                    \"t\": \".\"\n",
    "                }\n",
    "            ]\n",
    "        ],\n",
    "        [\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"US\",\n",
    "                    \"_token_array_position\": 12,\n",
    "                    \"n\": \"George\",\n",
    "                    \"t\": \"George \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"US\",\n",
    "                    \"_token_array_position\": 13,\n",
    "                    \"n\": \"reckons\",\n",
    "                    \"t\": \"reckons \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"US\",\n",
    "                    \"_token_array_position\": 14,\n",
    "                    \"n\": \"he\",\n",
    "                    \"t\": \"he \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"US\",\n",
    "                    \"_token_array_position\": 15,\n",
    "                    \"n\": \"had\",\n",
    "                    \"t\": \"had \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"US\",\n",
    "                    \"_token_array_position\": 16,\n",
    "                    \"n\": \"a\",\n",
    "                    \"t\": \"a \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"US\",\n",
    "                    \"_token_array_position\": 17,\n",
    "                    \"n\": \"booger\",\n",
    "                    \"t\": \"booger\"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"US\",\n",
    "                    \"_token_array_position\": 18,\n",
    "                    \"n\": \"-\",\n",
    "                    \"t\": \"-\"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"US\",\n",
    "                    \"_token_array_position\": 19,\n",
    "                    \"n\": \"flavored\",\n",
    "                    \"t\": \"flavored \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"US\",\n",
    "                    \"_token_array_position\": 20,\n",
    "                    \"n\": \"one\",\n",
    "                    \"t\": \"one \"\n",
    "                },\n",
    "                {\n",
    "                    \"_sigil\": \"US\",\n",
    "                    \"_token_array_position\": 21,\n",
    "                    \"n\": \"once\",\n",
    "                    \"t\": \"once\"\n",
    "                },\n",
    "                {\n",
    "                    \"_sigil\": \"US\",\n",
    "                    \"_token_array_position\": 22,\n",
    "                    \"n\": \".\",\n",
    "                    \"t\": \".\"\n",
    "                }\n",
    "            ]\n",
    "        ],\n",
    "        [\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"Film\",\n",
    "                    \"_token_array_position\": 24,\n",
    "                    \"n\": \"George\",\n",
    "                    \"t\": \"George \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"Film\",\n",
    "                    \"_token_array_position\": 25,\n",
    "                    \"n\": \"sweared\",\n",
    "                    \"t\": \"sweared \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"Film\",\n",
    "                    \"_token_array_position\": 26,\n",
    "                    \"n\": \"he\",\n",
    "                    \"t\": \"he \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"Film\",\n",
    "                    \"_token_array_position\": 27,\n",
    "                    \"n\": \"got\",\n",
    "                    \"t\": \"got \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"Film\",\n",
    "                    \"_token_array_position\": 28,\n",
    "                    \"n\": \"a\",\n",
    "                    \"t\": \"a \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"Film\",\n",
    "                    \"_token_array_position\": 29,\n",
    "                    \"n\": \"bogey\",\n",
    "                    \"t\": \"bogey\"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"Film\",\n",
    "                    \"_token_array_position\": 30,\n",
    "                    \"n\": \"-\",\n",
    "                    \"t\": \"-\"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"Film\",\n",
    "                    \"_token_array_position\": 31,\n",
    "                    \"n\": \"flavoured\",\n",
    "                    \"t\": \"flavoured \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"Film\",\n",
    "                    \"_token_array_position\": 32,\n",
    "                    \"n\": \"one\",\n",
    "                    \"t\": \"one \"\n",
    "                },\n",
    "                {\n",
    "                    \"_sigil\": \"Film\",\n",
    "                    \"_token_array_position\": 33,\n",
    "                    \"n\": \"once\",\n",
    "                    \"t\": \"once\"\n",
    "                },\n",
    "                {\n",
    "                    \"_sigil\": \"Film\",\n",
    "                    \"_token_array_position\": 34,\n",
    "                    \"n\": \".\",\n",
    "                    \"t\": \".\"\n",
    "                }\n",
    "            ]\n",
    "        ]\n",
    "    ],\n",
    "    \"witnesses\": [\n",
    "        \"UK\",\n",
    "        \"US\",\n",
    "        \"Film\"\n",
    "    ]\n",
    "}``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As you can see we have two properties whose values are one or more arrays: `table` that contains the results of the collation, and `witnesses` that contains the list of our witnesses. `table` contains three arrays: one for each witness. Each of these arrays has an object for each token with the following properties:\n",
    "\n",
    "- `_sigil`: the ID of the witness\n",
    "- `_token_array_position`: the position of the token in the array: this number may work also as an identifier of the object\n",
    "- `n`: the normalized version of the token. In this example, the values of `t` and `n` are always the same because we didn’t introduced in the input any information about normalization.\n",
    "- `t`: the token.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In this example, there are no gap tokens, thus every token is aligned. If we were to collate the following sentence:\n",
    "\n",
    "__Wit. A__: *The giant octopus likes crabs*\n",
    "\n",
    "__Wit. B__: *The octopus likes velvet crabs*\n",
    "\n",
    "We will have two `null` objects as “gap tokens”.\n",
    "\n",
    "\n",
    "\n",
    "``{\n",
    "    \"table\": [\n",
    "        [\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"A\",\n",
    "                    \"_token_array_position\": 0,\n",
    "                    \"n\": \"The\",\n",
    "                    \"t\": \"The \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"A\",\n",
    "                    \"_token_array_position\": 1,\n",
    "                    \"n\": \"giant\",\n",
    "                    \"t\": \"giant \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"A\",\n",
    "                    \"_token_array_position\": 2,\n",
    "                    \"n\": \"octopus\",\n",
    "                    \"t\": \"octopus \"\n",
    "                },\n",
    "                {\n",
    "                    \"_sigil\": \"A\",\n",
    "                    \"_token_array_position\": 3,\n",
    "                    \"n\": \"likes\",\n",
    "                    \"t\": \"likes \"\n",
    "                }\n",
    "            ],\n",
    "            null,\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"A\",\n",
    "                    \"_token_array_position\": 4,\n",
    "                    \"n\": \"crabs\",\n",
    "                    \"t\": \"crabs\"\n",
    "                }\n",
    "            ]\n",
    "        ],\n",
    "        [\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"B\",\n",
    "                    \"_token_array_position\": 6,\n",
    "                    \"n\": \"The\",\n",
    "                    \"t\": \"The \"\n",
    "                }\n",
    "            ],\n",
    "            null,\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"B\",\n",
    "                    \"_token_array_position\": 7,\n",
    "                    \"n\": \"octopus\",\n",
    "                    \"t\": \"octopus \"\n",
    "                },\n",
    "                {\n",
    "                    \"_sigil\": \"B\",\n",
    "                    \"_token_array_position\": 8,\n",
    "                    \"n\": \"likes\",\n",
    "                    \"t\": \"likes \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"B\",\n",
    "                    \"_token_array_position\": 9,\n",
    "                    \"n\": \"velvet\",\n",
    "                    \"t\": \"velvet \"\n",
    "                }\n",
    "            ],\n",
    "            [\n",
    "                {\n",
    "                    \"_sigil\": \"B\",\n",
    "                    \"_token_array_position\": 10,\n",
    "                    \"n\": \"crabs\",\n",
    "                    \"t\": \"crabs \"\n",
    "                }\n",
    "            ]\n",
    "        ]\n",
    "    ],\n",
    "    \"witnesses\": [\n",
    "        \"A\",\n",
    "        \"B\"\n",
    "    ]\n",
    "}``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Table: CSV and TSV\n",
    "\n",
    "The output methods `csv` and `tsv` produce comma-separated-value (CSV) and tab-separated-value (TSV) output, respectively, following the layout of the basic plain text table, where each row corresponds to a witness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "table = collate(collation, output='csv')\n",
    "print(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Table: XML and XML/TEI\n",
    "\n",
    "We can use the same procedure in order to export the table in XML or XML/TEI. The main difference between these two methods is that the latter produces a condensed version of the table only listing witnesses at points of divergence (also called a negative apparatus). To do this, we just keep specifying a different output format. Let's start with the XML output (that you can later post-process using XSLT or other tools)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "table = collate(collation, output='xml')\n",
    "print(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "And, finally, you can test the XML/TEI output that produces XML following the TEI parallel segmentation encoding guidelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "table = collate(collation, output='tei')\n",
    "print(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A more legible output is available by setting the `indent` parameter to `true`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "table = collate(collation, output=\"tei\", indent=True)\n",
    "print(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "### Graph: SVG\n",
    "\n",
    "Two types of SVG output are supported for visualizing the variant graph, `svg_simple` and `svg`. SVG output, like HTML output and unlike the plain text table output, is rendered automatically by the `collate()` function inside the Jupyter Notebook interface. The two SVG output formats are intended for use only inside Jupyter Notebook, and CollateX Python currently does not expose a method to save them for use elsewhere (but take a look at session 7b for a workaround)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "graph = collate(collation, output='svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The `svg` output method outputs a two-column table for each node in the variant graph. The upper left cell contains the `n` (normalized) value of the token and the upper right cell contains the number of witnesses that share that `n` value. Subsequent rows contains the `t` (textual, that is, diplomatic) value in the left column and the sigla of witnesses that attest that `t` value in the right column.\n",
    "\n",
    "Since our input in the previous cell doesn’t have a `n` value because we were working with plain text, we cannot see the difference. In the following example, we use a JSON input that contains additional information that affect the british slang normalization (*bogey* is normalized as *booger*) and the american spelling (*flavored* is normalized as *flavoured*)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#import the json library\n",
    "import json\n",
    "from collatex import *\n",
    "#assign to the variable `json_input` a JSON object with the pretokenized and normalized witnesses\n",
    "json_input = \"\"\"{ \"witnesses\": [\n",
    "        {\n",
    "            \"id\": \"UK\",\n",
    "            \"tokens\": [\n",
    "                {\"t\": \"George\"},\n",
    "                {\"t\": \"reckons\"},\n",
    "                {\"t\": \"he\"},\n",
    "                {\"t\": \"had\"},\n",
    "                {\"t\": \"a\"},\n",
    "                {\"t\": \"bogey\", \"n\": \"booger\"},\n",
    "                {\"t\": \"-\"},\n",
    "                {\"t\": \"flavoured\"},\n",
    "                {\"t\": \"one\"},\n",
    "                {\"t\": \"once\"},\n",
    "                {\"t\": \".\"}\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"id\": \"US\",\n",
    "            \"tokens\": [\n",
    "                {\"t\": \"George\"},\n",
    "                {\"t\": \"reckons\"},\n",
    "                {\"t\": \"he\"},\n",
    "                {\"t\": \"had\"},\n",
    "                {\"t\": \"a\"},\n",
    "                {\"t\": \"booger\"},\n",
    "                {\"t\": \"-\"},\n",
    "                {\"t\": \"flavored\", \"n\": \"flavoured\"},\n",
    "                {\"t\": \"one\"},\n",
    "                {\"t\": \"once\"},\n",
    "                {\"t\": \".\"}\n",
    "            ]\n",
    "        },\n",
    "                {\n",
    "            \"id\": \"Film\",\n",
    "            \"tokens\": [\n",
    "                {\"t\": \"George\"},\n",
    "                {\"t\": \"swears\"},\n",
    "                {\"t\": \"he\"},\n",
    "                {\"t\": \"had\"},\n",
    "                {\"t\": \"a\"},\n",
    "                {\"t\": \"bogey\", \"n\": \"booger\"},\n",
    "                {\"t\": \"-\"},\n",
    "                {\"t\": \"flavoured\"},\n",
    "                {\"t\": \"one\"},\n",
    "                {\"t\": \"once\"},\n",
    "                {\"t\": \".\"}\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "}\"\"\"\n",
    "#collate the witnesses passing the variable “json_input” as the first argument of the collate() function\n",
    "#store the result in a variable called 'graph'\n",
    "graph = collate(json.loads(json_input), output='svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The SVG output creates `start` and `end` nodes that mark the beginnings and ends of all witnesses. All three witnesses attest the same readings for “George”, “he” and “a”. The readings diverge with respect to the main verb of the predicate: The film script attests “swears” and Witnesses “UK” and “US” (the books) share the value of “reckons”. The edges are labeled according to the witnesses; the complete reading of any witness can be reconstructed by following the labeled edges for that witness."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "Separate information about `n` and `t` values is most important in cases involving complex custom normalization. For simpler output, the `svg_simple` type renders only the `n` values, and produces:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#create an instance of the CollateX engine\n",
    "collation = Collation()\n",
    "#add witnesses to the CollateX instance\n",
    "collation.add_plain_witness( \"UK\", \"George reckons he had a bogey-flavoured one once.\")\n",
    "collation.add_plain_witness( \"US\", \"George reckons he had a booger-flavored one once.\" )\n",
    "collation.add_plain_witness( \"Film\", \"George sweared he got a bogey-flavoured one once.\" )\n",
    "#collate the witnesses and store the result in a variable called 'graph'\n",
    "graph = collate(collation, output='svg_simple')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "### Summary: supplementary output parameters\n",
    "\n",
    "#### The `layout` parameter\n",
    "\n",
    "The `layout` parameter controls whether table output is “horizontal” (which is the default) or “vertical”. It is relevant only for output types `table` and `html`. Otherwise it is ignored: `html2` output is always vertical, and the other output types are not tabular.\n",
    "\n",
    "#### The `indent` parameter\n",
    "\n",
    "The `indent` parameter controls whether TEI-XML output is pretty-printed. The default is to serialize the entire XML output in a single line; setting `indent` to any value other than `None` will cause the output to be pretty-printed instead. The `indent` parameter is ignored for all methods except `tei`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Summary\n",
    "\n",
    "In the following table, possible values of the `output` parameter are listed in the left column, and their ability to combine with the `layout`, and `indent` parameters is indicated (“yes” ~ “no”) in the other columns.\n",
    "\n",
    "`output` | `layout` | `indent`|\n",
    "----|----|----|\n",
    "**table** | yes | no |\n",
    "**html** |  yes | no |\n",
    "**html2** | no | no |\n",
    "**svg_simple** | no | no |\n",
    "**svg** |  no | no |\n",
    "**xml** | no | no |\n",
    "**tei** | no | yes |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Exercise\n",
    "\n",
    "In this tutorial we have used the _Harry Potter_ example. Now try to produce a JSON or TEI output with the first paragraph of Darwin's *On the Origin of Species*  that we've just used in the previous session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "rise": {
   "scroll": true,
   "theme": "solarized"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}