examples/full-manual.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "With ASTrED, you can choose how much control to take over the input. In this notebook, we'll do everything manually!\n",
    "\n",
    "What this means is that we provide 1. tokenised; 2. dependency-parsed; 3. word-aligned data to create the\n",
    " `AlignedSentences` object that contains all the useful information that we want to gather. Note that this approach\n",
    "  is _super fast_: we do not need to rely on automatic tools and we can immediately calculate all metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!pip install astred pandas"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from astred import AlignedSentences, Sentence, Word"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "In this example, we will align \"Sometimes she asks me why I used to call her father Harold .\" with the Dutch sentence\n",
    " \"Soms vraagt ze waarom ik haar vader Harold noemde .\". Because we do not want to rely on automatic parsers, we need\n",
    " to create the `Word` objects manually. This is easy, though! For each word, you'll need 1. its text (Duh!); 2. the\n",
    " index of its parent word (starting from 1, use 0 as the head of the root node); 3. its\n",
    " [dependency label](https://universaldependencies.org/u/dep/index.html).\n",
    "\n",
    "We will first create a list of tuples that contains the information that we need. Then, by means of a list\n",
    " comprehension, we create a list of `Word`s and - in one go - we pass that list to the `Sentence` constructor.\n",
    "\n",
    "Note that we start counting at `1` (`enumerate(..., 1)`). This is very important (and the code will prevent you from\n",
    " starting at 0)! The reason is that every `Sentence` has a NULL word at index 0. This word is used as a reference\n",
    " alignment point for words that are not aligned. (But do not worry, those alignments do not impact any cross values.)\n",
    " The implication is that after an `AlignedSentences` object is created, a NULL word is added to the front of the\n",
    " respective `Sentence`s."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# (Text, id of head (0 for root), dependency label)\n",
    "en_words = [(\"Sometimes\", 3, \"advmod\"), (\"she\", 3, \"nsubj\"), (\"asks\", 0, \"root\"), (\"me\", 3, \"obj\"), (\"why\", 7, \"advmod\"),\n",
    "            (\"I\", 7, \"nsubj\"), (\"used\", 3, \"ccomp\"), (\"to\", 9, \"mark\"), (\"call\", 7, \"xcomp\"), (\"her\", 11, \"nmod\"),\n",
    "            (\"father\", 9, \"obj\"), (\"Harold\", 9, \"xcomp\"), (\".\", 3, \"punct\")]\n",
    "sent_en = Sentence([Word(id=idx, text=w[0], head=w[1], deprel=w[2]) for idx, w in enumerate(en_words, 1)])\n",
    "\n",
    "nl_words = [(\"Soms\", 2, \"advmod\"), (\"vraagt\", 0, \"root\"), (\"ze\", 2, \"nsubj\"),  (\"waarom\", 9, \"advmod\"),\n",
    "            (\"ik\", 9, \"nsubj\"), (\"haar\", 7, \"nmod\"), (\"vader\", 9, \"obj\"), (\"Harold\", 9, \"xcomp\"), (\"noemde\", 2, \"xcomp\"),  (\".\", 2, \"punct\")]\n",
    "sent_nl = Sentence([Word(id=idx, text=w[0], head=w[1], deprel=w[2]) for idx, w in enumerate(nl_words, 1)])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "In addition to manually creating the `Word`s we also provide the word alignment information in the typical Pharaoh\n",
    " format `i-j` where `i` is the index of a source token and `j` the index of the target token that it is aligned with.\n",
    " *Important*: even though we started counting at `1` to create the words, the word alignments are still 0-based, as\n",
    " usual, meaning that 0 refers to the first actual (non-NULL) word in the sentence. This means that you can easily\n",
    " re-use alignments that you retrieved from an alignment tool."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "aligns = \"0-0 1-2 2-1 4-3 5-4 6-8 7-8 8-8 9-5 10-6 11-7 12-9\"\n",
    "aligned = AlignedSentences(sent_en, sent_nl, word_aligns=aligns)\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The `aligned` object has now been created and contains all the information that you need. Here are some examples, but\n",
    "the other example notebooks may also give you a good idea of what is possible."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Print the target sentence as text\n",
    "print(aligned.tgt.text)\n",
    "# Note that Harold is index 8 because index 0 is now for NULL\n",
    "print(aligned.tgt[8].text)\n",
    "print(aligned.tgt[0])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "You can also easily iterate over sentences as if they were a list of words. Note that we explicitly ask for `Word`s\n",
    "that are not NULL words. If you want to iterate over all `Word`s, including NULL, then use `for word in aligned.src`."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "print(\"text\\tdeprel\\tcross\\tseq_cross\\tsacr_cross\\tastred_op\")\n",
    "for word in aligned.src.no_null_words:\n",
    "\tprint(word.text, word.deprel, word.cross, word.seq_group.cross, word.sacr_group.cross, word.tree.astred_op, sep=\"\\t\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "If you are working with Pandas, it is relatively straight-forward to make a DataFrame out of this. Again, this is\n",
    " definitely not the only information you can extract, but it serves as an example."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame.from_dict({w.text: [w.deprel, w.cross, w.sacr_group.cross, w.num_changes(), w.tree.astred_op]\n",
    "\t\t\t\t   for w in aligned.src.no_null_words},\n",
    "\t\t\t\t\t\t\torient=\"index\",\n",
    "\t\t\t\t\t\t\tcolumns=[\"deprel\", \"cross\", \"sacr_cross\", \"dep_changes\", \"astred_op\"])\n",
    "\n",
    "df"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "It is also possible to display the dependency trees. If you are working with Notebooks, you can make use of `display`,\n",
    " but then we first need to convert the tree into an NLTK tree with the method `to_string`."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# This cell does not work on remote environments such as Colab\n",
    "# Un-comment to try it on your local machine\n",
    "# from nltk.tree import Tree as NltkTree\n",
    "# from IPython.display import display\n",
    "#\n",
    "# display(NltkTree.fromstring(sent_en.tree.to_string()))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "If you are not using a notebook, and just want to show the tree during the execution of your script, you can use\n",
    " the class method `Tree` to draw trees. It makes use of NLTKs built-in tree viewer. This may *not* be ideal because it\n",
    " halts the execution of the script. That is, as long as the window with the tree is open, your code will not execute\n",
    " further. You need to close that window to continue. After you execute the following cell, you will also have to close\n",
    " the tree viewer before you can continue."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Uncomment if you want to execute this cell. It will halt the execution of the rest of the notebook.\n",
    "# Close pop-up window to continue execution.\n",
    "# This cell does not work on remote environments such as Colab.\n",
    "# Un-comment to try it on your local machine\n",
    "\n",
    "# from astred import Tree\n",
    "\n",
    "# Tree.draw_trees(sent_nl.tree)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As said before, when a `Sentence` is passed to `AlignSentences`, it will be changed - particularly because it gets a\n",
    " NULL word at the front. This means that `aligned.src` is the same as `sent_en`."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "print(\"Are they the same object?\", aligned.src is sent_en)\n",
    "print(\"First word in aligned.src:\", aligned.src[0])\n",
    "print(\"First word in sent_en:\", sent_en[0])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "However, it might be that you do not wish that the initial `Sentence` is modified in-place. If that is the case, you\n",
    " can pass `make_copy=True` to the constructor. In that case, a deep copy will be created of the input `Sentence`s\n",
    " so that they are not modified in-place. You will now notice that `sent_en` (the original sentence) is not modified."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Recreate Sentences, which were changed in-place before\n",
    "sent_en = Sentence([Word(id=idx, text=w[0], head=w[1], deprel=w[2]) for idx, w in enumerate(en_words, 1)])\n",
    "sent_nl = Sentence([Word(id=idx, text=w[0], head=w[1], deprel=w[2]) for idx, w in enumerate(nl_words, 1)])\n",
    "\n",
    "aligned = AlignedSentences(sent_en, sent_nl, word_aligns=aligns, make_copies=True)\n",
    "\n",
    "print(\"Are they the same object?\", aligned.src is sent_en)\n",
    "print(\"First word in aligned.src:\", aligned.src[0])\n",
    "print(\"First word in sent_en:\", sent_en[0])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Sentence information\n",
    "Although most information is available on the word level, aggregated values are also available for each sentence."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "print(\"Changes in dependency label\", aligned.src.num_changes(\"deprel\"))\n",
    "# Returns None because we did not include POS information\n",
    "print(\"Changes in POS\", aligned.src.num_changes(\"upos\"))\n",
    "print(\"Total number of word crosses\", aligned.src.word_cross)\n",
    "print(\"Total number of seq crosses\", aligned.src.seq_cross)\n",
    "print(\"Total number of SACr crosses\", aligned.src.sacr_cross)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}