tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6e63604b",
   "metadata": {},
   "source": [
    "# STAM Tutorial: Standoff Text Annotation for Pythonistas\n",
    "\n",
    "## Introduction\n",
    "\n",
    "[STAM](https://github.com/annotation/stam) is a data model, and accompanied\n",
    "tooling, for stand-off text annotation that allows researchers and developers\n",
    "to model annotations on text.\n",
    "\n",
    "An *annotation* is any kind of remark, classification/tagging on any particular\n",
    "portion(s) of a text, on the resource or annotation set as a whole, in which\n",
    "case we can interpret annotations as *metadata*, or on another annotation\n",
    "(*higher-order annotation*).\n",
    "\n",
    "Examples of annotation may be linguistic annotation, structure/layout\n",
    "annotation, editorial annotation, technical annotation, or whatever comes to\n",
    "mind. STAM does not define any vocabularies whatsoever. Instead, it provides a\n",
    "framework upon which you can model your annotations using whatever\n",
    "you see fit.\n",
    "\n",
    "The model is thoroughly explained [in its specification\n",
    "document](https://github.com/annotation/stam/blob/master/README.md). We\n",
    "summarize only the most important data structures here, these have direct\n",
    "counterparts (classes) in the python library we will be teaching in this\n",
    "tutorial: \n",
    "\n",
    "* `Annotation`  - A instance of annotation. Associated with an annotation is a\n",
    "  `Selector` to select the target of the annotation, and one or more\n",
    "  `AnnotationData` instances that hold the *body* or *content* of the\n",
    "  annotation. This is explicitly decoupled from the annotation instance itself\n",
    "  as multiple annotations may hold the very same content.\n",
    "* `Selector` - A selector identifies the target of an annotation and the part of the target that the annotation applies to. There are multiple types that are described [here](https://github.com/annotation/stam/blob/master/README.md#class-selector). The `TextSelector` is an important one that selects a target resource and a specific text selection within it by specifying an offset. \n",
    "* `AnnotationData` - A key/value pair that acts as *body* or *content* for one or more annotations. The key is a reference to `DataKey`, the value is a `DataValue`. (The term *feature* is also seen for this in certain annotation paradigms)\n",
    "* `DataKey` - A key as referenced by `AnnotationData`.\n",
    "* `DataValue` - A value with some type information (e.g. string, integer, float).\n",
    "* `TextResource` - A textual resource that is made available for annotation. This holds the actual textual content.\n",
    "* `TextSelection` - A particular selection of text within a resource, i.e. a subslice of the text.\n",
    "* `AnnotationDataSet` - An Annotation Data Set stores the keys (`DataKey`) and\n",
    "  values (`AnnotationData`) that are used by annotations. It effectively\n",
    "  defines a certain vocabulary, i.e. key/value pairs. How broad or narrow the\n",
    "  scope of the vocabulary is not defined by STAM but entirely up to the user. \n",
    "* `AnnotationStore` - The annotation store is essentially your *workspace*, it holds all\n",
    "  resources, annotation sets (i.e. keys and annotation data) and of course the\n",
    "  actual annotations. In the Python implementation it is a memory-based store\n",
    "  and you can put as much as you like into it (as long as it fits in memory).\n",
    "\n",
    "STAM is more than just a theoretical model, we offer practical implementations\n",
    "that allow you to work with it directly. In this tutorial we will be using Python and\n",
    "the Python library `stam`.\n",
    "\n",
    "**Note**: The STAM Python library is a so-called Python binding to a STAM library\n",
    "written in Rust. This means the library is not written in Python but is\n",
    "compiled to machine code and as such offers much better performance.\n",
    "\n",
    "## Installation\n",
    "\n",
    "First of all, you will need to install the STAM Python library from the [Python Package Index](https://pypi.org/project/stam/) as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0e7963c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: stam in ./env/lib/python3.11/site-packages (0.4.0)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.2\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install stam"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b423bd0c",
   "metadata": {},
   "source": [
    "## Annotating from scratch\n",
    "\n",
    "### Adding a text\n",
    "\n",
    "Let us start with a mini corpus consisting of two quotes from the book *\"Consider Phlebas\"* by renowned sci-fi author Iain M. Banks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "75113e71",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"\n",
    "# Consider Phlebas\n",
    "$ author=Iain M. Banks\n",
    "\n",
    "## 1\n",
    "Everything about us,\n",
    "everything around us,\n",
    "everything we know [and can know of] is composed ultimately of patterns of nothing;\n",
    "that’s the bottom line, the final truth.\n",
    "\n",
    "So where we find we have any control over those patterns,\n",
    "why not make the most elegant ones, the most enjoyable and good ones,\n",
    "in our own terms?\n",
    "\n",
    "## 2\n",
    "Besides,\n",
    "it left the humans in the Culture free to take care of the things that really mattered in life,\n",
    "such as [sports, games, romance,] studying dead languages,\n",
    "barbarian societies and impossible problems,\n",
    "and climbing high mountains without the aid of a safety harness.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11d6a3dc",
   "metadata": {},
   "source": [
    "This format of the text for STAM is in no way prescribed other than:\n",
    "\n",
    "* It must be plain text\n",
    "* It must be UTF-8 encoded\n",
    "* It should ideally be in Unicode Normalization Form C. (don't worry if this means nothing to you yet)\n",
    "\n",
    "Before we can do anything we need to import the STAM library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9b45abb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import stam"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a88536f",
   "metadata": {},
   "source": [
    "Let's add this text resource to an annotation store so we can annotate it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f117e093",
   "metadata": {},
   "outputs": [],
   "source": [
    "store = stam.AnnotationStore(id=\"tutorial\")\n",
    "resource_banks = store.add_resource(id=\"banks\", text=text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bad080be",
   "metadata": {},
   "source": [
    "Here we passed the text as a string, but it could just as well have been an\n",
    "external text file instead, the filename of which can be passed via the `file=` keyword\n",
    "argument.\n",
    "\n",
    "### Creating an annotation dataset (vocabulary)\n",
    "\n",
    "Our example text is a bit Markdown-like, we have a title header *\"Consider Phlebas\"*, and \n",
    "two subheaders (*1* and *2*) containing one quote from the book each. \n",
    "\n",
    "As our first annotations, let's try to annotate this coarse structure. At this\n",
    "point we're already in need of some vocabulary to express the notions of *title\n",
    "header*, *section header* and *quote*, as STAM does not define any vocabulary.\n",
    "It is up to you to make these choices on how to represent the data.\n",
    "\n",
    "An annotation data set effectively defines an vocabulary. Let's invent our own\n",
    "simple Annotation Data Set that defines the keys and values we use in this\n",
    "tutorial. In our `AnnotationDataSet` We can define a `DataKey` with ID `structuretype`, and have it\n",
    "takes values like `titleheader`, `sectionheader` and `quote`.\n",
    "\n",
    "We can explicitly add the set and the key. We give the dataset a public ID\n",
    "(*tutorial-set*), just as we previously assigned a public ID to both the\n",
    "annotationstore (*tutorial*) and the text resource (*banks*). It is good\n",
    "practise to assign IDs, though you can also let the library auto-generate them\n",
    "for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ffa41021",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = store.add_dataset(\"tutorial-set\")\n",
    "key_structuretype = dataset.add_key(\"structuretype\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12ade58c",
   "metadata": {},
   "source": [
    "### The first annotations with text selectors\n",
    "\n",
    "To annotate the title header, we need to select the part of the text where it\n",
    "occurs by finding the offset, which consists of a *begin* and *end* position. STAM\n",
    "follows the same indexing format Python does, in which positions are 0-indexed\n",
    "*unicode character points* (as opposed to (UTF-8) bytes) and where the end is\n",
    "non-inclusive. After some clumsy manual counting on the source text we discover\n",
    "the following coordinates hold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "674600ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert text[1:19] == \"# Consider Phlebas\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90bf74f2",
   "metadata": {},
   "source": [
    "And we make the annotation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a961dc2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "annotation = store.annotate(\n",
    "    target=stam.Selector.textselector(resource_banks, stam.Offset.simple(1,19)),\n",
    "    data={\"id\": \"Data1\", \"key\": key_structuretype, \"value\": \"titleheader\", \"set\": dataset },\n",
    "    id=\"Annotation1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b09c8551",
   "metadata": {},
   "source": [
    "A fair amount happened there. We selected a part of the text of\n",
    "`resource_banks` by offset, and associated `AnnotationData` with the annotation\n",
    "saying that the `structuretype` key has the value `titleheader`, both of which\n",
    "we invented as part of our `AnnotationDataSet` with ID `tutorial-set`. Last, we\n",
    "assigned an ID to both the `AnnotationData`, as well as to the `Annotation` as\n",
    "a whole. In this example we reused some of the variables we had created\n",
    "earlier, but we could have also written out in full as shown below: \n",
    "\n",
    "```\n",
    "annotation = store.annotate(\n",
    "    target=stam.Selector.textselector(resource_banks, stam.Offset.simple(1,19)),\n",
    "    data={\"id\": \"Data1\", \"key\": \"structuretype\", \"value\": \"titleheader\", \"set\": \"tutorial-set\" },\n",
    "    id=\"Annotation1\")\n",
    "```\n",
    "\n",
    "This would also have been perfectly fine, and moreover, it would also work fine\n",
    "without us explicitly creating the `AnnotationDataSet` and the key as we did\n",
    "before! Those would have been automatically created on-the-fly for us. The\n",
    "only disadvantage is that under the hood more lookups are needed, so this is\n",
    "slightly less performant than passing python variables.\n",
    "\n",
    "### Inspecting data (1)\n",
    "\n",
    "We can inspect the annotation we just added:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "51a74e95",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Annotation ID:  Annotation1\n",
      "Target text:  # Consider Phlebas\n",
      "Data: \n",
      " - Data ID:  Data1\n",
      "   Data Key:  structuretype\n",
      "   Data Value:  titleheader\n"
     ]
    }
   ],
   "source": [
    "print(\"Annotation ID: \", annotation.id())\n",
    "print(\"Target text: \", str(annotation))\n",
    "print(\"Data: \")\n",
    "for data in annotation.data():\n",
    "    print(\" - Data ID: \", data.id())\n",
    "    print(\"   Data Key: \", data.key().id())\n",
    "    print(\"   Data Value: \", str(data.value()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55086b1e",
   "metadata": {},
   "source": [
    "In the above example, we obtained an `Annotation` instance from the return value of the `annotate()` method. Once any annotation is in the store, we can retrieve it simply by its public ID using the `annotation()` method. An exception will be raised if the ID does not exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d8b635c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "annotation = store.annotation(\"Annotation1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "473b3623",
   "metadata": {},
   "source": [
    "A similar pattern holds for almost all other data structures in the STAM model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ddfff00c",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = store.dataset(\"tutorial-set\")            #AnnotationDataSet\n",
    "resource_banks = store.resource(\"banks\")           #TextResource\n",
    "key_structuretype = dataset.key(\"structuretype\")   #DataKey\n",
    "data = dataset.annotationdata(\"Data1\")             #AnnotationData"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa6eb373",
   "metadata": {},
   "source": [
    "There are also shortcut methods available to get keys and data directly from a\n",
    "store, without needing to first retrieve a dataset yourself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b7b73e0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "key_structuretype = store.key(\"tutorial-set\",\"structuretype\")   #DataKey\n",
    "data = store.key(\"tutorial-set\",\"Data1\")                        #AnnotationData"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c33631c4",
   "metadata": {},
   "source": [
    "### Annotating via `find_text()`\n",
    "\n",
    "We now continue by adding annotations for the two section headers. Counting offsets\n",
    "manually is rather cumbersome, so we use the `find_text()` method on `TextResource` to find our target for annotation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c3f08f6a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text ## 1 found at 44:48\n"
     ]
    }
   ],
   "source": [
    "results = resource_banks.find_text(\"## 1\")\n",
    "section1 = results[0]\n",
    "print(f\"Text {str(section1)} found at {section1.begin()}:{section1.end()}\")\n",
    "\n",
    "annotation = store.annotate(\n",
    "    target=stam.Selector.textselector(resource_banks, section1.offset()),\n",
    "    data={\"id\": \"Data2\", \"key\": \"structuretype\", \"value\": \"sectionheader\", \"set\": \"tutorial-set\" },\n",
    "    id=\"Annotation2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1a584d8",
   "metadata": {},
   "source": [
    "The `find_text()` method returns a list of `TextSelection` instances. These\n",
    "carry an `Offset` which is returned by the `offset()` method. Hooray, no more\n",
    "manual counting!\n",
    "\n",
    "We do the same for the last header:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "83ee736f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text ## 2 found at 365:369\n"
     ]
    }
   ],
   "source": [
    "results = resource_banks.find_text(\"## 2\")\n",
    "section2 = results[0]\n",
    "print(f\"Text {str(section2)} found at {section2.begin()}:{section2.end()}\")\n",
    "\n",
    "annotation = store.annotate(\n",
    "    target=stam.Selector.textselector(resource_banks, section2.offset()),\n",
    "    data={\"id\": \"Data2\", \"key\": \"structuretype\", \"value\": \"sectionheader\", \"set\": \"tutorial-set\" },\n",
    "    id=\"Annotation3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b14a1d53",
   "metadata": {},
   "source": [
    "### Inspecting data (2)\n",
    "\n",
    "In the previous code the attentive reader may have noted that we are reusing the `Data2` ID\n",
    "rather than introducing a new `Data3` ID, because the data for both\n",
    "`Annotation2` and `Annotation3` is in fact, identical.\n",
    "\n",
    "This is an important feature of STAM; annotations and their data are\n",
    "decoupled precisely because the data may be referenced by multiple annotations, and\n",
    "if that's the case, we only want to keep the data in memory once. We don't want\n",
    "a copy for every annotation. Say we have `AnnotationData` with key\n",
    "`structuretype` and value `word`, and use that to tag all words in the\n",
    "text, then it would be a huge amount of redundancy if there was no such\n",
    "decoupling between data and annotations. The fact that they all share the same data, also\n",
    "enables us to quickly look up all those annotations via a *reverse index* that is kept internally:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5f064446",
   "metadata": {},
   "outputs": [],
   "source": [
    "for annotationdata in store.data(set=\"tutorial-set\", key=\"structuretype\", value=\"sectionheader\"):\n",
    "    for annotation in annotationdata.annotations():\n",
    "        assert annotation.id() in (\"Annotation2\",\"Annotation3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf391073",
   "metadata": {},
   "source": [
    "This can also be done in one go, which is typically more performant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9406adbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "for annotation in store.data(set=\"tutorial-set\", key=\"structuretype\", value=\"sectionheader\").annotations():\n",
    "    assert annotation.id() in (\"Annotation2\",\"Annotation3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eef8a6ed",
   "metadata": {},
   "source": [
    "Here we used `data()` on the store as a whole, this method provides an easy way to retrieve data from scratch.\n",
    "We could have also started from an annotation dataset or even a key within it if we already have an instance to it, in that case we use the `data()` method and pass the key (`DataKey`), which will act as a filter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c283173d",
   "metadata": {},
   "outputs": [],
   "source": [
    "key = dataset.key(\"structuretype\")\n",
    "for annotation in dataset.data(key, value=\"sectionheader\").annotations():\n",
    "    assert annotation.id() in (\"Annotation2\",\"Annotation3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "688988d1",
   "metadata": {},
   "source": [
    "However, since we have the key already it is simpler and more performant to use\n",
    "it directly and reduce the example to the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "b327c1a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "key = dataset.key(\"structuretype\")\n",
    "for annotation in key.data(value=\"sectionheader\").annotations():\n",
    "    assert annotation.id() in (\"Annotation2\",\"Annotation3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54b0d5c0",
   "metadata": {},
   "source": [
    "The ability to use any STAM object as a departing point for retrieval of other\n",
    "objects is a characteristic of the API. The ability to pass arbitrary objects\n",
    "as a filter is also a characteristic that you will find on multiple methods.\n",
    "\n",
    "The `data()` method can also be used to search for all values indiscriminately:\n",
    "simply omit the `value` keyword parameter. Moreover, it can be used to search\n",
    "for non-exact values, using the following keyword arguments:\n",
    "\n",
    "* `value_not` - Negates a values \n",
    "* `value_greater` - Value must be greater than specified (int or float)\n",
    "* `value_less` - Value must be less than specified (int or float)\n",
    "* `value_greatereq` - Value must be greater than specified or equal (int or float)\n",
    "* `value_lesseq` - Value must be less than specified or equal (int or float)\n",
    "* `value_in` - Value must match any in the tuple (this is a logical OR statement)\n",
    "* `value_not_in` - Value must not match any in the tuple\n",
    "* `value_in_range` - Must be a numeric 2-tuple with min and max (inclusive) values\n",
    "* `value_not_in_range` - Must be a numeric 2-tuple with min and max (inclusive) values\n",
    "\n",
    "The `data()` method takes filter parameter as positional arguments. You can\n",
    "pass as many as you like. The object you pass as filter determines what is\n",
    "being filtered, you can pass a `DataKey` instance, an `AnnotationData` instance,\n",
    "or even an `Annotation`. You can also pass the result of earlier data or annotation\n",
    "requests (`Data`, `Annotations`). If you want to filter against *one/any* of multiple\n",
    "values, use a tuple or list of any homogeneous type.\n",
    "\n",
    "Searching for data and then retrieving the corresponding annotations is a very\n",
    "common operation and easily accomplished by simply adding `.annotations()`, as\n",
    "we've seen in the above examples.\n",
    "\n",
    "We can apply data filtering operations directly to `annotations()` using the\n",
    "same keyword arguments we saw for `data()`. The following example provides\n",
    "identical results as the earlier one, but the way of getting there is\n",
    "slightly different (this takes all annotations first, and tests the data filter\n",
    "on each, the other example takes the data first, and goes over all annotations\n",
    "that make use of the data):\n",
    "\n",
    "```\n",
    "key = dataset.key(\"structuretype\")\n",
    "for annotation in store.annotations(key, value=\"sectionheader\"):\n",
    "    assert annotation.id() in (\"Annotation2\",\"Annotation3\")\n",
    "```\n",
    "\n",
    "If you're interested in the underlying text selections, then you can just add\n",
    "`.textselections()`. This chaining of methods on collections is one of the\n",
    "characteristics of the STAM API.\n",
    "\n",
    "### Annotations via text selections\n",
    "\n",
    "Now we will annotate the quotes themselves. The first one starts after the first\n",
    "subheader (Annotation2) and ends just before the next subheader (Annotation3).\n",
    "That would include some ugly leading and trailing whitespace/newlines, though.\n",
    "We use the `textselection()` method to obtain a textselection to our computed\n",
    "offset and subsequently strip the whitespace using the `strip_text()` method,\n",
    "effectively shrinking our textselection a bit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "693e0483",
   "metadata": {},
   "outputs": [],
   "source": [
    "quote1_selection = resource_banks.textselection(stam.Offset.simple(section1.end(), section2.begin() - 1)).strip_text(\" \\t\\r\\n\")\n",
    "quote1 = store.annotate(\n",
    "    target=stam.Selector.textselector(resource_banks, quote1_selection.offset()),\n",
    "    data={\"id\": \"Data3\", \"key\": \"structuretype\", \"value\": \"quote\", \"set\": \"tutorial-set\" },\n",
    "    id=\"AnnotationQuote1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d757a5da",
   "metadata": {},
   "source": [
    "The second quote goes until the end of the text, which we can retrieve using\n",
    "the `textlen()` method. This method is preferred over doing things in native\n",
    "python like `len(str(banks))` because it is way more efficient:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "057a76a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "quote2_selection = resource_banks.textselection(stam.Offset.simple(section2.end(), resource_banks.textlen())).strip_text(\" \\t\\r\\n\")\n",
    "quote2 = store.annotate(\n",
    "    target=stam.Selector.textselector(resource_banks, quote2_selection.offset()),\n",
    "    data={\"id\": \"Data3\", \"set\": \"tutorial-set\"},\n",
    "    id=\"AnnotationQuote2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3b3a9a4",
   "metadata": {},
   "source": [
    "In this example we also show that, since we reference existing\n",
    "`AnnotationData`, just specifying the ID and the set suffices. Or even shorter and better, you could pass\n",
    "a variable that is an instance of `AnnotationData`.\n",
    "\n",
    "There is another structural type we could annotate: the lines with\n",
    "corresponding line numbers. This is easy to do by splitting the text on\n",
    "newlines, for which we use the method `split_text()` on `TextResource`. As you\n",
    "see, various Python methods such as `split()`, `strip()`, `find()` have\n",
    "counterparts in STAM that have a `*_text()` suffix and which return\n",
    "`TextSelection` instances and carry offset information:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "3c7a6d20",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Line 1: \n",
      "Line 2: # Consider Phlebas\n",
      "Line 3: $ author=Iain M. Banks\n",
      "Line 4: \n",
      "Line 5: ## 1\n",
      "Line 6: Everything about us,\n",
      "Line 7: everything around us,\n",
      "Line 8: everything we know [and can know of] is composed ultimately of patterns of nothing;\n",
      "Line 9: that’s the bottom line, the final truth.\n",
      "Line 10: \n",
      "Line 11: So where we find we have any control over those patterns,\n",
      "Line 12: why not make the most elegant ones, the most enjoyable and good ones,\n",
      "Line 13: in our own terms?\n",
      "Line 14: \n",
      "Line 15: ## 2\n",
      "Line 16: Besides,\n",
      "Line 17: it left the humans in the Culture free to take care of the things that really mattered in life,\n",
      "Line 18: such as [sports, games, romance,] studying dead languages,\n",
      "Line 19: barbarian societies and impossible problems,\n",
      "Line 20: and climbing high mountains without the aid of a safety harness.\n",
      "Line 21: \n"
     ]
    }
   ],
   "source": [
    "for linenr, line in enumerate(resource_banks.split_text(\"\\n\")):\n",
    "    linenr += 1      #make it 1-indexed as is customary for line numbers\n",
    "    print(f\"Line {linenr}: {str(line)}\")\n",
    "    store.annotate(\n",
    "        target=stam.Selector.textselector(resource_banks, line.offset()),\n",
    "        data=[ \n",
    "            {\"id\": \"Data4\", \"key\": \"structuretype\", \"value\": \"line\", \"set\": \"tutorial-set\" },\n",
    "            {\"id\": f\"DataLine{linenr}\", \"key\": \"linenr\", \"value\": linenr, \"set\": \"tutorial-set\" }\n",
    "        ],\n",
    "        id=f\"AnnotationLine{linenr}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81e44e93",
   "metadata": {},
   "source": [
    "In this example we also extended our vocabulary on-the-fly with a new field `linenr`. All line annotations carry two `AnnotationData` elements. Remember we can easily retrieve the data and any annotations on it with `data()` and `annotations()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "82d3b6c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "everything we know [and can know of] is composed ultimately of patterns of nothing;\n"
     ]
    }
   ],
   "source": [
    "line8 = dataset.data(\"linenr\", 8).annotations(limit=1)[0]\n",
    "print(str(line8))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa9a5f89",
   "metadata": {},
   "source": [
    "Methods that return collections such as `data()`,`annotations()`, `textselections()` often take an optional `limit` parameter (sometimes as a keyword argument, sometimes as a normal parameter). This parameter limits the amount of results returned. Using it can improve performance in certain cases. In the above example we know we're only going to use one result, so it is a good idea to set (here we happen to also know that there is only one result for `linenr` 8, so strictly speaking the parameter wouldn't be necessary, but we ignore that for sake of teaching the use of `limit`).\n",
    "\n",
    "When annotating, we don't have to work with the resource as a whole but can\n",
    "also start relative from any text selection we have.  Let's take line eight and\n",
    "annotate the first word of it (*\"everything\"*) manually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d2308520",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text selection spans: 92:102\n"
     ]
    }
   ],
   "source": [
    "line8_textselection = line8.textselections(limit=1)[0] #there could be multiple, but in our cases thus-far we only have one\n",
    "firstword = line8_textselection.textselection(stam.Offset.simple(0,10))  #we make a textselection on a textselection\n",
    "\n",
    "#internally, the text selection will always use absolute coordinates for the resource:\n",
    "print(f\"Text selection spans: {firstword.begin()}:{firstword.end()}\")\n",
    "\n",
    "annotation = store.annotate(\n",
    "    target=stam.Selector.textselector(resource_banks, firstword.offset()),\n",
    "    data= {\"key\": \"structuretype\", \"value\": \"word\", \"set\": \"tutorial-set\" },\n",
    "    id=f\"AnnotationLine8Word1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b233eae",
   "metadata": {},
   "source": [
    "### Converting offsets\n",
    "\n",
    "We know the first word of line eight is also part of quote one, for which we already made an annotation (`AnnotationQuote1`) before.\n",
    "Say we are interested in knowing *where* in quote one the first word of line eight is, we can now easily compute so as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5a30917c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Offset in quote one: 43:53\n"
     ]
    }
   ],
   "source": [
    "offset = firstword.relative_offset(quote1_selection)\n",
    "print(f\"Offset in quote one: {offset.begin()}:{offset.end()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e429a43b",
   "metadata": {},
   "source": [
    "While we are at it, another conversion option that may come handy when working\n",
    "on a lower-level is the conversion from/to UTF-8 byte offsets. Both STAM and\n",
    "Python use unicode character points. Internally STAM already maps these to\n",
    "UTF-8 byte offsets for things like text slicing, but if you need this\n",
    "information you can extract it explicitly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "9f87e805",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Byte offset: 92:102\n"
     ]
    }
   ],
   "source": [
    "beginbyte = resource_banks.utf8byte(firstword.begin())\n",
    "endbyte = resource_banks.utf8byte(firstword.end())\n",
    "print(f\"Byte offset: {beginbyte}:{endbyte}\")\n",
    "\n",
    "#and back again:\n",
    "beginpos = resource_banks.utf8byte_to_charpos(beginbyte)\n",
    "endpos = resource_banks.utf8byte_to_charpos(endbyte)\n",
    "\n",
    "assert beginpos == firstword.begin()\n",
    "assert endpos == firstword.end()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da98e965",
   "metadata": {},
   "source": [
    "In this case they happen to be equal because we're basically only using ASCII\n",
    "in our text, but as soon as you deal with multibyte characters (diacritics,\n",
    "other scripts, etc), they will not!\n",
    "\n",
    "### Tokenisation via regular expressions\n",
    "\n",
    "What else can we annotate? We can mark all individual words or tokens,\n",
    "effectively performing simple *tokenisation*. For this, we will use the regular\n",
    "expression search that is built into the STAM library, `find_text_regex()`. The\n",
    "regular expressions follow [Rust's regular expression\n",
    "syntax](https://docs.rs/regex/latest/regex/#syntax) which may differ slightly\n",
    "from Python's native implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "bea05e9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Annotating \"Consider\" at 3:11 as word\n",
      "Annotating \"Phlebas\" at 12:19 as word\n",
      "Annotating \"author\" at 22:28 as word\n",
      "Annotating \"Iain\" at 29:33 as word\n",
      "Annotating \"M\" at 34:35 as word\n",
      "Annotating \".\" at 35:36 as punctuation\n",
      "Annotating \"Banks\" at 37:42 as word\n",
      "Annotating \"1\" at 47:48 as word\n",
      "Annotating \"Everything\" at 49:59 as word\n",
      "Annotating \"about\" at 60:65 as word\n",
      "Annotating \"us\" at 66:68 as word\n",
      "Annotating \",\" at 68:69 as punctuation\n",
      "Annotating \"everything\" at 70:80 as word\n",
      "Annotating \"around\" at 81:87 as word\n",
      "Annotating \"us\" at 88:90 as word\n",
      "Annotating \",\" at 90:91 as punctuation\n",
      "Annotating \"everything\" at 92:102 as word\n",
      "Annotating \"we\" at 103:105 as word\n",
      "Annotating \"know\" at 106:110 as word\n",
      "Annotating \"and\" at 112:115 as word\n",
      "Annotating \"can\" at 116:119 as word\n",
      "Annotating \"know\" at 120:124 as word\n",
      "Annotating \"of\" at 125:127 as word\n",
      "Annotating \"is\" at 129:131 as word\n",
      "Annotating \"composed\" at 132:140 as word\n",
      "Annotating \"ultimately\" at 141:151 as word\n",
      "Annotating \"of\" at 152:154 as word\n",
      "Annotating \"patterns\" at 155:163 as word\n",
      "Annotating \"of\" at 164:166 as word\n",
      "Annotating \"nothing\" at 167:174 as word\n",
      "Annotating \"that\" at 176:180 as word\n",
      "Annotating \"s\" at 181:182 as word\n",
      "Annotating \"the\" at 183:186 as word\n",
      "Annotating \"bottom\" at 187:193 as word\n",
      "Annotating \"line\" at 194:198 as word\n",
      "Annotating \",\" at 198:199 as punctuation\n",
      "Annotating \"the\" at 200:203 as word\n",
      "Annotating \"final\" at 204:209 as word\n",
      "Annotating \"truth\" at 210:215 as word\n",
      "Annotating \".\" at 215:216 as punctuation\n",
      "Annotating \"So\" at 218:220 as word\n",
      "Annotating \"where\" at 221:226 as word\n",
      "Annotating \"we\" at 227:229 as word\n",
      "Annotating \"find\" at 230:234 as word\n",
      "Annotating \"we\" at 235:237 as word\n",
      "Annotating \"have\" at 238:242 as word\n",
      "Annotating \"any\" at 243:246 as word\n",
      "Annotating \"control\" at 247:254 as word\n",
      "Annotating \"over\" at 255:259 as word\n",
      "Annotating \"those\" at 260:265 as word\n",
      "Annotating \"patterns\" at 266:274 as word\n",
      "Annotating \",\" at 274:275 as punctuation\n",
      "Annotating \"why\" at 276:279 as word\n",
      "Annotating \"not\" at 280:283 as word\n",
      "Annotating \"make\" at 284:288 as word\n",
      "Annotating \"the\" at 289:292 as word\n",
      "Annotating \"most\" at 293:297 as word\n",
      "Annotating \"elegant\" at 298:305 as word\n",
      "Annotating \"ones\" at 306:310 as word\n",
      "Annotating \",\" at 310:311 as punctuation\n",
      "Annotating \"the\" at 312:315 as word\n",
      "Annotating \"most\" at 316:320 as word\n",
      "Annotating \"enjoyable\" at 321:330 as word\n",
      "Annotating \"and\" at 331:334 as word\n",
      "Annotating \"good\" at 335:339 as word\n",
      "Annotating \"ones\" at 340:344 as word\n",
      "Annotating \",\" at 344:345 as punctuation\n",
      "Annotating \"in\" at 346:348 as word\n",
      "Annotating \"our\" at 349:352 as word\n",
      "Annotating \"own\" at 353:356 as word\n",
      "Annotating \"terms\" at 357:362 as word\n",
      "Annotating \"?\" at 362:363 as punctuation\n",
      "Annotating \"2\" at 368:369 as word\n",
      "Annotating \"Besides\" at 370:377 as word\n",
      "Annotating \",\" at 377:378 as punctuation\n",
      "Annotating \"it\" at 379:381 as word\n",
      "Annotating \"left\" at 382:386 as word\n",
      "Annotating \"the\" at 387:390 as word\n",
      "Annotating \"humans\" at 391:397 as word\n",
      "Annotating \"in\" at 398:400 as word\n",
      "Annotating \"the\" at 401:404 as word\n",
      "Annotating \"Culture\" at 405:412 as word\n",
      "Annotating \"free\" at 413:417 as word\n",
      "Annotating \"to\" at 418:420 as word\n",
      "Annotating \"take\" at 421:425 as word\n",
      "Annotating \"care\" at 426:430 as word\n",
      "Annotating \"of\" at 431:433 as word\n",
      "Annotating \"the\" at 434:437 as word\n",
      "Annotating \"things\" at 438:444 as word\n",
      "Annotating \"that\" at 445:449 as word\n",
      "Annotating \"really\" at 450:456 as word\n",
      "Annotating \"mattered\" at 457:465 as word\n",
      "Annotating \"in\" at 466:468 as word\n",
      "Annotating \"life\" at 469:473 as word\n",
      "Annotating \",\" at 473:474 as punctuation\n",
      "Annotating \"such\" at 475:479 as word\n",
      "Annotating \"as\" at 480:482 as word\n",
      "Annotating \"sports\" at 484:490 as word\n",
      "Annotating \",\" at 490:491 as punctuation\n",
      "Annotating \"games\" at 492:497 as word\n",
      "Annotating \",\" at 497:498 as punctuation\n",
      "Annotating \"romance\" at 499:506 as word\n",
      "Annotating \",\" at 506:507 as punctuation\n",
      "Annotating \"studying\" at 509:517 as word\n",
      "Annotating \"dead\" at 518:522 as word\n",
      "Annotating \"languages\" at 523:532 as word\n",
      "Annotating \",\" at 532:533 as punctuation\n",
      "Annotating \"barbarian\" at 534:543 as word\n",
      "Annotating \"societies\" at 544:553 as word\n",
      "Annotating \"and\" at 554:557 as word\n",
      "Annotating \"impossible\" at 558:568 as word\n",
      "Annotating \"problems\" at 569:577 as word\n",
      "Annotating \",\" at 577:578 as punctuation\n",
      "Annotating \"and\" at 579:582 as word\n",
      "Annotating \"climbing\" at 583:591 as word\n",
      "Annotating \"high\" at 592:596 as word\n",
      "Annotating \"mountains\" at 597:606 as word\n",
      "Annotating \"without\" at 607:614 as word\n",
      "Annotating \"the\" at 615:618 as word\n",
      "Annotating \"aid\" at 619:622 as word\n",
      "Annotating \"of\" at 623:625 as word\n",
      "Annotating \"a\" at 626:627 as word\n",
      "Annotating \"safety\" at 628:634 as word\n",
      "Annotating \"harness\" at 635:642 as word\n",
      "Annotating \".\" at 642:643 as punctuation\n"
     ]
    }
   ],
   "source": [
    "expressions = [\n",
    "    r\"\\w+(?:[-_]\\w+)*\", #this detects words,possibly with hyphens or underscores as part of it\n",
    "    r\"[\\.\\?,/]+\", #this detects a variety of punctuation\n",
    "    r\"[0-9]+(?:[,\\.][0-9]+)*\", #this detects numbers, possibly with a fractional part\n",
    "]\n",
    "structuretypes = [\"word\", \"punctuation\", \"number\"]\n",
    "\n",
    "for i, matchresult in enumerate(resource_banks.find_text_regex(expressions)):\n",
    "    #(we only have one textselection per match, but an regular expression may result in multiple textselections if capture groups are used)\n",
    "    textselection = matchresult['textselections'][0]\n",
    "    structuretype = structuretypes[matchresult['expression_index']]\n",
    "    print(f\"Annotating \\\"{textselection}\\\" at {textselection.offset()} as {structuretype}\")\n",
    "    store.annotate(\n",
    "        target=stam.Selector.textselector(resource_banks, textselection.offset()),\n",
    "        data=[ \n",
    "            {\"key\": \"structuretype\", \"value\": structuretype, \"set\": \"tutorial-set\" }\n",
    "        ],\n",
    "        id=f\"AnnotationToken{i+1}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "275a37c2",
   "metadata": {},
   "source": [
    "In this code, each `matchresult` tracks which of the three expressions was\n",
    "matches, in `matchresult['expression_index']`. We conveniently use that\n",
    "information to tie new values for `structuretype`, all of which will be added\n",
    "to our vocabulary (`AnnotationDataSet`) on-the-fly."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "def7193b",
   "metadata": {},
   "source": [
    "### Annotating Metadata\n",
    "\n",
    "Thus-far we have only seen annotations directly on the text, using\n",
    "`Selector.textselector()`, but STAM has various other selectors. Users may\n",
    "appreciate if you add a bit of metadata about your texts. In STAM, these are\n",
    "annotations that point at the resource as a whole using a\n",
    "`Selector.resourceselector()`, rather than at the text specifically. We add one\n",
    "metadata annotation with various new fields:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "ae1ab36e",
   "metadata": {},
   "outputs": [],
   "source": [
    "annotation = store.annotate(\n",
    "    target=stam.Selector.resourceselector(resource_banks),\n",
    "    data=[ \n",
    "        {\"key\": \"name\", \"value\": \"Culture quotes from Iain Banks\", \"set\": \"tutorial-set\" },\n",
    "        {\"key\": \"compiler\", \"value\": \"Dirk Roorda\", \"set\": \"tutorial-set\" },\n",
    "        {\"key\": \"source\", \"value\": \"https://www.goodreads.com/work/quotes/14366-consider-phlebas\", \"set\": \"tutorial-set\" },\n",
    "        {\"key\": \"version\", \"value\": \"0.2\", \"set\": \"tutorial-set\" },\n",
    "    ],\n",
    "    id=\"Metadata1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f61f8f90",
   "metadata": {},
   "source": [
    "Similarly, we could annotate an `AnnotationDataSet` (our vocabulary) with metadata, using a `Selector.datasetselector()`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abc99264",
   "metadata": {},
   "source": [
    "## Navigating through your data\n",
    "\n",
    "### Basic iterating and counting\n",
    "\n",
    "If you followed all of the previous section, we now have a fair amount of annotations. In fact, we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "8fc7c8e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "153 annotations\n",
      "1 resource\n",
      "1 annotation dataset\n",
      "6 datakeys in our dataset\n",
      "31 annotationdata instances in our dataset\n"
     ]
    }
   ],
   "source": [
    "print(f\"{store.annotations_len()} annotations\")\n",
    "print(f\"{store.resources_len()} resource\")\n",
    "print(f\"{store.datasets_len()} annotation dataset\")\n",
    "print(f\"{dataset.keys_len()} datakeys in our dataset\")\n",
    "print(f\"{dataset.data_len()} annotationdata instances in our dataset\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cddbe8a",
   "metadata": {},
   "source": [
    "If we zoom in on the annotation data in our annotation dataset, we can extract some interesting frequency statistics right away:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "a8d0e2c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "structuretype: titleheader occurs in 1 annotation(s)\n",
      "structuretype: sectionheader occurs in 2 annotation(s)\n",
      "structuretype: quote occurs in 2 annotation(s)\n",
      "structuretype: line occurs in 21 annotation(s)\n",
      "linenr: 1 occurs in 1 annotation(s)\n",
      "linenr: 2 occurs in 1 annotation(s)\n",
      "linenr: 3 occurs in 1 annotation(s)\n",
      "linenr: 4 occurs in 1 annotation(s)\n",
      "linenr: 5 occurs in 1 annotation(s)\n",
      "linenr: 6 occurs in 1 annotation(s)\n",
      "linenr: 7 occurs in 1 annotation(s)\n",
      "linenr: 8 occurs in 1 annotation(s)\n",
      "linenr: 9 occurs in 1 annotation(s)\n",
      "linenr: 10 occurs in 1 annotation(s)\n",
      "linenr: 11 occurs in 1 annotation(s)\n",
      "linenr: 12 occurs in 1 annotation(s)\n",
      "linenr: 13 occurs in 1 annotation(s)\n",
      "linenr: 14 occurs in 1 annotation(s)\n",
      "linenr: 15 occurs in 1 annotation(s)\n",
      "linenr: 16 occurs in 1 annotation(s)\n",
      "linenr: 17 occurs in 1 annotation(s)\n",
      "linenr: 18 occurs in 1 annotation(s)\n",
      "linenr: 19 occurs in 1 annotation(s)\n",
      "linenr: 20 occurs in 1 annotation(s)\n",
      "linenr: 21 occurs in 1 annotation(s)\n",
      "structuretype: word occurs in 109 annotation(s)\n",
      "structuretype: punctuation occurs in 17 annotation(s)\n",
      "name: Culture quotes from Iain Banks occurs in 1 annotation(s)\n",
      "compiler: Dirk Roorda occurs in 1 annotation(s)\n",
      "source: https://www.goodreads.com/work/quotes/14366-consider-phlebas occurs in 1 annotation(s)\n",
      "version: 0.2 occurs in 1 annotation(s)\n"
     ]
    }
   ],
   "source": [
    "for data in dataset:\n",
    "    count = data.annotations_len()\n",
    "    print(f\"{data.key()}: {data.value()} occurs in {count} annotation(s)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "675d2442",
   "metadata": {},
   "source": [
    "We can also aggregate only by key, although that is slightly less informative for our example case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "7df1af54",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "structuretype occurs in 152 annotation(s)\n",
      "linenr occurs in 21 annotation(s)\n",
      "name occurs in 1 annotation(s)\n",
      "compiler occurs in 1 annotation(s)\n",
      "source occurs in 1 annotation(s)\n",
      "version occurs in 1 annotation(s)\n"
     ]
    }
   ],
   "source": [
    "for key in dataset.keys():\n",
    "    count = key.annotations_count()   #this one is called _count instead of _len because it is not instantaneous like the other one\n",
    "    print(f\"{key} occurs in {count} annotation(s)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa471847",
   "metadata": {},
   "source": [
    "Just like we iterated over the annotation dataset above, we can also iterate over various things in the `AnnotationStore`. Let's write a small script that simply prints out most of the things in our store. At this point though, the output will get a bit verbose:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "62830d7d",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Datasets:\n",
      " - ID: tutorial-set\n",
      "Resources:\n",
      " - ID: banks\n",
      " - Text length: 644\n",
      "Annotations:\n",
      " - ID: Annotation1\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 1:19\n",
      "   Target text: ['# Consider Phlebas']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data1\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: titleheader\n",
      " - ID: Annotation2\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 44:48\n",
      "   Target text: ['## 1']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data2\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: sectionheader\n",
      " - ID: Annotation3\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 365:369\n",
      "   Target text: ['## 2']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data2\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: sectionheader\n",
      " - ID: AnnotationQuote1\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 49:363\n",
      "   Target text: ['Everything about us,\\neverything around us,\\neverything we know [and can know of] is composed ultimately of patterns of nothing;\\nthat’s the bottom line, the final truth.\\n\\nSo where we find we have any control over those patterns,\\nwhy not make the most elegant ones, the most enjoyable and good ones,\\nin our own terms?']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data3\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: quote\n",
      " - ID: AnnotationQuote2\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 370:643\n",
      "   Target text: ['Besides,\\nit left the humans in the Culture free to take care of the things that really mattered in life,\\nsuch as [sports, games, romance,] studying dead languages,\\nbarbarian societies and impossible problems,\\nand climbing high mountains without the aid of a safety harness.']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data3\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: quote\n",
      " - ID: AnnotationLine1\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 0:0\n",
      "   Target text: ['']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine1\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 1\n",
      " - ID: AnnotationLine2\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 1:19\n",
      "   Target text: ['# Consider Phlebas']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine2\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 2\n",
      " - ID: AnnotationLine3\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 20:42\n",
      "   Target text: ['$ author=Iain M. Banks']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine3\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 3\n",
      " - ID: AnnotationLine4\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 43:43\n",
      "   Target text: ['']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine4\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 4\n",
      " - ID: AnnotationLine5\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 44:48\n",
      "   Target text: ['## 1']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine5\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 5\n",
      " - ID: AnnotationLine6\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 49:69\n",
      "   Target text: ['Everything about us,']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine6\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 6\n",
      " - ID: AnnotationLine7\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 70:91\n",
      "   Target text: ['everything around us,']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine7\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 7\n",
      " - ID: AnnotationLine8\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 92:175\n",
      "   Target text: ['everything we know [and can know of] is composed ultimately of patterns of nothing;']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine8\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 8\n",
      " - ID: AnnotationLine9\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 176:216\n",
      "   Target text: ['that’s the bottom line, the final truth.']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine9\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 9\n",
      " - ID: AnnotationLine10\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 217:217\n",
      "   Target text: ['']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine10\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 10\n",
      " - ID: AnnotationLine11\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 218:275\n",
      "   Target text: ['So where we find we have any control over those patterns,']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine11\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 11\n",
      " - ID: AnnotationLine12\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 276:345\n",
      "   Target text: ['why not make the most elegant ones, the most enjoyable and good ones,']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine12\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 12\n",
      " - ID: AnnotationLine13\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 346:363\n",
      "   Target text: ['in our own terms?']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine13\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 13\n",
      " - ID: AnnotationLine14\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 364:364\n",
      "   Target text: ['']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine14\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 14\n",
      " - ID: AnnotationLine15\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 365:369\n",
      "   Target text: ['## 2']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine15\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 15\n",
      " - ID: AnnotationLine16\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 370:378\n",
      "   Target text: ['Besides,']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine16\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 16\n",
      " - ID: AnnotationLine17\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 379:474\n",
      "   Target text: ['it left the humans in the Culture free to take care of the things that really mattered in life,']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine17\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 17\n",
      " - ID: AnnotationLine18\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 475:533\n",
      "   Target text: ['such as [sports, games, romance,] studying dead languages,']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine18\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 18\n",
      " - ID: AnnotationLine19\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 534:578\n",
      "   Target text: ['barbarian societies and impossible problems,']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine19\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 19\n",
      " - ID: AnnotationLine20\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 579:643\n",
      "   Target text: ['and climbing high mountains without the aid of a safety harness.']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine20\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 20\n",
      " - ID: AnnotationLine21\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 644:644\n",
      "   Target text: ['']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  Data4\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: line\n",
      "    - ID:  DataLine21\n",
      "      Set: tutorial-set\n",
      "      Key: linenr\n",
      "      Value: 21\n",
      " - ID: AnnotationLine8Word1\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 92:102\n",
      "   Target text: ['everything']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken1\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 3:11\n",
      "   Target text: ['Consider']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken2\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 12:19\n",
      "   Target text: ['Phlebas']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken3\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 22:28\n",
      "   Target text: ['author']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken4\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 29:33\n",
      "   Target text: ['Iain']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken5\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 34:35\n",
      "   Target text: ['M']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken6\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 35:36\n",
      "   Target text: ['.']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken7\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 37:42\n",
      "   Target text: ['Banks']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken8\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 47:48\n",
      "   Target text: ['1']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken9\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 49:59\n",
      "   Target text: ['Everything']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken10\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 60:65\n",
      "   Target text: ['about']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken11\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 66:68\n",
      "   Target text: ['us']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken12\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 68:69\n",
      "   Target text: [',']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken13\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 70:80\n",
      "   Target text: ['everything']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken14\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 81:87\n",
      "   Target text: ['around']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken15\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 88:90\n",
      "   Target text: ['us']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken16\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 90:91\n",
      "   Target text: [',']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken17\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 92:102\n",
      "   Target text: ['everything']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken18\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 103:105\n",
      "   Target text: ['we']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken19\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 106:110\n",
      "   Target text: ['know']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken20\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 112:115\n",
      "   Target text: ['and']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken21\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 116:119\n",
      "   Target text: ['can']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken22\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 120:124\n",
      "   Target text: ['know']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken23\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 125:127\n",
      "   Target text: ['of']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken24\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 129:131\n",
      "   Target text: ['is']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken25\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 132:140\n",
      "   Target text: ['composed']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken26\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 141:151\n",
      "   Target text: ['ultimately']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken27\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 152:154\n",
      "   Target text: ['of']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken28\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 155:163\n",
      "   Target text: ['patterns']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken29\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 164:166\n",
      "   Target text: ['of']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken30\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 167:174\n",
      "   Target text: ['nothing']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken31\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 176:180\n",
      "   Target text: ['that']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken32\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 181:182\n",
      "   Target text: ['s']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken33\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 183:186\n",
      "   Target text: ['the']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken34\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 187:193\n",
      "   Target text: ['bottom']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken35\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 194:198\n",
      "   Target text: ['line']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken36\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 198:199\n",
      "   Target text: [',']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken37\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 200:203\n",
      "   Target text: ['the']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken38\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 204:209\n",
      "   Target text: ['final']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken39\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 210:215\n",
      "   Target text: ['truth']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken40\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 215:216\n",
      "   Target text: ['.']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken41\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 218:220\n",
      "   Target text: ['So']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken42\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 221:226\n",
      "   Target text: ['where']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken43\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 227:229\n",
      "   Target text: ['we']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken44\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 230:234\n",
      "   Target text: ['find']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken45\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 235:237\n",
      "   Target text: ['we']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken46\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 238:242\n",
      "   Target text: ['have']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken47\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 243:246\n",
      "   Target text: ['any']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken48\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 247:254\n",
      "   Target text: ['control']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken49\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 255:259\n",
      "   Target text: ['over']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken50\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 260:265\n",
      "   Target text: ['those']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken51\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 266:274\n",
      "   Target text: ['patterns']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken52\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 274:275\n",
      "   Target text: [',']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken53\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 276:279\n",
      "   Target text: ['why']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken54\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 280:283\n",
      "   Target text: ['not']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken55\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 284:288\n",
      "   Target text: ['make']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken56\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 289:292\n",
      "   Target text: ['the']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken57\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 293:297\n",
      "   Target text: ['most']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken58\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 298:305\n",
      "   Target text: ['elegant']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken59\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 306:310\n",
      "   Target text: ['ones']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken60\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 310:311\n",
      "   Target text: [',']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken61\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 312:315\n",
      "   Target text: ['the']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken62\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 316:320\n",
      "   Target text: ['most']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken63\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 321:330\n",
      "   Target text: ['enjoyable']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken64\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 331:334\n",
      "   Target text: ['and']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken65\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 335:339\n",
      "   Target text: ['good']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken66\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 340:344\n",
      "   Target text: ['ones']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken67\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 344:345\n",
      "   Target text: [',']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken68\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 346:348\n",
      "   Target text: ['in']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken69\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 349:352\n",
      "   Target text: ['our']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken70\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 353:356\n",
      "   Target text: ['own']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken71\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 357:362\n",
      "   Target text: ['terms']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken72\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 362:363\n",
      "   Target text: ['?']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken73\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 368:369\n",
      "   Target text: ['2']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken74\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 370:377\n",
      "   Target text: ['Besides']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken75\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 377:378\n",
      "   Target text: [',']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken76\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 379:381\n",
      "   Target text: ['it']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken77\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 382:386\n",
      "   Target text: ['left']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken78\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 387:390\n",
      "   Target text: ['the']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken79\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 391:397\n",
      "   Target text: ['humans']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken80\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 398:400\n",
      "   Target text: ['in']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken81\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 401:404\n",
      "   Target text: ['the']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken82\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 405:412\n",
      "   Target text: ['Culture']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken83\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 413:417\n",
      "   Target text: ['free']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken84\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 418:420\n",
      "   Target text: ['to']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken85\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 421:425\n",
      "   Target text: ['take']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken86\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 426:430\n",
      "   Target text: ['care']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken87\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 431:433\n",
      "   Target text: ['of']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken88\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 434:437\n",
      "   Target text: ['the']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken89\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 438:444\n",
      "   Target text: ['things']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken90\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 445:449\n",
      "   Target text: ['that']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken91\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 450:456\n",
      "   Target text: ['really']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken92\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 457:465\n",
      "   Target text: ['mattered']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken93\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 466:468\n",
      "   Target text: ['in']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken94\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 469:473\n",
      "   Target text: ['life']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken95\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 473:474\n",
      "   Target text: [',']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken96\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 475:479\n",
      "   Target text: ['such']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken97\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 480:482\n",
      "   Target text: ['as']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken98\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 484:490\n",
      "   Target text: ['sports']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken99\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 490:491\n",
      "   Target text: [',']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken100\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 492:497\n",
      "   Target text: ['games']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken101\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 497:498\n",
      "   Target text: [',']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken102\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 499:506\n",
      "   Target text: ['romance']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken103\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 506:507\n",
      "   Target text: [',']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken104\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 509:517\n",
      "   Target text: ['studying']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken105\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 518:522\n",
      "   Target text: ['dead']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken106\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 523:532\n",
      "   Target text: ['languages']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken107\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 532:533\n",
      "   Target text: [',']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken108\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 534:543\n",
      "   Target text: ['barbarian']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken109\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 544:553\n",
      "   Target text: ['societies']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken110\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 554:557\n",
      "   Target text: ['and']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken111\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 558:568\n",
      "   Target text: ['impossible']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken112\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 569:577\n",
      "   Target text: ['problems']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken113\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 577:578\n",
      "   Target text: [',']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: AnnotationToken114\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 579:582\n",
      "   Target text: ['and']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken115\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 583:591\n",
      "   Target text: ['climbing']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken116\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 592:596\n",
      "   Target text: ['high']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken117\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 597:606\n",
      "   Target text: ['mountains']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken118\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 607:614\n",
      "   Target text: ['without']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken119\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 615:618\n",
      "   Target text: ['the']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken120\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 619:622\n",
      "   Target text: ['aid']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken121\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 623:625\n",
      "   Target text: ['of']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken122\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 626:627\n",
      "   Target text: ['a']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken123\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 628:634\n",
      "   Target text: ['safety']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken124\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e511af70>]\n",
      "   Target offset: 635:642\n",
      "   Target text: ['harness']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: word\n",
      " - ID: AnnotationToken125\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e52a7bd0>\n",
      "   Target resources: [<stam.TextResource object at 0x7840e52a7bd0>]\n",
      "   Target offset: 642:643\n",
      "   Target text: ['.']\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: structuretype\n",
      "      Value: punctuation\n",
      " - ID: Metadata1\n",
      "   Target selector type: <stam.SelectorKind object at 0x7840e511af70>\n",
      "   Target resources: []\n",
      "   Target offset: None\n",
      "   Target text: []\n",
      "   Target annotations:  []\n",
      "   Data:\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: name\n",
      "      Value: Culture quotes from Iain Banks\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: compiler\n",
      "      Value: Dirk Roorda\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: source\n",
      "      Value: https://www.goodreads.com/work/quotes/14366-consider-phlebas\n",
      "    - ID:  None\n",
      "      Set: tutorial-set\n",
      "      Key: version\n",
      "      Value: 0.2\n"
     ]
    }
   ],
   "source": [
    "print(\"Datasets:\")\n",
    "for dataset in store.datasets():\n",
    "    print(f\" - ID: {dataset.id()}\")\n",
    "\n",
    "print(\"Resources:\")\n",
    "for resource in store.resources():\n",
    "    print(f\" - ID: {resource.id()}\")\n",
    "    print(f\" - Text length: {resource.textlen()}\")\n",
    "\n",
    "print(\"Annotations:\")\n",
    "for annotation in store.annotations():\n",
    "    print(f\" - ID: {annotation.id()}\")\n",
    "    print(f\"   Target selector type: {annotation.selector_kind()}\")\n",
    "    print(f\"   Target resources: {annotation.resources()}\")\n",
    "    print(f\"   Target offset: {annotation.offset()}\")\n",
    "    print(f\"   Target text: {annotation.text()}\")\n",
    "    print(f\"   Target annotations: \", [ a.id() in annotation.annotations_in_targets() ])\n",
    "    print(f\"   Data:\")\n",
    "    for data in annotation:\n",
    "        print(f\"    - ID:  {data.id()}\")\n",
    "        print(f\"      Set: {data.dataset().id()}\")\n",
    "        print(f\"      Key: {data.key()}\")\n",
    "        print(f\"      Value: {data.value()}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db1e09e4",
   "metadata": {},
   "source": [
    "### Finding data\n",
    "\n",
    "We already introduced the methods `annotations()`, `data()` and `textselections()` in a previous sections.\n",
    "They return collections, classes like `Annotations`, `Data` or `TextSelections`, which in turn \n",
    "contain instances of `Annotation`, `AnnotationData`, and `TextSelection`,\n",
    "respectively.\n",
    "\n",
    "Internally the STAM library maintains various forward and reverse indices,\n",
    "representing relationships between all kinds of entities in the STAM model. The\n",
    "aforementioned methods operate via these indices.\n",
    "\n",
    "The `annotations()` method is often a lookup via the reverse index. We have\n",
    "already seen some example of it. Another nice example of the reverse index is\n",
    "that it allows us to obtain annotations for any arbitrary selection of the text\n",
    "we make:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "e91592d4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " - ID: AnnotationToken28\n",
      "   Text: patterns\n",
      "   Data:\n",
      "      structuretype=word\n"
     ]
    }
   ],
   "source": [
    "textselection = resource_banks.textselection(stam.Offset.simple(155,163))\n",
    "for annotation in textselection.annotations():\n",
    "    print(f\" - ID: {annotation.id()}\")\n",
    "    print(f\"   Text: {str(annotation)}\")\n",
    "    print(f\"   Data:\")\n",
    "    for data in annotation:\n",
    "        print(f\"      {data.key()}={data.value()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec178c1a",
   "metadata": {},
   "source": [
    "Of course, I cheated a bit here and knew in advance there was going to be a\n",
    "match for this offset, but the point to take home is that given any\n",
    "*textselection*, you can easily get annotations that reference it.\n",
    "\n",
    "In the above example we iterate over all annotations and then over all the data\n",
    "pertaining to the found annotations. Often though, you are searching for\n",
    "specific data and would have some kind of extra test in there. This is\n",
    "accomplished by passing filters via positional arguments or keyword arguments\n",
    "like `value`, to the `annotations()` method. We have seen an example of this\n",
    "before, here is another:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9081fa44",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " - ID: AnnotationToken28\n",
      "   Text: patterns\n"
     ]
    }
   ],
   "source": [
    "textselection = resource_banks.textselection(stam.Offset.simple(155,163))\n",
    "dataset = store.dataset(\"tutorial-set\")\n",
    "key = dataset.key(\"structuretype\")\n",
    "for annotation in textselection.annotations(key, value=\"word\"):\n",
    "    print(f\" - ID: {annotation.id()}\")\n",
    "    print(f\"   Text: {str(annotation)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36ec74d1",
   "metadata": {},
   "source": [
    "The use of filters in methods like `annotations()` and `data()` is always\n",
    "preferable to manually writing it out in lower-level code, because the internal\n",
    "library is more performant and passing data back and forth to Python always\n",
    "comes with a performance penalty.\n",
    "\n",
    "In the example above, however, we see that we filter on data, but do not actually get the data that was matched as a return value. If you do want that, you need a two-step process as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "bc1ccc1f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " - ID: AnnotationToken28\n",
      "   Text: patterns\n",
      "   Data: word\n"
     ]
    }
   ],
   "source": [
    "textselection = resource_banks.textselection(stam.Offset.simple(155,163))\n",
    "dataset = store.dataset(\"tutorial-set\")\n",
    "key = dataset.key(\"structuretype\")\n",
    "for annotation in textselection.annotations(key, value=\"word\"):\n",
    "    print(f\" - ID: {annotation.id()}\")\n",
    "    print(f\"   Text: {str(annotation)}\")\n",
    "    annotationdata = annotation.data(key, value=\"word\",limit=1)[0]\n",
    "    print(f\"   Data: {str(annotationdata)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a75be79",
   "metadata": {},
   "source": [
    "Sometimes you don't really care to retrieve the data or the annotations, but\n",
    "merely want to test whether certain data is present on an annotation and return\n",
    "a boolean. For this use can use methods like `test_annotations()` and `test_data()`, which take the same\n",
    "keyword parameters for filtering as their counterparts `annotations()` and `data()`, but instead of returning a collection, it simply returns a boolean, which is more performant.\n",
    "\n",
    "This following example confirms to us that the textselection is indeed a word:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "66157fba",
   "metadata": {},
   "outputs": [],
   "source": [
    "textselection = resource_banks.textselection(stam.Offset.simple(155,163))\n",
    "dataset = store.dataset(\"tutorial-set\")\n",
    "key = dataset.key(\"structuretype\")\n",
    "assert textselection.test_data(key, value=\"word\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7698575",
   "metadata": {},
   "source": [
    "It is possible to retrieve all *known* text selections for a given\n",
    "resource. A text selection is 'known' if there is at least one annotation that\n",
    "references it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "7dc7b0f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "# Consider Phlebas\n",
      "Consider\n",
      "Phlebas\n",
      "$ author=Iain M. Banks\n",
      "author\n",
      "Iain\n",
      "M\n",
      ".\n",
      "Banks\n",
      "\n",
      "## 1\n",
      "1\n",
      "Everything about us,\n",
      "everything around us,\n",
      "everything we know [and can know of] is composed ultimately of patterns of nothing;\n",
      "that’s the bottom line, the final truth.\n",
      "\n",
      "So where we find we have any control over those patterns,\n",
      "why not make the most elegant ones, the most enjoyable and good ones,\n",
      "in our own terms?\n",
      "Everything about us,\n",
      "Everything\n",
      "about\n",
      "us\n",
      ",\n",
      "everything around us,\n",
      "everything\n",
      "around\n",
      "us\n",
      ",\n",
      "everything we know [and can know of] is composed ultimately of patterns of nothing;\n",
      "everything\n",
      "we\n",
      "know\n",
      "and\n",
      "can\n",
      "know\n",
      "of\n",
      "is\n",
      "composed\n",
      "ultimately\n",
      "of\n",
      "patterns\n",
      "of\n",
      "nothing\n",
      "that’s the bottom line, the final truth.\n",
      "that\n",
      "s\n",
      "the\n",
      "bottom\n",
      "line\n",
      ",\n",
      "the\n",
      "final\n",
      "truth\n",
      ".\n",
      "\n",
      "So where we find we have any control over those patterns,\n",
      "So\n",
      "where\n",
      "we\n",
      "find\n",
      "we\n",
      "have\n",
      "any\n",
      "control\n",
      "over\n",
      "those\n",
      "patterns\n",
      ",\n",
      "why not make the most elegant ones, the most enjoyable and good ones,\n",
      "why\n",
      "not\n",
      "make\n",
      "the\n",
      "most\n",
      "elegant\n",
      "ones\n",
      ",\n",
      "the\n",
      "most\n",
      "enjoyable\n",
      "and\n",
      "good\n",
      "ones\n",
      ",\n",
      "in our own terms?\n",
      "in\n",
      "our\n",
      "own\n",
      "terms\n",
      "?\n",
      "\n",
      "## 2\n",
      "2\n",
      "Besides,\n",
      "it left the humans in the Culture free to take care of the things that really mattered in life,\n",
      "such as [sports, games, romance,] studying dead languages,\n",
      "barbarian societies and impossible problems,\n",
      "and climbing high mountains without the aid of a safety harness.\n",
      "Besides,\n",
      "Besides\n",
      ",\n",
      "it left the humans in the Culture free to take care of the things that really mattered in life,\n",
      "it\n",
      "left\n",
      "the\n",
      "humans\n",
      "in\n",
      "the\n",
      "Culture\n",
      "free\n",
      "to\n",
      "take\n",
      "care\n",
      "of\n",
      "the\n",
      "things\n",
      "that\n",
      "really\n",
      "mattered\n",
      "in\n",
      "life\n",
      ",\n",
      "such as [sports, games, romance,] studying dead languages,\n",
      "such\n",
      "as\n",
      "sports\n",
      ",\n",
      "games\n",
      ",\n",
      "romance\n",
      ",\n",
      "studying\n",
      "dead\n",
      "languages\n",
      ",\n",
      "barbarian societies and impossible problems,\n",
      "barbarian\n",
      "societies\n",
      "and\n",
      "impossible\n",
      "problems\n",
      ",\n",
      "and climbing high mountains without the aid of a safety harness.\n",
      "and\n",
      "climbing\n",
      "high\n",
      "mountains\n",
      "without\n",
      "the\n",
      "aid\n",
      "of\n",
      "a\n",
      "safety\n",
      "harness\n",
      ".\n"
     ]
    }
   ],
   "source": [
    "for textselection in resource_banks.textselections():\n",
    "    print(textselection)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91628cfd",
   "metadata": {},
   "source": [
    "It's easy to see how you can combine some of the examples to retrieve all\n",
    "annotations in a reverse way (i.e. via the text).\n",
    "\n",
    "You can consider a STAM model as a graph in which the annotations, resource,\n",
    "data make up the nodes. The forward indices and reverse indices encode how\n",
    "these nodes are related and form the edges of the graph. These edges can be\n",
    "traversed in almost any direction using the various methods at your disposal in\n",
    "this STAM library. Methods like `data()`,`annotations()`, `textselections()`\n",
    "and their filtering abilities, as well as their test counterparts, essential\n",
    "tools to accomplish this.\n",
    "\n",
    "### Text Relations\n",
    "\n",
    "Now we get to the fun part. When you select any two parts of a text, i.e. create two text selections, then between these\n",
    "text selections there can be a number of *relationships* that hold true or not:\n",
    "\n",
    "* The text selections may overlap\n",
    "* The text selections may be embedded entirely in one another (one overlaps fully with the other)\n",
    "* The text selections may come before or after another with any amount of distance in between\n",
    "* The text selections may succeed or precede another, one's end is the other's begin of vice versa.\n",
    "* The text selections may have the very same begin and/or end offset\n",
    "\n",
    "In STAM, the `TextSelectionOperator` captures these relationships.\n",
    "\n",
    "Remember our example in which we annotated the first word of line eight? The\n",
    "textselection for this word *is embedded* within the textselection for line\n",
    "eight as a whole. We can test that as follows using the `test()` method on\n",
    "`TextSelection`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "07b22240",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert firstword.test(stam.TextSelectionOperator.embedded(), line8_textselection)\n",
    "\n",
    "# the reverse then also holds:\n",
    "assert line8_textselection.test(stam.TextSelectionOperator.embeds(), firstword)\n",
    "\n",
    "# an embedding is essentially a stricter form of an overlap relation, so this holds too:\n",
    "assert firstword.test(stam.TextSelectionOperator.overlaps(), line8_textselection)\n",
    "assert line8_textselection.test(stam.TextSelectionOperator.overlaps(), firstword)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "126c1938",
   "metadata": {},
   "source": [
    "Not only can we test any given text selections, we can use this functionality\n",
    "to actively *find* text selections that are in a particular relationship with\n",
    "another, in other words we find *related text selections*. This is a\n",
    "core feature of the STAM library and a primary method of finding text\n",
    "selections and their annotations. We use the `related_text()` method for this.\n",
    "\n",
    "Let's find all text selections (which we previously annotated) in line eight:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "4bf053a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "everything @92:102\n",
      "we @103:105\n",
      "know @106:110\n",
      "and @112:115\n",
      "can @116:119\n",
      "know @120:124\n",
      "of @125:127\n",
      "is @129:131\n",
      "composed @132:140\n",
      "ultimately @141:151\n",
      "of @152:154\n",
      "patterns @155:163\n",
      "of @164:166\n",
      "nothing @167:174\n"
     ]
    }
   ],
   "source": [
    "for textselection in line8_textselection.related_text(stam.TextSelectionOperator.embeds()):\n",
    "    print(f\"{textselection} @{textselection.offset()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ea8c225",
   "metadata": {},
   "source": [
    "Often, what we are interested in is not the text selections as such, but the annotations that reference these text selections.\n",
    "Simply add `.annotations()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "3b8d59c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " - ID: AnnotationLine8Word1\n",
      "   Text: everything\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken17\n",
      "   Text: everything\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken18\n",
      "   Text: we\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken19\n",
      "   Text: know\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken20\n",
      "   Text: and\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken21\n",
      "   Text: can\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken22\n",
      "   Text: know\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken23\n",
      "   Text: of\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken24\n",
      "   Text: is\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken25\n",
      "   Text: composed\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken26\n",
      "   Text: ultimately\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken27\n",
      "   Text: of\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken28\n",
      "   Text: patterns\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken29\n",
      "   Text: of\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken30\n",
      "   Text: nothing\n",
      "   Data:\n",
      "      structuretype=word\n"
     ]
    }
   ],
   "source": [
    "for annotation in line8_textselection.related_text(stam.TextSelectionOperator.embeds()).annotations():\n",
    "    print(f\" - ID: {annotation.id()}\")\n",
    "    print(f\"   Text: {str(annotation)}\")\n",
    "    print(f\"   Data:\")\n",
    "    for data in annotation:\n",
    "        print(f\"      {data.key()}={data.value()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d711c52c",
   "metadata": {},
   "source": [
    "The `related_text()` method is available on `TextSelection` (and `TextSelections`) and `Annotation` (and `Annotations`) in\n",
    "which case the latter is again a shortcut so you don't have to retrieve the\n",
    "text selections yourself first. As said before: do use all the shortcuts the\n",
    "library offers, because the more the library can do for you, the more\n",
    "performant things are, as it's compiled to machine code and not written in\n",
    "Python itself.\n",
    "\n",
    "In the last output, you may note that we got two annotations for the first word\n",
    "of line eight, that's because we did one manually, and the other one via our\n",
    "regular-expression based tokeniser.\n",
    "\n",
    "In the previous example all we got was data with key `structuretype` and value `word`. We could have specifically selected for this by adding some filters to `annotations()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "bd43a9a3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " - ID: AnnotationLine8Word1\n",
      "   Text: everything\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken17\n",
      "   Text: everything\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationLine8Word1\n",
      "   Text: everything\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken17\n",
      "   Text: everything\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken18\n",
      "   Text: we\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken19\n",
      "   Text: know\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken20\n",
      "   Text: and\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken21\n",
      "   Text: can\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken22\n",
      "   Text: know\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken23\n",
      "   Text: of\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken24\n",
      "   Text: is\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken25\n",
      "   Text: composed\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken26\n",
      "   Text: ultimately\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken27\n",
      "   Text: of\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken28\n",
      "   Text: patterns\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken29\n",
      "   Text: of\n",
      "   Data:\n",
      "      structuretype=word\n",
      " - ID: AnnotationToken30\n",
      "   Text: nothing\n",
      "   Data:\n",
      "      structuretype=word\n"
     ]
    }
   ],
   "source": [
    "key = store.dataset(\"tutorial-set\").key(\"structuretype\")\n",
    "for annotation in line8_textselection.related_text(stam.TextSelectionOperator.embeds()).annotations(key, value=\"word\"):\n",
    "    print(f\" - ID: {annotation.id()}\")\n",
    "    print(f\"   Text: {str(annotation)}\")\n",
    "    print(f\"   Data:\")\n",
    "    for data in annotation:\n",
    "        print(f\"      {data.key()}={data.value()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76058b98",
   "metadata": {},
   "source": [
    "### Querying with STAMQL \n",
    "\n",
    "Instead of querying data using the various python objects and methods we have\n",
    "seen thus-far, it is also possible to formulate a query in a query language\n",
    "called STAMQL. The query language is `described in detail here\n",
    "<https://github.com/annotation/stam/tree/master/extensions/stam-query>_`. We\n",
    "will only cover some of the basics here and show how to call it from Python.\n",
    "\n",
    "A query starts with a `SELECT` statement, then a return type specifying what\n",
    "kind of data you want the query to return (`ANNOTATION`, `DATA`, `TEXT`,\n",
    "`KEY`,`DATASET`). Then you must specify a variable name to bind the results we obtain to (variables always start with a `?` in STMAQL) and `WHERE` statement introducing a series of one or more constraints, each ends with a semicolon. \n",
    "\n",
    "Let's illustrate all this with an example, we obtain obtain line 8 from our\n",
    "data, which we had explicitly annotated earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "6d9eca05",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ID:  AnnotationLine8\n",
      "Text:  everything we know [and can know of] is composed ultimately of patterns of nothing;\n"
     ]
    }
   ],
   "source": [
    "query = \"\"\"\n",
    "SELECT ANNOTATION ?a WHERE\n",
    "    DATA \"tutorial-set\" \"linenr\" = 8;\n",
    "\"\"\"\n",
    "\n",
    "for results in store.query(query):\n",
    "    for result in results:\n",
    "        annotation = result['a']\n",
    "        assert isinstance(annotation, stam.Annotation)\n",
    "        print(\"ID: \", annotation.id())\n",
    "        print(\"Text: \", str(annotation))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "021e5fea",
   "metadata": {},
   "source": [
    "Here we formulated a query in STAMQL and passed it to the `query()` method as a\n",
    "string, and this gives us the results back in a list of dictionaries. The keys\n",
    "in the dictionary correspond to the variable binds we chose in the `SELECT`\n",
    "statement (without the `?` prefix). In this case we obtain one result\n",
    "containing one variable `a`.\n",
    "\n",
    "Instead of querying for the annotation, we could have queried directly for the\n",
    "text as well, we could also add extra constraints that must all be satisfied:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "177c3e62",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "everything we know [and can know of] is composed ultimately of patterns of nothing;\n"
     ]
    }
   ],
   "source": [
    "query = \"\"\"\n",
    "SELECT TEXT ?t WHERE\n",
    "    DATA \"tutorial-set\" \"linenr\" = 8;\n",
    "    DATA \"tutorial-set\" \"structuretype\" = \"line\";\n",
    "\"\"\"\n",
    "\n",
    "for results in store.query(query):\n",
    "    for result in results:\n",
    "        print(result['t'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1a627d3",
   "metadata": {},
   "source": [
    "Querying for text rather than annotations has a subtle difference when you add multiple `DATA` constraints like we did above. If we query for text, then it selects text which has annotations with the specified data. The data does not necessarily have to pertain to the same annotation (as long as it covers the same text). If you query for annotations and have multiple `DATA` constraints, then a *single annotation* must have both data items.\n",
    "\n",
    "The query language supports *query composition* to chain multiple queries/subqueries together. A subquery is introduced using curled braces. Take a look at the following example where we again select line 8, and then all words in line 8 (here we use a textual overlap relation):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "72b6ea37",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ID:  AnnotationLine8Word1\n",
      "Text:  everything\n",
      "ID:  AnnotationToken17\n",
      "Text:  everything\n",
      "ID:  AnnotationToken18\n",
      "Text:  we\n",
      "ID:  AnnotationToken19\n",
      "Text:  know\n",
      "ID:  AnnotationToken20\n",
      "Text:  and\n",
      "ID:  AnnotationToken21\n",
      "Text:  can\n",
      "ID:  AnnotationToken22\n",
      "Text:  know\n",
      "ID:  AnnotationToken23\n",
      "Text:  of\n",
      "ID:  AnnotationToken24\n",
      "Text:  is\n",
      "ID:  AnnotationToken25\n",
      "Text:  composed\n",
      "ID:  AnnotationToken26\n",
      "Text:  ultimately\n",
      "ID:  AnnotationToken27\n",
      "Text:  of\n",
      "ID:  AnnotationToken28\n",
      "Text:  patterns\n",
      "ID:  AnnotationToken29\n",
      "Text:  of\n",
      "ID:  AnnotationToken30\n",
      "Text:  nothing\n"
     ]
    }
   ],
   "source": [
    "query = \"\"\"\n",
    "SELECT ANNOTATION ?line WHERE\n",
    "    DATA \"tutorial-set\" \"linenr\" = 8;\n",
    "{\n",
    "    SELECT ANNOTATION ?word WHERE\n",
    "        RELATION ?line EMBEDS;\n",
    "        DATA \"tutorial-set\" \"structuretype\" = \"word\";\n",
    "}\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "for results in store.query(query):\n",
    "    for result in results:\n",
    "        #the ?line annotation will be returns for each\n",
    "        assert 'line' in result\n",
    "        annotation = result['word']\n",
    "        assert isinstance(annotation, stam.Annotation)\n",
    "        print(\"ID: \", annotation.id())\n",
    "        print(\"Text: \", str(annotation))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7dc0943",
   "metadata": {},
   "source": [
    "The constraint ``RELATION ?line EMBEDS;`` in the subquery is essential here, it\n",
    "can be read as \"?line embeds ?word\" and ensures that there is a specific\n",
    "textual relation between the two select statements. . It is even a requirement\n",
    "in a subquery to have a constraint that refers back to the parent query. Each\n",
    "subquery can itself have a subquery to you can build long chains.\n",
    "\n",
    "Aside from `EMBEDS`, there other relations you can use such as `OVERLAPS`,\n",
    "`PRECEDES`, `SUCCEEDS`, `BEFORE`, `AFTER`, `SAMEBEGIN`, `SAMEEND`, `EQUALS`.\n",
    "These are the STAMQL keywords representing the `TextSelectionOperator` you have\n",
    "already seen before.\n",
    "\n",
    "You have the choice whether to express your queries through STAMQL or using\n",
    "Python objects and methods. Internally, the stam library will convert the\n",
    "latter to the former whenever you apply any filtering, so there is not too much\n",
    "difference performance-wise. There is some performance overhead though in the\n",
    "conversion of results when you call `query()` explicitly with a STAMQL query.\n",
    "\n",
    "When calling `query()`, you may inject context variables yourself via keyword\n",
    "arguments. These will subsequently be available to be used as constraints in\n",
    "your query. As an example, we repeat the previous query but inject the line\n",
    "variable manually, we already had an instance to it laying around anyway:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "dce9503c",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ID:  AnnotationLine8Word1\n",
      "Text:  everything\n",
      "ID:  AnnotationToken17\n",
      "Text:  everything\n",
      "ID:  AnnotationToken18\n",
      "Text:  we\n",
      "ID:  AnnotationToken19\n",
      "Text:  know\n",
      "ID:  AnnotationToken20\n",
      "Text:  and\n",
      "ID:  AnnotationToken21\n",
      "Text:  can\n",
      "ID:  AnnotationToken22\n",
      "Text:  know\n",
      "ID:  AnnotationToken23\n",
      "Text:  of\n",
      "ID:  AnnotationToken24\n",
      "Text:  is\n",
      "ID:  AnnotationToken25\n",
      "Text:  composed\n",
      "ID:  AnnotationToken26\n",
      "Text:  ultimately\n",
      "ID:  AnnotationToken27\n",
      "Text:  of\n",
      "ID:  AnnotationToken28\n",
      "Text:  patterns\n",
      "ID:  AnnotationToken29\n",
      "Text:  of\n",
      "ID:  AnnotationToken30\n",
      "Text:  nothing\n"
     ]
    }
   ],
   "source": [
    "query = \"\"\"\n",
    "SELECT ANNOTATION ?word WHERE\n",
    "    RELATION ?line EMBEDS;\n",
    "    DATA \"tutorial-set\" \"structuretype\" = \"word\";\n",
    "\"\"\"\n",
    "\n",
    "for results in store.query(query, line=line8_texselection):\n",
    "    for result in results:\n",
    "        annotation = result['word']\n",
    "        assert isinstance(annotation, stam.Annotation)\n",
    "        print(\"ID: \", annotation.id())\n",
    "        print(\"Text: \", str(annotation))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7ce0797",
   "metadata": {},
   "source": [
    "## Advanced annotation\n",
    "\n",
    "### Higher-order Annotation\n",
    "\n",
    "All annotations we have done so far reference the text as a whole with absolute\n",
    "offsets via a *TextSelector*, even though we formulated some of these offsets\n",
    "(first word of line eight) in relative terms.\n",
    "\n",
    "STAM also allows you to adopt another annotation paradigm in which you point an\n",
    "annotation not at a text via *TextSelector*, but at another annotation via an\n",
    "*AnnotationSelector*, and that other annotation, or the final one of however\n",
    "many there are in between, points at the text with a *TextSelector*. You can\n",
    "specify an offset, which will then be interpreted relative to the\n",
    "[text selection of] the targeted annotation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "ca323655",
   "metadata": {},
   "outputs": [],
   "source": [
    "line8 = store.annotation(\"AnnotationLine8\")\n",
    "annotation = store.annotate(\n",
    "    target=stam.Selector.annotationselector(line8, stam.Offset.simple(0,10)),\n",
    "    data= {\"key\": \"structuretype\", \"value\": \"word\", \"set\": \"tutorial-set\" },\n",
    "    id=f\"AnnotationLine8Word1_explicit\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df1ca685",
   "metadata": {},
   "source": [
    "Here we are effectively annotating an annotation, so we call this a form of\n",
    "*higher-order annotation*. We explicitly capture and model a relationship.\n",
    "Whether to do this explicitly or use the STAM library's functionality to\n",
    "resolve it implicitly is entirely up to you, the modeller, and your use-case!\n",
    "\n",
    "We can also do higher-order annotation to associate metadata with annotations,\n",
    "such as encoding the person who did the annotation. In such cases, we can choose\n",
    "not to reference the text at all, because the annotation no longer says something\n",
    "about the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "b8e475d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "line8 = store.annotation(\"AnnotationLine8\")\n",
    "annotation = store.annotate(\n",
    "    target=stam.Selector.annotationselector(line8),\n",
    "    data= [\n",
    "        {\"key\": \"annotator\", \"value\": \"Maarten van Gompel\", \"set\": \"tutorial-set\" },\n",
    "        {\"key\": \"datetime\", \"value\": \"2023-04-18T17:48:56\", \"set\": \"tutorial-set\" },\n",
    "    ],\n",
    "    id=f\"AnnotationAnnotator\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25254b49",
   "metadata": {},
   "source": [
    "Note that we invented some more keys that were added on-the-fly to our annotation dataset (i.e. the vocabulary).\n",
    "\n",
    "This too, needn't be a higher-order annotation, you can chose to associate the\n",
    "`AnnotationData` directly with the annotation. The idea about an annotation\n",
    "though, is that once it is made, it is immutable; no adding/editing of\n",
    "annotation data or targets at later points in time. Information such as\n",
    "annotators and date/time information could well be associated with the\n",
    "annotation upon creation, but sometimes there may be data which you want to\n",
    "associate with an annotation at a later point in time. That would be a use case\n",
    "for higher-order annotation.\n",
    "\n",
    "### Complex selectors\n",
    "\n",
    "Rather than point at a single target, sometimes you want to annotate something\n",
    "that can not be captured by a single simple selector. Take for example, again, line eight from our text:\n",
    "\n",
    "*everything we know [and can know of] is composed ultimately of patterns of nothing*\n",
    "\n",
    "Say we want to annotate the parts of the sentence without the portion in square\n",
    "brackets, then a single text selection could not capture it because it is\n",
    "discontinuous. Two text selections, however, do the job. To combine the two text selectors (or any other type of simple selector)\n",
    "STAM has the *CompositeSelector*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "351cc037",
   "metadata": {},
   "outputs": [],
   "source": [
    "part1 = line8_textselection.textselection(stam.Offset.simple(0,18))\n",
    "part2 = line8_textselection.textselection(stam.Offset.simple(37,82))\n",
    "line8mainsentence = store.annotate(\n",
    "    target=stam.Selector.compositeselector(\n",
    "        stam.Selector.textselector(resource_banks, part1.offset()),\n",
    "        stam.Selector.textselector(resource_banks, part2.offset()),\n",
    "    ),\n",
    "    data= [\n",
    "        {\"key\": \"structuretype\", \"value\": \"mainsentence\", \"set\": \"tutorial-set\" },\n",
    "    ],\n",
    "    id=f\"AnnotationLine8Mainsentence\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e8604cc",
   "metadata": {},
   "source": [
    "If we ask the STAM library to get the text using `str()`, it will concatenate\n",
    "the parts with a space, which may not always be appropriate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "7db2980a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"everything we know is composed ultimately of patterns of nothing\"\n"
     ]
    }
   ],
   "source": [
    "print(f\"\\\"{line8mainsentence}\\\"\")\n",
    "assert str(line8mainsentence) == \"everything we know is composed ultimately of patterns of nothing\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7886db14",
   "metadata": {},
   "source": [
    "Use the `text()` method instead if you want to retain the parts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "8ab868c6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['everything we know', 'is composed ultimately of patterns of nothing']\n"
     ]
    }
   ],
   "source": [
    "print(line8mainsentence.text())\n",
    "assert line8mainsentence.text() == [\"everything we know\", \"is composed ultimately of patterns of nothing\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4517412a",
   "metadata": {},
   "source": [
    "In a similar fashion, you can also call the `textselections()` methods to\n",
    "obtain all text selections. We already used this method before and remarked it\n",
    "always returns a `TextSelections` collection and not just a single text `TextSelection`, now you know why.\n",
    "\n",
    "When the composite selector is used, the target *must* be interpreted jointly;\n",
    "the annotation applies to the whole composition rather than to individual\n",
    "parts. \n",
    "\n",
    "There is also the *MultiSelector*, which selects multiple targets and the\n",
    "annotation applies to each of them individually and independently. It offers a\n",
    "convenient way to express multiple annotations more concisely, conserving\n",
    "memory usage.\n",
    "\n",
    "Last, there is the *DirectionalSelector* which expresses multiple targets with\n",
    "a very specific order that is meaningful. For example, taking line eight again,\n",
    "we can express the dependency relation where the word *ultimately* is an\n",
    "adverbial modifier to the verb *composed*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "4e7d34bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "head = line8_textselection.textselection(stam.Offset.simple(41,48))\n",
    "dependant = line8_textselection.textselection(stam.Offset.simple(50,59))\n",
    "line8mainsentence = store.annotate(\n",
    "    target=stam.Selector.directionalselector(\n",
    "        stam.Selector.textselector(resource_banks, head.offset()),\n",
    "        stam.Selector.textselector(resource_banks, dependant.offset()),\n",
    "    ),\n",
    "    data= [\n",
    "        {\"key\": \"dependency\", \"value\": \"advmod\", \"set\": \"tutorial-set\" },\n",
    "    ],\n",
    "    id=f\"AnnotationDependency\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18a1dbf1",
   "metadata": {},
   "source": [
    "You can interpret the different selectors under a directional selector akin to\n",
    "positional function parameters. You, the modeller, determine how the ordering\n",
    "is interpreted.\n",
    "\n",
    "## Editing annotations\n",
    "\n",
    "We already explained how this is a bad idea and should be avoided: the\n",
    "canonical way to edit an annotation is to remove the old annotation from the\n",
    "store and make a new one. Removing an annotation, or any other STAM object, can\n",
    "be done by passing it to `AnnotationStore.remove()`.\n",
    "\n",
    "We can dive into the motivation behind this constraint a bit more: From a\n",
    "semantic perspective annotations are essentially a commentary about something\n",
    "else. If that what you comment on is subject to change, possibly unbeknownst to\n",
    "you, then such a change might invalidate your commentary, as it is no\n",
    "longer the same thing as what you based your comment on! The STAM model\n",
    "prevents these pitfalls. \n",
    "\n",
    "Nevertheless, at the low-level there are ways around this constraint. After\n",
    "all, as long as you don't publish the annotations you have some liberty in editing them.\n",
    "Currently though, the Python library does not yet expose this.\n",
    "\n",
    "When using `AnnotationStore.remove()` on any variable, you must yourself take\n",
    "care not to use that variable again. Also note that removing an item will\n",
    "removing everything that depends on it. So if you remove an item like an\n",
    "annotation, text resource or data item, then all annotations on it and\n",
    "everything that references it will be automatically removed as well."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0cd7b88",
   "metadata": {},
   "source": [
    "## Saving and loading data\n",
    "\n",
    "All this time we've been annotating but have not committed our results to any\n",
    "form of persistent storage. You will likely want to save your annotation store\n",
    "to file, and load it all again at any later point in time.\n",
    "\n",
    "STAM's canonical serialisation format is [STAM JSON](https://github.com/annotation/stam#stam-json):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "035111a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "store.set_filename(\"tutorial.store.stam.json\")\n",
    "store.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9be14d6d",
   "metadata": {},
   "source": [
    "The `save()` method will use the filename that the annotation store was\n",
    "initially loaded from. We had none yet, so we set it via `set_filename()`\n",
    "first. In our current example, everything is saved into a single JSON file.\n",
    "\n",
    "However, the `set_filename()` method is also available on `AnnotationDataSet`\n",
    "and `TextResource`. If set, these are kept in stand-off files. Annotation data\n",
    "sets usually use STAM JSON, but text resources generally just use plain text.\n",
    "The extension you use determines the file format.\n",
    "\n",
    "There is also a [STAM CSV\n",
    "format](https://github.com/annotation/stam/tree/master/extensions/stam-csv)\n",
    "defined as an extension which is supported by this library. Whereas\n",
    "the JSON format is *very* verbose (=large files), the CSV is a bit more concise.\n",
    "\n",
    "Loading an annotation store (including all stand-off files) is as simple as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "e3d4349e",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "store2 = stam.AnnotationStore(file=\"tutorial.store.stam.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75040471",
   "metadata": {},
   "source": [
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "env",
   "language": "python",
   "name": "env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}