Permalink
Branch: master
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
336 lines (285 sloc) 17.4 KB

RDFa Annotation Client - Annotation Data Models

For the RDFa Annotation Client and its communicating with the server (e.g. Alexandria) we need to establish a data model for the annotations. The idea is to base this on the W3C Web Annotation Model.

Basic model

The most basic target is an entire resource, such as a single van Gogh letter.

{
  "@context": "http://www.w3.org/ns/anno.jsonld",
  "created": 1484229754,
  "body": [
    {
      "vocabulary": "DBpedia",
      "value": "Communication",
      "purpose": "classifying",
      "id": "http://dbpedia.org/resource/Communication"
    }
  ],
  "motivation": "classifying",
  "creator": "marijn",
  "type": "Annotation",
  "target": [
    {
      "type": "Text",
      "source": "urn:vangogh:let001"
    }
  ],
  "id": "urn:uuid:e9d01a37-9dca-4f96-bc6a-5c44012d6e9e"
}

Some discussion points:

  • Marijn: the value of the created property is generated by the annotation server. In the current prototype I'm using seconds since epoch, but Alexandria probably has a different format.
  • Marijn: In the body I've added a few extra properties, i.e. type, value and vocabulary so that the annotation describes itself more (easier to interpret) and the info can be displayed to the user without needing to do a DBpedia lookup.
  • Marijn: The motivation classifying is taken from the official W3C list of motivations. I don't like this interpretation of motivation. Classifying is not a motivation, i.e., it doesn't say why or in what context I'm classifying the target. The W3C framework also has a property purpose which it only allows as the property of a body of type TextualBody or SpecificResource. I would prefer to use purpose as a property on all bodies [This makes me a bit worried on how you want to count annotations. For the user, I'd think that each body would count as an annotation. It would be confusing to have a single annotation, just because the annotation share a target. PB], since that conveys better what the purpose of the annotation is. When classifying, the purpose of an annotation is to classify a resource. It also makes more sense with multiple bodies, allowing a different purpose per annotation body, e.g. classifying and commenting in a single annotation have different purposes. [But as we discussed earlier, 'pupose' or motivation can be used for many different sorts of things. 'use in thesis chapter 2' is also a motivation. The W3C motivations are more like an ontological description of the possible functions of annotations. PB]

Annotations linking and structural relationships between resources and sub-resources

The RDFa Annotation Client uses a special type of annotation to register part-of relationships between annotatable resources.

The W3C Motivation is linking an annotatable resource to an annotatable part of it. The larger resource is the annotation target, the part is the annotation body. The reason for making the larger resource the target is that it allows a uniform recursive lookup operation for annotations on a resource:

  1. get the set A1 of all annotations where resource r is the target,
  2. recursively get all annotations An of part-of resources of r that are annotation bodies in An-1

Steps:

  • The annotation client scans for resource information in RDFa attributes in elements with a annotation-target-observer class.
  • The annotation client parses the annotatable things ontology and selects annotatable resources within the annotation-target-observer elements.
  • The annotation client sends a request to the annotation server per annotatable resource to GET all annotations on that resource.
  • The annotation server responds with a list of annotations, including:
    • annotations indicating relations between the resource and sub-resources,
    • annotations on resource and on any of its sub-resources,
    • recursively: annotations on annotations in the previous step
  • The annotation client does not display annotations that indicate the structural relation between resource and sub-resource, i.e. linking annotations with annotation bodies that are annotatable things in the annotation-target-observer window.

Below is old material, needs updated based on the structural relationship annotations in the previous section

Fragment selector

When the annotation targets a resource that is part of a larger resource, a fragment selector is used to refine the target.

In the example below, the receiver metadata of the letter is the annotation target, identified through a FragmentSelector on the letter:

{
  "@context": "http://www.w3.org/ns/anno.jsonld",
  "created": 1483949925,
  "body": [
    {
      "vocabulary": "DBpedia",
      "value": "Theo van Gogh (art dealer)",
      "purpose": "classifying",
      "id": "http://dbpedia.org/resource/Theo_van_Gogh_(art_dealer)"
    }
  ],
  "motivation": "classifying",
  "creator": "marijn",
  "type": "Annotation",
  "target": [
    {
      "type": "Text",
      "source": "urn:vangogh:let001",
      "selector": {
        "conformsTo": "http://tools.ietf.org/rfc/rfc3870",
        "value": "urn:vangogh:let001.receiver",
        "type": "FragmentSelector"
      }
    }
  ],
  "id": "urn:uuid:8f62be7d-de56-464b-8d9a-5fb0b69fc00b"
}

Discussion points:

  • Marijn: the reason to use a fragment selector is that the annotation target makes explicit that urn:vangogh:let001.receiver is part of urn:vangogh:let001. For retrieving annotations by resource, it's now trivial to ask the annotation server to return all annotations related to the letter urn:vangogh:let001. Of course, this doesn't solve retrieval of annotations for resource hierarchies with more than two levels (e.g. van Gogh letter collection > van Gogh letter > receiver, how to get all annotations for all letters in the collection from the annotation server, assuming the annotation server has no knowledge of the resource hierarchy).
  • The value of the conformsTo property is taken from a fixed list proposed by W3C. No sure this is the right choice.
  • Perhaps it makes sense here to include information from the Van Gogh ontology here, to indicate that the selector targets a Receiver within a Letter. [The rdfa in the html doc describes the relation between correspondence and letter (using terms from the ontology). So that (the rdfa) is where the annotation client/server can retreive the info about the letter being part of a collection. But does the server know where the rdfa of a certain annotated resource resides? Perhaps it doesn't. The rdfa might even be gone? So perhaps the server should copy that hierarchy information PB]

Text selectors

Instead of selecting a fragment of the resource as target, a user can also select a text fragment as the target. There are two options for text selectors:

  1. text position selector: represents the target through the start and end offsets of the selected text in the resource, e.g. start at character offset 76, end at character offset 84.
  2. text quote selector: represents the target through the string of selected text, in combination with short strings before and after it to provide context.

In the two examples below, the same text selection September 29 is targeted, first with a TextPositionSelector, then with a TextQuoteSelector:

Text position selector

{
  "@context": "http://www.w3.org/ns/anno.jsonld",
  "created": 1483959766,
  "body": [
    {
      "vocabulary": "DBpedia",
      "value": "September 29",
      "purpose": "classifying",
      "id": "http://dbpedia.org/resource/September_29"
    }
  ],
  "motivation": "classifying",
  "creator": "marijn",
  "type": "Annotation",
  "target": [
    {
      "type": "Text",
      "source": "urn:vangogh:let001",
      "selector": {
         "start": 164,
         "type": "TextPositionSelector",
         "end": 176
      }
    }
  ],
  "id": "urn:uuid:3a006d50-15c9-4647-b325-77e6bbb24ad5"
}

Text quote selector

{
  "@context": "http://www.w3.org/ns/anno.jsonld",
  "created": 1484230981,
  "body": [
    {
      "vocabulary": "DBpedia",
      "value": "September 29",
      "purpose": "classifying",
      "id": "http://dbpedia.org/resource/September_29"
    }
  ],
  "motivation": "classifying",
  "creator": "marijn",
  "type": "Annotation",
  "target": [
    {
      "type": "Text",
      "source": "urn:vangogh:let001",
      "selector": {
        "prefix": ". The Hague, Sunday, ",
        "suffix": " 1872\n\nSource status:",
        "type": "TextQuoteSelector",
        "exact": "29 September"
      }
    }
  ],
  "id": "urn:uuid:9e808982-33ad-45b4-859f-e385352ca6cd"
}

Discussion points:

  • Marijn: There are pros and cons with each selector type.
    • The position selector is more precise and robust against the problem of identifying the right target when there are repeated text fragments (typical in e.g. poems and lyrics and common phrases in long texts). But if the same resource is displayed in a different viewer with different choices of what to enrich (e.g. inclusion of IgnorableElement type and SelectWholeElement property), the offsets can be different, so the position selector targets a different piece of text. It also makes the annotation itself harder to interpret, since the textual target is not included.
    • The quote selector has the reverse pros and cons. It makes the annotation easier to interpret and is robust against changes in RDFa enrichment choices, but can't resolve target ambiguity when the quote and its context occurs multiple times in the target.
  • However, it is possible to combine the two text selectors, see below.

Combining fragment and text selectors

It is also possible that the target is a selected text fragment that is part of sub-resource of the whole or top-level resource, e.g. a passage (text selection) in a paragraph (sub-resource) in a letter (top-level resource). [But the paragraph is not really a fragment selector. It's a urn in its own right. PB]

In that case selector refinement is used. In the above case, a FragmentSelector is refinedBy a TextPositionSelector and/or a TextQuoteSelector. The annotation below targets the passage droppeltjes within the 4th paragraph of the letter:

{
  "@context": "http://www.w3.org/ns/anno.jsonld",
  "created": 1484220285,
  "body": [
    {
      "vocabulary": "DBpedia",
      "value": "Precipitation (meteorology)",
      "purpose": "classifying",
      "id": "http://dbpedia.org/resource/Precipitation_(meteorology)"
    }
  ],
  "motivation": "classifying",
  "creator": "marijn",
  "type": "Annotation",
  "target": [
    {
      "type": "Text",
      "source": "urn:vangogh:let001",
      "selector": {
        "conformsTo": "http://tools.ietf.org/rfc/rfc3870",
        "value": "urn:vangogh:let001:p.4",
        "type": "FragmentSelector",
        "refinedBy": [
          {
            "prefix": "had, en tusschen de ",
            "suffix": " door toch nog al ee",
            "type": "TextQuoteSelector",
            "exact": "droppeltjes"
          }
        ]
      }
    }
  ],
  "id": "urn:uuid:46c8fd5f-d038-497d-a7a3-1814b8d65201"
}

Discussion points:

  • Marijn: the W3C model seems to allow at most one selector and an additional level of refinement (so max 3 levels: source, selector.value and refinedBy). The nice thing here is that it allows specifying the top level resource (urn:vangogh:let001), the most specific targeted part that contains the selected text fragment (urn:vangogh:let001:p.4) and the text selection (droppeltjes). This allows search on all three parts to retrieve relevant annotations.

Improving robustness: combining text position and text quote selection

When using refinedBy in the W3C model, it is possible to use multiple refinements to indicate the same target. That is, all refinement selectors should represent the same target.

The annotation below combines TextPositionSelector and TextQuoteSelector representing the same passage droppeltjes:

{
  "@context": "http://www.w3.org/ns/anno.jsonld",
  "created": 1484220285,
  "body": [
    {
      "vocabulary": "DBpedia",
      "value": "Precipitation (meteorology)",
      "purpose": "classifying",
      "id": "http://dbpedia.org/resource/Precipitation_(meteorology)"
    }
  ],
  "motivation": "classifying",
  "creator": "marijn",
  "type": "Annotation",
  "target": [
    {
      "type": "Text",
      "source": "urn:vangogh:let001",
      "selector": {
        "conformsTo": "http://tools.ietf.org/rfc/rfc3870",
        "value": "urn:vangogh:let001:p.4",
        "type": "FragmentSelector",
        "refinedBy": [
          {
            "prefix": "had, en tusschen de ",
            "suffix": " door toch nog al ee",
            "type": "TextQuoteSelector",
            "exact": "droppeltjes"
          },
          {
            "start": 54,
            "type": "TextPositionSelector",
            "end": 65
          }
        ]
      }
    }
  ],
  "id": "urn:uuid:46c8fd5f-d038-497d-a7a3-1814b8d65201"
}

Discussion points:

  • Marijn: This allows bootstrapping the alignment between information about the target stored in the annotation and the target in the resource itself. E.g. with a combination of text position and text quote it is possible to identify changes in how the resource is displayed. If the text in the displayed resource corresponding to the position selector no longer corresponds to the text in the quote selector, the client knows that RDFa structure has changed and can look for the quoted text nearest to the stored positions. [This seems the best solution. And then warn the user about the annotations possibly being messed up. You could even provide functionality to give a warning to someone changing the source that a certain change would invalidate some annotations and ask what should be done. PB]

Annotating Audiovisual Resources

The same data model can be used to annotate audiovisual resources, as long as they are accompanied by RDFa information, in e.g. a <div> element that surrounds the video or the time bar in which the start and end selectors are displayed.

{
  "@context": "http://www.w3.org/ns/anno.jsonld",
  "created": 1484229754,
  "body": [
    {
      "vocabulary": "GTAA",
      "value": "Waterbronnen",
      "purpose": "classifying",
      "id": "http://data.beeldengeluid.nl/gtaa/28667"
    }
  ],
  "motivation": "classifying",
  "creator": "marijn",
  "type": "Annotation",
  "target": [
    {
      "type": "Video",
      "source": "http://player.vimeo.com/video/110756897",
      "selector": {
        "type": "FragmentSelector",
        "conformsTo": "http://www.w3.org/TR/media-frags/",
        "value": "#t=149.391,188.939"
      }
    }
  ],
  "id": "urn:uuid:e9d01a37-9dca-4f96-bc6a-5c44012d6e9e"
}

Additional properties

  • annotation context: The W3C model allows specifying agents involved in the annotation process:
    • annotation client: e.g. client name and version, github repository, [BB: This property is provided by the client?]
    • annotation server: e.g. server name and version, API location, [BB: This property is provided by the annotation server?]
    • location of resource viewer: i.e. URL of web site that displays the resource and embeds the annotation client.
  • Layer/Tier support: for linguistic analysis and analysis of time-based media (e.g. video, audio), annotation tools offer annotations in tiers. [If annotation purpose / motivaton cannot be used to create meaningful groupings from a user perspective (to look at later, move to bibliography, use for term paper) the layer facility could be used to group the annotations in sets. PB] Linguistic annotation is often in the form of different layers for part-of-speech tags, syllables, phonemes, phrases or named entities. This ties individual annotations together in an ordered list (e.g. the part-of-speech tags of all the words in a text, or the shot boundaries of a film). The W3C model supports the collection of annotations as an ordered list. The order property enforces non-overlapping annotations, which fits with the typical understanding of tiered annotations in tools like ELAN, Atlas.ti and NVivo.

Extensions to the W3C model

The W3C model allows for extensions to the basic framework. We could consider adding (or at least leaving room for) properties capturing information about:

  • Ontology/vocabulary: controlled vocabularies used in the RDFa enriched resource (e.g. van Gogh letters ontology). This could in principle be used by the annotation server to reason about relationships between stored annotations and the resources they target.
  • Research context: e.g. research project, questions, methodology, intention of use. This allows further interpretation of the annotations and why certain choices were made by the annotator.
  • Content: content aspects can be represented using the RDF Content Representation model.