Annotation protocol for validation task

[edufonseca]

Currently, one task is available in the FSD: https://datasets.freesound.org/fsd/contribute/validate_annotations/?help=1

Considering the less controlled scenario that is crowdsourcing, the annotation protocol should be improved before the platform launch. This is a huge task, that could be split into smaller ones. But for now it is defined as one in order to show the big picture and concentrate the discussion.

We assume that the user is familiar with the platform, and has chosen to contribute in this dataset (FSD), for which he/she has clicked on the corresponding button (currently: Validate Category Annotations). The way we see the protocol is:

1. User is presented with the task instructions or guidelines, either in text or video format (Improve guidelines for validation task #28), including, at least:

• Task definition
• Protocol description, using the platform

2. User chooses a sound category (either from our priority list or according to his interests) (Split AudioSet ontology into groups for crowdsourcing validations #29).

3. Training phase. This would show context of the selected category AND those of the siblings/parents in order to help the rater create his judgement before proceeding with the task. This phase may include:

• Explicit part:
  • Showing categories' hierarchy, direct children, descriptions, at once, for inspection
  • Showing also good sound examples for every category (Examples for training phase of validation task #30)
  • This is to be done for the selected category, its siblings and parent(s)
• Hidden part (of which the rater is not aware), presented within the Validation phase, consisting of one page of sounds with:
  • Good examples (clearly belonging to the category, to train the subject) (Examples for training phase of validation task #30), and bad examples (probably randomly selected)
  • Use the examples as Quality Control (Quality control #31 )

4. Validation phase. Where the actual task takes place.

• Quality control should be implemented (Quality control #31 ) to make sure gathered annotations are consistent
• We should have a priority system for the presentation of the annotations to be validated within every category (Add strategy for prioritizing annotations to be voted  #23 )
• Fix number of annotations/category (ie, a number of pages) per rater. Could be a fixed number or fixed range. When it is done, propose options:
  • continue with another category from the list where the initial one was chosen
  • go to beginning

[xavierfav]

The thing left about this issue is there #97.