Annotation view inside Curation often does not reflect perfect agreement among annotators

[bustrofedico]

In several cases, while all annotators appear to agree 100% on the annotation, the curator still has to select the annotation manually. Not sure why.

[bustrofedico]

There are several issues in the curation part of the tool, possibly related to what reported above.

Something I am struggling with at the moment is that the Annotation pane reports tagging which differs from that of all annotators. I wonder if there is any sort of partial string matching going on. The task is character annotation in fiction and two characters are 'Rita Raco' and 'Sergeant Greg Raco'. The Annotation pane in many cases reports 'Rita Raco' when all annotators agreed on 'Sergeant Greg Raco'.

[reckart]

The curation code operates based on layer type, offsets, and feature values - as far as I remember, the actual text of the annotations is not taken into account.

It seems there are indeed some bugs. I still have data from a user lying here since a while waiting for debugging. I'll try to up to prio on this.

[reckart]

If you have not yet curated data in the document: could you please check if the agreement remains undetected if you use the "re-merge" button?

[reckart]

You might want to export the project as a backup before trying the re-merge.

[bustrofedico]

a-ha! That did help a whole lot with both issues, thanks for the suggestion, thank you so much! (I am using your tools extensively, I am likely to report more issues, as I proceed).

I admit I don't fully understand what re-merge actually does and when I should use it. I am guessing it is generally a good idea to try re-merging when I stumble into issues such as those described in this ticket.

[reckart]

Re-merge deletes all the curated data in the current document and populates it from scratch based on the annotators current results.

Normally, you would want to wait until all annotators that work on a given document have marked it as "finished" before you start curating.

But naturally you may sometimes open a document in the curation page before all annotators are done. At that point, the annotation tool will look at the annotations of all users who have finished so far and populate the curation view based on their agreement.

If at a later point more annotators finish working on the document, you may want to "re-merge" in order to re-populate the curation view based on their annotations as well. This discards any curations you may already have performed manually.

Sounds a bit confusing? It is - we have already described an improvement which would allow the curator to start working only after all annotators assigned to a given document have completed it (#135) - but it is not implemented yet.

[bustrofedico]

Thanks for the explanation, that's exactly what happened. I did start curation way before everyone was done. Re-merging is saving me a ton of time, thanks again for the tip!