Documents already curated still appear in annotators' list

[j-lnaddef]

It has already been kind of mentioned in #135, but it would be nice if normal users (annotators) could see only documents that have not been curated yet, in order to avoid redundant work.

[reckart]

A document becomes available to a curator when an annotator marks a document as finished. After that, the document is still visible to annotators, but they can no longer edit it. It is also show in a different color in the "open document" dialog. Why should it be completely hidden from annotators?

[j-lnaddef]

Hum, then I guess there is a bug as all documents marked as [CURATION_FINISHED] still appear as regular documents to other users and they can still edit them.

[reckart]

That should not happen... the users should still be able to open them, but they should not be able to create any annotations or edit annotations. Everything should be in a read-only mode.

Ah, mind that every user finishes their documents individually - there is no cross-user effect.

Are you asking for anything different from #135?

[j-lnaddef]

I am not sure what exactly is asked in #135.
Let's say there are 2 annotators and 1 curator. If annotator_1 processes a document and curator curates that document, I would expect that annotator_2 could know it. For now, annotator_2 has no idea what the state of the document is, and will process it again even though it is a waste of time as curation has already been done.
Ideal behavior would be to have an option to hide those documents completely from annotators, as in projects with hundreds or thousands of documents you want to show only relevant ones to annotators.

[reckart]

#135 asks that a curator should only be allowed to start curating a document once all annotators that are assigned to that document have marked it as "finished".

[j-lnaddef]

In that case I am asking something completely different from #135 :).
For now I locally changed de.tudarmstadt.ukp.clarin.webanno.api.dao.DocumentServiceImpl so that listAnnotableDocuments does not return documents in state CURATION_FINISHED, even though I do not like modifying the codebase too much as then it becomes hard to merge with your changes.
Do you think it will eventually be possible for a project admin to chose weather or not display those documents to annotators?

[reckart]

I'd like to discuss this a bit further because I am not sure if hiding these documents is a good solution.

My understanding is that you want to avoid that annotators work on documents that the curator is already processing or has finished processing.

The "normal" workflow should be like this:

• the project manager uses the monitoring page to assign documents to annotators. I.e. the project manager "locks" all documents that a given annotator should not work on - the annotators do not see the documents that are "locked" for them
• all annotators to whom documents are assigned annotate these documents and then mark them as finished - the annotators still see these documents, but they cannot edit them anymore
• once all annotators are finished, the curator should start reviewing - Allow curation only after all annotators to who a document was assigned marked it as complete #135 asks that curators cannot start curating before all annotators are finished and IMHO that makes sense

It seems you might have a different workflow in mind:

• all documents are available to all annotators
• one or more annotator marks a document as finished
• a curators starts reviewing this document and finishes it - then you want to hide it from the annotators so that no new annotators start working on it

If my assumption is correct, it seems to me that there are two problems with this approach:

• the curator might accidentally start curating a document before a minimal number of annotators has worked on it
• if the document is only hidden to annotators once curation has finished, annotators might still start working on it while the curation is still in progress. However, the curator cannot take into account new annotations that come in after curation has already started. The curator might have to re-merge the documents and start all over when another annotator finishes a document.

WDYT?

[j-lnaddef]

What do you mean by

However, the curator cannot take into account new annotations that come in after curation has already started.

It seemed to me that even if a curation has started, new annotations from other people will show the next time the curator opens the document in curation, aren't they?
If that is the case, the only potential issue would be if a curator has the document currently opened and a new annotation adds up for the document, it will not be taken into account. It seems to me that the case will be seldom enough that some projects can afford it.

There is a main blocking problem with the "normal" workflow imho : the fact that the project manager has to assign document to users is really handicapping. In a project wit a lot of users and a lot of documents it is not feasible. It takes way too much time and project managers will just not use it. Sometimes you do not care too much either about who annotates which document, you just want the documents annotated.

The goal here is just to have a smooth workflow with the least efforts.
If a curator accidentally starts a curation before enough annotations have been made, it is not such a big deal as he will see it and can just return to it later. If annotators keep annotating documents already curated it is a bigger deal as they are just doing work for nothing.

[reckart]

It seemed to me that even if a curation has started, new annotations from other people will show the next time the curator opens the document in curation, aren't they?

Yes, but the automatic merging only happens the first time that the curator opens a document or when the curator explicitly re-merges. And why the curator does that, all the curated annotations are discarded and replaced with the automatically merged ones, meaning that the curator has to start over from scratch.

I think in order to support our opportunistic workflow, it would be best if

• annotation of documents was blocked already when curation has started (not finished)
• starting curation should not be an implicit action when the curator opens a document in the curation panel for the first time, but rather an explicit action, i.e. after opening a document in the curation panel the curator would explicitly have to press some "start curation" button. And until that button is pressed, the curator would only be able to compare annotator's annotations to each other, but not have access to the curation panel at the top.