Feature Request: Peer Review (count annotation tag agreement, add comment etc)

[vikaskyadav]

Peer Review System

This is going to be a great system in the coming time. I am planning to add some kind of basic peer review system to the projects. Multiple people can review the annotations/tags made by peers and can add or delete their own annotations/tags. Also, I think basic commenting place where people can write their opinion(if needed) as why they chose some tag or marked something.

Need more pointers. Help appreciated !

[c-w]

To enable the tag review workflow with minimal changes to the project, we could mostly re-use the existing UI. We could have a new user permission like "tag reviewer" who can see an "approve/deny tag" button on the data labeling UI:

Other tag review workflows such as inter-annotator agreement may be a more involved change.

[icoxfog417]

#258 is merged. If you need more feature, please raise a new issue.

[vwoloszyn]

I've designed a tool for computing the inter-annotator agreement. It might be useful :)
https://github.com/vwoloszyn/diaa