Mechanical Turk Integration

[la61nn0vate]

Integration of CVAT with MTurk for deploying work as HITs would be very useful for such projects. Need to integrate Turkic framework of VATIC with CVAT.
I would also like to contribute to your project. Please help me in setting up the development environment for the same.

[nmanovic]

Hi,

It is a feature in our mind which we don't have resources to implement till EOY for sure. If you can help it will be awesome. Development environment is described here: #18.

Several notes:

• Need to implement a separate Django application for the feature
• It isn't good to use turkic from VATIC because it isn't supported anymore. I would suggest using an official API (boto3?) - http://boto3.readthedocs.io/en/latest/reference/services/mturk.html
• Need to think how split a task on many HITs (by frame/image?) because HITs should be trivial. Also need to automatically create "verification" tasks to check results. In many case it will be necessary to merge annotation from several HITs.
• GUI to publish, view progress, ... Need to have sketch of it.
• The application should have REST API.

In general it is a big feature which cannot be easily implemented.

[nmanovic]

General requirements:

• Easy interface to create/review/accept an annotation task for MTurk (separate Django app).
• Probably need to have a separate instance on AWS for such tasks.
• Automatic authorization from MTurk, special group to access only links from MTurk django app
• Email notifications about deadlines to verify annotation tasks, etc...

Requirements (image annotation):

• One image per a segment. Thus it is easy to post such job as HIT.
• Several jobs for the same segment.
• Automatically correlate annotations in different jobs for the same segment. Automatically find "trash" in annotations and discard such HIT.
• Minimum number of jobs for a segment is 3.
• Maximum number of jobs for a segment is 7. After that need to verify or discard HITs due to low quality.
• One user cannot do annotations for the same segment (several similar jobs). Need to improve auth/permissions to access only specific jobs.

[nik13]

Hi, Wanted to know if any progress has been made in this direction :)

[nmanovic]

@nik13 , we are not working in the direction at the moment.

[nmanovic]

Nice overview of using a crowd sourcing platform (Toloka): https://www.youtube.com/watch?v=yyybisXHFc0

[serwansj]

Has there been any progress on the integration with crowdsourcing platforms?

[nmanovic]

I hope to have some news in Q2-Q3 on the topic...

[eliabruni]

Very interested in this too!