Amazon Mechanical Turk

Running speech experiments on AMT

On line marketplace | artificial artificial intelligence | data annotation

To ensure quality

• Make instructions clear and concise
  
• Insert positive and negative controls to screen out bad turkers
  
• Include a small amount of gold standard data (also, weight the turkers)
  
• Sometimes there is not a postive correlation between the amount of payment and the quality of the work
  
• Design a test set and choose those who meet the standard
  
• Block the bad turkers
  

Tutorial:
Beginners-Guide
Getting Started with MTurk to Gather Training Data for AI and ML

Ref:
Chris Callison-Burch and Mark Dredze.2010.Creating Speech and Language Data With Amazon’s Mechanical Turk CSLDAMT