Notifications
Notifications are an underused aspect of the MTurk application. Notifications are an automated method to receive updates about a HITType while it is available to workers. This is useful for monitoring any of the following events:
- A Worker accepts a HIT
- A Worker abandons an assignment
- A Worker returns an assignment
- A Worker submits an assignment
- A HIT becomes "reviewable"
- A HIT expires
Note: Notifications are configured at the HITType level, so all HITs of a given HITType will trigger notifications once configured.
To setup a notification for a given HITType, simply create a HITType and then use GenerateNotification and SetHITTypeNotification to configure a notification for that HITType. The easiest way to receive notifications is via email. The following example shows how to receive an email any time a HIT of the HITType expires:
hittype <-
RegisterHITType(title="Example 1",
description="Example",
reward=".20",
duration=seconds(hours=1),
keywords="example")
a <- GenerateNotification("requester@example.com", event.type = "HITExpired")
SetHITTypeNotification(hit.type = hittype$HITTypeId,
notification = a,
active = TRUE)If you want to test this notification (i.e., see whether it is working as expected), it is possible to send a test notification by passing the notification object (returned by GenerateNotification) to SendTestEventNotification.
SendTestEventNotification(a, test.event.type="HITExpired")This should trigger an email from mturk-noreply@amazon.com with a subject line containing Your Event Notification to the specified account, which looks something like:
Greetings from Amazon Mechanical Turk,
You are receiving this email because you subscribed to be notified when
certain events related to your HITs or Qualifications occurred.
Specific event information is shown below:
Event Type: HITExpired
Event Time: 2014-11-18T20:01:06Z
HIT Type ID: 09876543210987654321
HIT ID: 12345678901234567890
Assignment ID: 1234567890123456789012345678901234567890
Event Type: HITExpired
Event Time: 2014-11-18T20:01:06Z
HIT Type ID: 09876543210987654321
HIT ID: 12345678901234567890
Assignment ID: 1234567890123456789012345678900987654321
Sincerely,
Amazon Mechanical Turk
https://requestersandbox.mturk.com
410 Terry Avenue North
SEATTLE, WA 98109-5210 USA
An alternative to email notifications uses the Amazon Simple Queue Service (SQS) to track notifications and make them retrievable later. While email notifications are convenient for small projects with a small number of notifications, larger projects might trigger dozens if not hundreds or thousands of notifications. At this scale, email becomes an inconvenient method of tracking notifications. SQS offers an alternative way of receiving notifications and, possibly, automatically processing those notifications.
Setting up SQS notifications is similar to email notifications. Indeed, the same functions are used, but there is a bit of additional work required to setup an SQS queue to receive those notifications. This tutorial from AWS provides some details.
Begin by visiting the SQS console. If it's your first time logging in, you'll see a screen like the following asking you to Create New Queue:
A window will popup asking you to enter a name for the queue. In this example, we'll call the queue "MTurkR". Some additional options can be specified, but are safe to ignore. This will create a new queue that you can further configure. At the bottom of the new screen, select the Permissions tab and click the Add a Permission button:
A window will popup asking you to configure this permission. You need to grant the MTurk service permission to send notifications to this queue on your behalf. Specifically, you want to setup the following options:
- Set Effect to
Allow - Set Principal to
arn:aws:iam::755651556756:user/MTurk-SQS - Set Actions to
SendMessage
Leave everything else as-is. The result should be something like the following:
Click Add Permission and confirm on the popup window by clicking Yes, Add Condition.
Your SQS queue is now created. Note that you can send all notifications for all HITTypes to the same queue, or you can setup different queues for different HITTypes. This is up to you.
It is now possible to use MTurkR to configure notifications to be sent to this queue. The process works the same as sending email notifications, except you need to find the URL for your queue which you will specify instead of an email address in GenerateNotification. You can find this at the bottom of your main queue page:
You then include this URL as the value of destination in GenerateNotification. Note that you also need to specify transport = "SQS" rather than the default transport = "Email" (as seen above). Otherwise the code is identical:
hittype <-
RegisterHITType(title="Example 1",
description="Example",
reward=".20",
duration=seconds(hours=1),
keywords="example")
b <- GenerateNotification("https://sqs.us-east-1.amazonaws.com/1234567890/MTurkR", "SQS", event.type = "HITExpired")
SetHITTypeNotification(hit.type = hittype$HITTypeId,
notification = b,
active = TRUE)You can then test this notification using SendTestEventNotification:
SendTestEventNotification(b, test.event.type="HITExpired")You will then be able to see the resulting notification in your SQS queue. You can then view notifications by clicking Queue Actions at the top of the main queue page and selecting View/Delete Messages:
You will then see a list of notifications. You can click More Details on each notification item to see the details of the notification:
The contents are similar to the details that were reported in the email notification example. Of course, viewing the notifications in this format is probably not very helpful. But, using SQS means you would have programmatic access to all notifications via the SQS API.