Qualification Requests

This tutorial walks through how requesters can manage Qualification Requests, which emerge when a requester creates a "requestable" QualificationType and one or more workers request to be granted that Qualification.

When a requester creates a QualificationType, that QualificationType is added to the public list of Qualifications that workers can earn. All QualificationTypes are publicly visible to all workers and to other requesters. For example, the Qualifications tab on the worker site lists all Qualifications. As of December, 2014, this includes over 30,000 QualificationTypes.

(Note: These QualificationTypes can also be viewed in MTurkR using SearchQualificationsTypes(only.mine = FALSE, return.all = TRUE).)

QualificationTypes vary by whether they are "requestable" and what, if any, Qualification Test the worker must complete before they can be granted the Qualification:

1. Some are "requestable," meaning that a worker can ask to be given the Qualification without completing a Qualification Test but must wait for the request to manually grant the Qualification
2. Some are requestable but auto-granted, meaning that the worker simply needs to request the Qualification and, upon request, is automatically granted the Qualification. This happens when a QualificationType is created to automatically grant a particular score. Using CreateQualificationType, this requires setting options auto = TRUE and specifically a value for the auto.value option.
3. Some are requestable but involve a Qualification Test, meaning the worker can complete a short Test to qualify but must await manual approval from the requester before earning the Qualification.
4. Some are requestable and involve an automatically scored Qualification Test. In these cases, the Qualification Test appears the same as in (3), but the requester has supplied an AnswerKey to automatically score the worker's test answers and, conditionally, grant a particular Qualification Score. Because they are automatically scored, this setup is very helpful for pre-screening workers before allowing them to complete a HIT
5. Finally, some Qualifications are not requestable, meaning they will show up in the Qualifications tab of the worker site but cannot be requested.

Any time a requester creates a requestable QualificationType, workers can initiate Qualification Requests to the requester. These requests are not visible in the Requester User Interface, but will accumulate over time, particularly if a given QualificationType is attached to a currently available HITType. The remainder of the tutorial shows how to handle Qualification Requests (for scenarios 1 and 3, above) using MTurkR.

Note: An important thing to remember about Qualification Requests is that they operate independently of your actions as a request to assign, update, or revoke Qualifications manually. As Amazon's MTurk documentation notes:

AssignQualification does not affect any pending Qualification requests for the Qualification by the Worker. If you assign a Qualification to a Worker, then later grant a Qualification request made by the Worker, the granting of the request may modify the Qualification score. To resolve a pending Qualification request without affecting the Qualification the Worker already has, reject the request with the RejectQualificationRequest operation.

Qualification Requests using MTurkR

Let's start by creating a QualificationType that is requestable.

qual <- CreateQualificationType(name="A QualificationType Demonstration",
           description="This qualification is for demonstration purposes",
           status = "Active")

This will return a dataframe as object qual that contains details about the newly created QualificationType. If we visit the MTurk worker site and search for "A QualificationType Demonstration", this Qualification will show up as requestable (by default):

The QualificationType shows us with the Title and Description fields we supplied above. (If any of the features of the QualificationType were incorrect, we can use UpdateQualificationType to change, e.g., the title or description.)

As we can see, no users have this Qualification. The details for the Qualification also say:

Retake Delay: This Qualification cannot be retaken once requested.

This means that a worker can only request the Qualification one time. If we set a period of time (in seconds) for the optional retry.delay option in CreateQualificationType, workers will be able to re-request a Qualification after the specified amount of time has passed. This may be appropriate if the QualificationType includes a Qualification Test and you want to allow workers more than one chance to pass the test.

With the QualificationType created and visible, we can then use GetQualificationRequests to retrieve any Qualification Requests that workers have initiated for this Qualification:

GetQualificationRequests(qual$QualificationTypeId)
## 0 Requests Retrieved
## [1] QualificationRequestId QualificationTypeId    SubjectId              SubmitTime             Answer                
## <0 rows> (or 0-length row.names)

As we can see in this example, there are no requests. Note: If we call GetQualificationRequests with no arguments, it will return requests for all QualificationTypes instead of just for this QualificationType.

Back on the worker site, a worker can initiate a Qualification Request by clicking Request this Qualification. Because the QualificationType does not include a Test, workers will see a page like the following:

By clicking Continue, the worker's request will be submitted and they will be taken back to the HIT tab and shown a confirmation message:

This means we can go back to R and see if we've received the Qualification Request:

requests <- GetQualificationRequests(qual$QualificationTypeId)
## 1 Requests Retrieved
requests
##           QualificationRequestId            QualificationTypeId      SubjectId           SubmitTime Answer
## 2 3NDIBRFPIG65K4ZB2M4YXZ497CZETM 3386SE2C9GAP5V0TW3YRS9PNK8NVP3 A1RO9UJNWXMU65 2014-12-06T14:17:59Z   <NA>

The output from MTurkR shows that QualificationRequestId, which is the identifier for this worker's request of this particular QualificationType. The details also include the QualificationTypeId for the Qualification, the worker's WorkerId (listed as SubjectId due to an inconsistency in Amazon's language), and the time the request was submitted. (Note: If the QualificationType involved a test, details of the worker's test answers would be included as well.)

We then have two options: we can either grant the request or reject the request. Both are simple.

Granting a Request

To grant the request, we simply pass the QualificationRequestId to GrantQualification along with a values option specifying what value the worker should receive for the Qualification:

GrantQualification(requests$QualificationRequestId[1], values = 100)
## 1: Qualification (3NDIBRFPIG65K4ZB2M4YXZ497CZETM) Granted
##           QualificationRequestId Value Valid
## 1 3NDIBRFPIG65K4ZB2M4YXZ497CZETM   100  TRUE

The output from MTurkR will confirm with the operation succeeded. The worker will then receive an email automatically indicating that their request has been granted. It will look something like:

Greetings from Amazon Mechanical Turk,

You have been granted the following Qualification:
 - A QualificationType Demonstration, with a score of "100"

Qualifications you earn allow you to complete new kinds of HITs.  Each Qualification has a score associated with it.  You can complete HITs for which you have the required Qualifications and a sufficiently high score.

To begin working on HITs requiring the Qualifications you possess, click on the link below:
https://www.mturk.com/mturk/findhits?match=true

To view all of your Amazon Mechanical Turk Qualifications, please visit the page below:
https://www.mturk.com/mturk/findquals?earned=true

Sincerely,
Amazon Mechanical Turk
https://www.mturk.com
410 Terry Avenue North
SEATTLE, WA 98109-5210 USA

And back on the worker site, the worker can click on the Qualifications tab and click Qualifications Assigned to You to see that they have been granted the Qualification:

If you need to change the worker's score on the Qualification, use UpdateQualificationScore:

UpdateQualificationScore(requests$QualificationTypeId[1], requests$SubjectId[1], 50)
## 1: Qualification Score for Worker A1RO9UJNWXMU65 updated to 50
## 1 Qualification Scores Updated
##              QualificationTypeId       WorkerId Value Valid
## 1 3386SE2C9GAP5V0TW3YRS9PNK8NVP3 A1RO9UJNWXMU65    50  TRUE

To revoke a worker's Qualification, simply use RevokeQualification:

RevokeQualification(requests$QualificationTypeId[1], requests$SubjectId[1])
##         WorkerId            QualificationTypeId Reason Valid
## 1 A1RO9UJNWXMU65 3386SE2C9GAP5V0TW3YRS9PNK8NVP3         TRUE

The worker will, again, receive an email describing the revocation:

Greetings from Amazon Mechanical Turk,

One of your Qualifications has been revoked by the Requester who maintains the Qualification.  Please see details about the Qualification below.

Qualification Revoked: A QualificationType Demonstration
Requester: Thomas J. Leeper
Grant Date: December 6, 2014
Revoke Date: December 6, 2014
Revoke Reason: Requester did not give a reason

You may request the Qualification again on or after December 6, 2014.  For more information about Qualifications, please see the FAQ:

https://www.mturk.com/mturk/help?helpPage=main

Sincerely,
Amazon Mechanical Turk
https://www.mturk.com
410 Terry Avenue North
SEATTLE, WA 98109-5210 USA

Rejecting a Request

To reject the request, we simply use RejectQualification:

RejectQualification(requests$QualificationRequestId[1])
## 1: Qualification (3HI1N53U00I246G92V78KKP071K6KN) Rejected
##           QualificationRequestId Reason Valid
## 1 3HI1N53U00I246G92V78KKP071K6KN   <NA>  TRUE

When rejecting a request, we can optionally specify a reason option, which is a message (max 1024 characters) that will be passed on to the worker explaining the rejection. As when granting a Qualification request, the worker will receive an email indicating that their request has been rejected that looks something like:

Greetings from Amazon Mechanical Turk,

One of your Qualification requests has been rejected by the Requester who maintains the Qualification.  Please see details about the Qualification request below.

Qualification Rejected: A QualificationType Demonstration
Requester: Thomas J. Leeper
Request Date: December 6, 2014
Rejection Date: December 6, 2014
Rejection Reason:

You may request the Qualification again on or after December 6, 2014.  For more information about Qualifications, please see the FAQ:

https://www.mturk.com/mturk/help?helpPage=main

Sincerely,
Amazon Mechanical Turk
https://www.mturk.com
410 Terry Avenue North
SEATTLE, WA 98109-5210 USA

If the QualificationType was configured with a retry delay, the worker will be able to re-request the Qualification after the specified period of time has passed.

If for some reason you accidentally reject a Qualification request, it is not possible to use GrantQualification to correct the mistake. Instead you need to use AssignQualification (using the QualificationTypeId for the QualificationType) to assign a particular score to the worker:

AssignQualification(requests$QualificationTypeId[1], requests$SubjectId[1], 100)
## Qualification (3386SE2C9GAP5V0TW3YRS9PNK8NVP3) Assigned to worker A1RO9UJNWXMU65
##         WorkerId            QualificationTypeId Value Notified Valid
## 1 A1RO9UJNWXMU65 3386SE2C9GAP5V0TW3YRS9PNK8NVP3   100    FALSE  TRUE

When assigning Qualifications in this way, the default behavior is to not email the worker. If you'd like to email the worker, simply set the notify = TRUE option in AssignQualification. The result will be an email to the worker that looks like the following:

Greetings from Amazon Mechanical Turk,

You are receiving this email because a requester has granted you the following qualification:

Qualification:        A QualificationType Demonstration
Requester:            Thomas J. Leeper
Qualification Value:  100

For more information about this qualification, please contact the requester directly.

Sincerely,
Amazon Mechanical Turk
https://www.mturk.com
410 Terry Avenue North
SEATTLE, WA 98109-5210 USA

Qualification Requests using the MTurkR Graphical Interface

To create a QualificationType in the wizard (as we did above, on the command line), simply navigate to the Qualifications tab and select Create a Qualification Type:

A window will popup allowing you to specify various options for the QualificationType. The required fields are Name, Description, and Status. Clicking OK will create the QualificationType:

To manage Qualification Requests using the MTurkR graphical wizard, simply initiate the wizard using mturkr.wizard(), navigate to the Qualifications tab, and select Get Qualification Requests:

This will open a series of windows. The most important of these is labeled Set QualificationType. This window contains a single field allowing you to enter the QualificationTypeId of the QualificationType you just created (or any other QualificationType):

If you don't know this number (and you probably wouldn't know it), you can press Search for QualificationTypes. This will open a small window allowing you to search among the QualificationTypes you've created:

You can press OK without specifying any options. This will then open yet another window that lists all of your QualificationTypes by name and QualificationTypeId:

Select the QualificationType that you want to manage requests for, and click OK. The window will close and the QualificationTypeId field in the Set QualificationType window will be populated appropriated:

By pressing OK, this window will close and you will be taken back to the Qualification Requests window. It will automatically be populated with any open requests. If there are none, you'll see something like:

On the other hand, if there are an pending requests, they'll be listed individually. To view any details of the request (for example, answers to a Qualification Test), simply select the request from the list and press View Selected. Details, if any, will be printed to the console:

Granting a Request

To grant a request, simply select it and then press Grant Selected. A small window will open allowing you to enter a score for the Qualification. Pressing OK will grant the request with the specified score:

The list of requests will then be changed to reflect that the request has been granted:

Rejecting a Request

To reject a request, simply select it and then press Reject Selected. A small window will open allowing you to enter an optional rejection reason. Pressing OK will reject the request:

The worker will be notified via email of the rejection and the list of requests will be modified to reflect that the request has been rejected:

As you can see, managing Qualification Requests will the graphical wizard is extremely simple and requires almost no programming knowledge.