Add custom MTurk qualification support

[evankirkiles]

Currently, the PsiTurk custom qualification feature only allows for binary has-this-qualification-been-granted support. However, it's not hard at all to abstract the qualification logic to allow for supplying a JSON file of more advanced qualification requirements, which would allow users to provide their own MTurk-formatted qualification tests with performance thresholds directly to the HITs, like seen in this blog post.

I've been using this updated version of PsiTurk in all of my recently created HITs and it works wonderfully alongside a script I built to upload custom qualification tests to MTurk and return their ID. I've seen this issue crop up before (#458) and the code is so simple to fix it and saves so much work that I figure a PR would be beneficial for a lot of people. I can also add in format checking to make sure that the advanced_qualifications JSON file has the correct structure, but overall I definitely think this feature could only be a positive addition.

[coveralls]

Coverage decreased (-0.07%) to 59.838% when pulling ffe13ad on evankirkiles:master into 1defaf5 on NYUCCL:master.

[deargle]

Thanks for this! Format checking would be great -- what is the basic requirement anyway, a list of dicts?

[evankirkiles]

Yeah, it uses an array of dicts. For example, this qualification requirement I'm using adds the qualification test "3V8AA37DQPTARZNHJHM1A5CEB6ROW1" to the HIT. It shows up in the qualification requirements as a test workers can take, and only allows workers who get 100% on it to continue on to the HIT.

./advanced_quals.json

[
    {
        "QualificationTypeId": "3V8AA37DQPTARZNHJHM1A5CEB6ROW1",
        "Comparator": "EqualTo",
        "IntegerValues": [100]
    }
]

Now that I think about it, an incorrectly formatted Qualification ID will raise an error regardless upon trying to submit the HIT to MTurk––it might be better just to allow all the checking to be done by AWS itself rather than try to do it beforehand and potentially incorrectly. In that case, all that really needs to be done here is adding an example of this feature's usage to the demo and docs.

[deargle]

The error would be pretty obscure if a dict was passed instead of an array of dicts -- can we just check that it's an array of dicts, and let boto handle the rest? And yeah a short example for the docs would be good.

…

On Mon, Apr 12, 2021, 10:45 AM Evan Kirkiles ***@***.***> wrote: Yeah, it uses an array of dicts. For example, this qualification requirement I'm using adds the qualification test "3V8AA37DQPTARZNHJHM1A5CEB6ROW1" to the HIT. It shows up in the qualification requirements as a test workers can take, and only allows workers who get 100% on it to continue on to the HIT. [ { "QualificationTypeId": "3V8AA37DQPTARZNHJHM1A5CEB6ROW1", "Comparator": "EqualTo", "IntegerValues": [100] } ] Now that I think about it, an incorrectly formatted Qualification ID will raise an error regardless upon trying to submit the HIT to MTurk––it might be better just to allow all the checking to be done by AWS itself rather than try to do it beforehand and potentially incorrectly. In that case, all that really needs to be done here is adding an example of this feature's usage to the demo and docs. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#493 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI6Y7PKUEAS5TNCHGCFZVTTIMPRVANCNFSM42X43N4A> .

[evankirkiles]

OK, added list-of-dicts type checking and a boilerplate example in the example/ folder.

[deargle]

I'm pushing some changes to make it so that no advanced_quals path is set by default. I would push them directly to this PR, but that would be your master branch, so I pushed to this branch: https://github.com/NYUCCL/psiTurk/tree/advanced_quals can you test that and make sure it's still accepting your advanced quals? I don't want to go through setting up the unit tests to use dummy ones right now.

[evankirkiles]

Okay, merged those changes––everything still good on my end with my own advanced quals.

[deargle]

👍 thanks!