Skip to content

Commit

Permalink
Issue #90: proposal for dealing with repeats
Browse files Browse the repository at this point in the history
if allow_repeats = False after a period where it is True then use the highest status associated with that worker as the exclusion criteria.
  • Loading branch information
gureckis committed Feb 9, 2017
1 parent dd89fe8 commit 3d77ccf
Showing 1 changed file with 20 additions and 7 deletions.
27 changes: 20 additions & 7 deletions psiturk/experiment.py
Original file line number Diff line number Diff line change
Expand Up @@ -186,13 +186,26 @@ def check_worker_status():
else:
worker_id = request.args['workerId']
assignment_id = request.args['assignmentId']
try:
part = Participant.query.\
filter(Participant.workerid == worker_id).\
filter(Participant.assignmentid == assignment_id).one()
status = part.status
except exc.SQLAlchemyError:
status = NOT_ACCEPTED
allow_repeats = CONFIG.getboolean('HIT Configuration', 'allow_repeats')
if allow_repeats: # if you allow repeats focus on current worker/assignment combo
try:
part = Participant.query.\
filter(Participant.workerid == worker_id).\
filter(Participant.assignmentid == assignment_id).one()
status = part.status
except exc.SQLAlchemyError:
status = NOT_ACCEPTED
else: # if you disallow repeats search for highest status of anything by this worker
try:
matches = Participant.query.\
filter(Participant.workerid == worker_id).all()
numrecs = len(matches)
if numrecs==0: # this should be caught by exception, but just to be safe
status = NOT_ACCEPTED
else:
status = max([record.status for record in matches])
except exc.SQLAlchemyError:
status = NOT_ACCEPTED
resp = {"status" : status}
return jsonify(**resp)

Expand Down

2 comments on commit 3d77ccf

@deargle
Copy link
Collaborator

@deargle deargle commented on 3d77ccf Feb 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A problem with this approach is, if they complete just once, they can bomb the next 1,000 and they'll never be stopped because max(status) will be that completed one.

EDIT: sorry, disregard this comment. This is confusing to me, too. I'm still mulling.

@deargle
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I have the whole vision now. Your code here is a nice approach.

Please sign in to comment.