Merge pull request #370 from lookit/spam-updates

Spam updates

docs/source/community-study-approval-process.rst

@@ -20,44 +20,84 @@ benefits all of us to uphold high standards for all Lookit studies.
 Families who have a good experience - the study is fun, it runs
 smoothly, the instructions are easy to follow, they get a gift card
 promptly or get quick friendly answers to their questions, their child
-isn’t upset by the stimuli - will come back. Positive media coverage
+isn't upset by the stimuli - will come back. Positive media coverage
 likewise benefits all of us.
 
-To make sure all studies meet community standards, we require two forms
-of review before studies are posted: at least one other researcher
-outside your lab will provide peer feedback, and Lookit staff will
-review as well. While this may feel like an “extra step” compared with
-in-lab research, it’s an opportunity to improve both the participant
-experience and the science - saving you time later. So far, researchers who were frustrated by the extra steps have been convinced once they see how much their studies improve in the process.
-
-What about external studies and 'live' sessions?
-================================================
-
-From the beginning of Lookit through most of 2021, all studies listed on Lookit were also run within Lookit, using the Lookit experiment runner. In contrast, the new “External Study'' type recruits participants via Lookit but the experiment itself takes place elsewhere. This opens up the possibility for researchers to run “asynchronous” studies with any desired platform (e.g., Qualtrics) or “synchronous” studies (e.g., scheduled Zoom sessions).
-
-Study review for *asynchronous* studies that take place outside of Lookit are subject to the same review process as Lookit-internal studies. Again, this is because we all share a reputation, and positive or negative experiences that affect a family in one study affect all of us. When you ask for peer review, you must have the 'Lookit part' of your study created and set up exactly as your participants would experience it; researchers who act as reviewers for it should start at Lookit and run through the entire study experience exactly as a participant would.
-
-Study review for *synchronous* (video) studies are also subject to peer and internal Lookit review. Right now, much of our peer review documentation assumes that studies are asynchronous, but you should expect to demonstrate your entire procedure (from scheduling all the way to debriefing) to at least one other researcher during this process. Researchers who act as reviewers should start from a Lookit study link, sign up for a session and communicate with the experimenter exactly as a participant would, and attend & record a video chat session including all warmup, consent, and debrief that a participant would experience.
-
-
-If you don’t have IRB approval or an access agreement yet
+To make sure all studies meet community standards, we use two forms
+of review before studies are posted: another researcher
+outside your lab providing feedback ("peer review"), and Lookit staff reviewing
+studies before approving them for posting ("admin review"). While this may feel 
+like "extra steps" compared with
+in-lab research, it's an opportunity to improve both the participant
+experience and the science - saving you time later. So far, researchers 
+who were frustrated by the extra steps have been convinced once they 
+see how much their studies improve in the process.
+
+
+What are the rules for external studies and 'live' sessions?
+=============================================================
+
+From the beginning of Lookit through most of 2021, all studies listed on Lookit 
+were also run within Lookit, using the Lookit experiment runner. In contrast, 
+the new "External Study" type recruits participants via Lookit but the 
+experiment itself takes place elsewhere. This opens up the possibility for 
+researchers to run "asynchronous" studies with any desired platform 
+(e.g., Qualtrics) or "synchronous" studies (e.g., scheduled Zoom sessions).
+
+In conjunction with the Children Helping Science/Lookit 
+merger, we are expediting reviewing for external studies. 
+Under this process, peer review is *optional* (but highly recommended) and 
+admin review will be expedited; however, all external studies must still meet 
+:ref:`basic requirements<basic_review_checklist>`.
+
+You are also strongly encouraged to review the 
+:ref:`study guidelines<self_review_checklist>` that we use for reviewing Lookit experiments,
+and are always welcome to give and solicit peer review for any study type on Slack. This is 
+probably the best way to learn about what features make studies successful on our
+platform, and we recommend that all new researchers contribute peer reviews so that
+you have seen at least one other remote study.  Graduate students, and undergraduate students
+who are conducting mentored research, are considered peers and do not need 
+to get special permission to contribute peer reviews.
+
+
+If you don't have IRB approval or an access agreement yet
 =========================================================
 
-You can still go ahead with steps 1 and 2 below so you’re ready to go as
-soon as possible! The "Lab" your study is associated with on Lookit will have to be approved to test before you'll be able to submit in Step 3, which means you'll need the
-access agreement set at that point.
+You can still go ahead with steps 1 and 2 below so you're ready to go as
+soon as possible! The "Lab" your study is associated with on Lookit will 
+have to be approved to test before you'll be able to submit in Step 3, 
+which means you'll need the access agreement set at that point.
 
 Step 1: study preparation and self-review
 ================================================
 
-First, get your study working smoothly, the way you want it to, and make any revisions from within-lab feedback. Please work through the :ref:`self-review checklist <self_review_checklist>` before requesting peer review!
+First, get your study working smoothly, the way you want it to, and make 
+any revisions from within-lab feedback. Please work through the :ref:`self-review checklist <self_review_checklist>` 
+before requesting peer review!
 
 .. _peer review:
 
 Step 2: peer review
 =====================
 
-Next, post a preview link for your study on the Lookit Slack #researchers channel to  gather some feedback from researchers *outside your group*. This is like getting a bunch of fresh eyes at a lab meeting made up of researchers who work online!
+Next, post a preview link for your study on the Lookit Slack #researchers channel to  
+gather some feedback from researchers *outside your group*. This is like getting a bunch 
+of fresh eyes at a lab meeting made up of researchers who work online! Check out
+the Slack channel to see many previous examples of how to post review requests, 
+including links to your study and to the peer review form. Notice also that whenever 
+someone posts a study review request, you will also see them volunteering to review
+other recently posted studies - trading reviews in this way is how we keep the system going!
+
+The goal of peer review is to have someone go through the study *exactly as a 
+family browsing CHS would do*.  This means that you should always direct reviewers to
+the Lookit preview link, even if your study takes place externally. For instance, 
+although much of our peer review documentation 
+assumes that studies are asynchronous, for a synchronous study you should expect to demonstrate 
+your entire procedure, from scheduling all the way to debriefing. Researchers who act as reviewers will
+start from a Lookit study link, sign up for a session and communicate with the 
+experimenter exactly as a participant would, and attend & record a video chat 
+session including all warmup, consent, and debrief that a participant would experience.
+
 
 Peer reviewers can reference the :ref:`peer review checklist <peer_review_checklist>`
 and will be looking for things like:
@@ -80,20 +120,21 @@ When your study is ready and you’ve responded to the peer feedback,
 you can :ref:`submit your study <study status>` for Lookit approval so it can go live.
 This is the point where you'll need to have your access agreement set up.
 
-When you submit the study, you will be prompted to note any non-standard elements that require specific approval per the Terms of Use (e.g., integration of
-additional about participants from another source) as well as what
+When you submit the study, you will be prompted to note any non-standard elements 
+that require specific approval per the Terms of Use (e.g., integration of
+additional information about participants from another source) as well as what
 changes you made based on peer feedback.
 
 At this point, Lookit staff will review your study, focusing
 primarily on the participant experience. This is also when any custom
-code you’re using will be reviewed for security or functionality
+code you're using will be reviewed for security or functionality
 issues.
 
-Initial internal review can generally be completed within a week.
+Initial internal review can generally be completed within 10 business days.
 Revisions may be requested before the study can be approved to run.
 To minimize the number of rounds of review needed, researchers are
 strongly encouraged to polish their studies as much as possible
-before submitting - please don’t use us as a proofreading service!
+before submitting - please don't use us as a proofreading service!
 
 .. admonition:: Outcomes of Lookit review
 
@@ -121,7 +162,8 @@ click "save" letting you know which fields will require re-approval to change.
 
 For minor changes ("we fixed a typo", "we clarified instructions", "we're stopping data
 collection for some conditions", etc.) approval is quick - you do not go back into the
-same queue as for initial submissions. We approve studies the same day (often within 1 business day, but no promises).
+same queue as for initial submissions. We approve studies the same day 
+(often within 1 business day, but no promises).
 
 If you want, you can let us know you'd like the study restarted in addition to
 reapproved. By default we just approve it and you can restart it when you're ready.

docs/source/community-study-review-basic-requirements.rst

@@ -3,19 +3,17 @@
 Basic review requirements
 ==========================
 
-In May 2023, we are beginning a reviewing experiment for external studies, in 
-conjunction with the merger of the Children Helping Science and Lookit platforms. 
-
-During this period, external studies will be expedited for posting, but must 
-still meet some basic requirements in 
+To accomodate the growing research community and meet the needs of different
+kinds of studies, external studies (Bring-your-own Study Link, Bring-your-own Meeting)
+will be expedited for posting, but must still meet some basic requirements in 
 order to be approved. If you have questions about any of these
-requirements, please ask them on Slack or email Melissa at mekline@mit.edu. 
+requirements, please ask them on Slack. 
 
 In addition to the requirements listed below, you are also strongly encouraged
 to review the :ref:`study guidelines<self_review_checklist>` 
 that we use for reviewing Lookit experiments. You are also welcome to give and 
 solicit peer review for your studies on Slack, 
-and we recommend that all studies consider doing this!
+and we recommend that all studies do this!
 
 Study form
 -----------
@@ -24,12 +22,28 @@ You must provide complete information for each of the fields on the study form,
 including compensation information and eligibility.  :ref:`Here is a detailed
 guide<study fields>` to each of the fields on the study form. 
 
+Paying participants
+--------------------
+
 .. admonition:: Note
 
    You are not required to pay participants - some studies ask for volunteer 
    participants - but you must clearly state what compensation is offered, including
    any limitations on who can receive compensation.
 
+Beginning December 2023, studies are required to implement a visual check that study participants are 
+acting in good faith (i.e. that a child of the correct age is present and the family is 
+attempting to participate in the study as designed) prior to compensating them. 
+
+This is being required because we have experienced spam participants (e.g. people making 
+many accounts and trying to get paid over and over again.) Our participant pool is 
+extremely high quality, and *keeping* it this way by keeping bad-faith participants out of your datasets is a shared 
+responsibility. Studies that don't screen their participants teach the scammers 
+that they can get paid by lying about their information on CHS!
+
+To learn how to implement this requirement for different kinds of studies, and for more 
+information about how to check for red flags (and green flags) to distinguish
+spammers from honestly confused families, please see the page on :ref:`spam prevention<spam_prevention>`. 
 
 Terms of use
 ------------

docs/source/index.rst

@@ -80,6 +80,7 @@ Families may take part in studies from multiple labs over time. Having one centr
    researchers-manage-consent
    researchers-experiment-data
    researchers-day-to-day-study-operation.md
+   researchers-spam-prevention.md
    community-participant-recruitment.md
    researchers-use-api
 

docs/source/researchers-day-to-day-study-operation.rst

@@ -91,20 +91,36 @@ the collection of email addresses for other compliance reasons, you can add
 an additional survey page to your study to ask for the participant's email,
 as long as it is clear that that information will only be used to send payment. 
 
-Participant compensation should never depend on the child’s behavior -
+Participant compensation should never depend on the child's behavior -
 *even if the child fusses out and/or the data is unusable*. In general,
 this means we try to pay anyone who submits a valid consent video.
-Payment for the parent’s/child’s time is ethical; accidentally paying
+Payment for the parent's/child's time is ethical; accidentally paying
 the parent to take extraordinary measures to get their child to sit
-through the study because they think that’s necessary to get paid is not
-:)
-
-The exception to the above guideline is that if someone tries the study
+through the study because they think that's necessary to get paid is not.
+The exception to this guideline is that if someone tries the study
 but only gets a very short way in (maybe only consent), then comes back
 and does the study again, you would generally pay them just once - even
 if you might have erred on the side of caution and paid them even for
 the first attempt.
 
+.. admonition:: Spam users and fraud on CHS
+
+  Very rarely, you may have people take your study who are not operating in
+  good faith - the most egregious example is someone attempting to participate
+  without a child present, but we have also seen cases where the same person 
+  makes several accounts to get paid multiple times, or where people lie about
+  their location or child's age to make themselves 'eligible' for studies. 
+
+  **Every member of your research team** who is involved in compensating families
+  for participation must be aware of this issue and the steps that researchers
+  are expected to take to avoid this.  Because CHS is a shared platform 
+  with a shared reputation and shared participant pool, it is **everyone's** 
+  responsibility to avoid paying the small number of people to try to take
+  advantage of this resouce to the detriment of our research goals. 
+
+  See :ref:`this page <spam_prevention>` for information on procedures
+  related to discouraging scammer participants on Children Helping Science. 
+
 You are free to put limits on how many times / how often people can
 participate and be compensated, and to require that the child be, say,
 in the age range for the study in order to participate. Basically, stuff
@@ -115,18 +131,18 @@ min/max used for automatic warnings; see `the
 docs <https://lookit.readthedocs.io/en/develop/researchers-using-platform.html#creating-a-study>`__.)
 If a parent participates with a child well outside the age range, you
 might want to email them to thank them for participating, let them know
-it’s fine to check out the study and you hope they found it interesting
-but since this is for x-month-olds you won’t be able to use their data
+it's fine to check out the study and you hope they found it interesting
+but since this is for x-month-olds you won't be able to use their data
 or provide compensation.
 
 Rarely, adults without children may check out a study and even make a
 consent recording. We tell our students not to do this but you never
 know :) To avoid feeling obligated to pay them (which would probably be
-surprising to them too) you’re welcome to state in your compensation
-info that the child needs to be visible in the consent video. (You don’t
+surprising to them too) you're welcome to state in your compensation
+info that the child needs to be visible in the consent video. (You don't
 actually have to enforce that for people who get the kid later, which is
 reasonable - but this way if someone ONLY submits the consent video and
-doesn’t have a child present, you don’t have to pay them.)
+doesn't have a child present, you don't have to pay them.)
 
 Parents who ran into a technical problem and want to try again
 ----------------------------------------------------------------

docs/source/researchers-experiment-data.rst

@@ -1,3 +1,5 @@
+.. _researchers_experiment_data:
+
 Downloading data
 ==============================
 

docs/source/researchers-manage-consent.rst

@@ -39,6 +39,18 @@ Consent rulings can be changed after an initial ruling is made; for instance, yo
 
 The most recent consent ruling, the time of that ruling, any comment, and the name of the researcher who made the ruling, will be included in the JSON/CSV data for this response.
 
+-----------------------------------
+Consenting and 'fake participants'
+-----------------------------------
+
+Occasionally, people make a CHS account (or multiple accounts!!) in order to try 
+and collect payments without participating in good faith. In addition to the 
+video itself, the account information shown below the video is designed to be 
+helpful in evaluating and reporting potential fake participants.
+
+For information on dealing with this, please see our page on :ref:`spam prevention<spam_prevention>`. 
+
+
 --------------------
 Response statistics
 --------------------

docs/source/researchers-set-study-fields.rst

@@ -88,7 +88,16 @@ Explain the purpose of your study here (1-3 sentences). This should address what
 =============================
 Compensation
 =============================
-Provide a description of any compensation for participation, including when and how participants will receive it and any limitations or eligibility criteria (e.g., only one gift card per participant, being in age range for study, child being visible in consent video). Please see the `Terms of Use <https://lookit.mit.edu/termsofuse/>`_ for details on allowable compensation and restrictions. If this field is left blank (which is okay if you're not providing compensation beyond the joy of participation) it will not be displayed to participants.
+Provide a description of any compensation for participation, including when and how 
+participants will receive it and any limitations or eligibility criteria (e.g., only one 
+gift card per participant, being in age range for study, child being visible in consent 
+video). Please see the `Terms of Use <https://lookit.mit.edu/termsofuse/>`_ for details 
+on allowable compensation and restrictions. If this field is left blank (which is okay 
+if you're not providing compensation beyond the joy of participation) it will not be 
+displayed to participants.
+
+Studies which compensate participants are responsible for following our :ref:`spam prevention<spam_prevention>`
+policies.
 
 =============================
 Exit URL