Skip to content
Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
dailybeast_teen_suicide_experimental.pdf
dailybeast_teen_suicide_experimental.qsf
nyt_weight_loss_experimental.pdf
nyt_weight_loss_experimental.qsf
readme.txt
shared_control.pdf
shared_control.qsf

readme.txt

Survey methodology and replication instructions
-----------------------------------------------
Soham Sankaran (April 8, 2019)

In this folder are the survey materials for the experiment accompanying the article titled "At the Edge of Deception: Towards a quantitative standard of journalistic accuracy", which can be found at https://soh.am/writes/at_the_edge_of_deception/.

There were three surveys used for the experiment: shared_control, nyt_weight_loss_experimental, and dailybeast_teen_suicide_experimental. The surveys were conducted on Qualtrics (https://qualtrics.com) Each survey is associated with a pdf file and qsf file in this directory.

Each pdf file contains all the questions asked in that version of the survey in readable form. Note that in the actual survey, each question appears on a separate page -- the respondent must answer the current question and click the next arrow to get to the next question, and they are not allowed to progress if their answer is blank. All questions take the form of radio buttons, that is, the respondent must pick exactly one of the offered alternatives as their answer. In addition, note that respondents are not allowed to return to the article text (in the experimental group surveys) once they read it at the beginning of the survey.

In addition, the following questions in each survey are presented with the order of the alternatives randomized to prevent ordering effects from confounding the results:

a) If you had to pick only one, which of the following statements about losing weight do you think is more true?

b) Before you read the article, which of the following statements about losing weight did you think was more true?

c) Did reading the article change your mind?

d) What U.S. political party do you identify with?

e) Do you think the US Food and Drug Administration (FDA) should prevent the promotion of diets based on calorie counting?

f) Which of the following rates do you think is HIGHER?

g) If a choice had to be made, which of the following causes should receive MORE funding?

h) Before reading this article, which suicide rate did you think was HIGHER?

i) Did reading the article change your mind about which suicide rate was higher?

In order to replicate the experiment, you may import the qsf files provided into Qualtrics and use them to run your own.

Note that each arm of each experiment described here was conducted on Amazon Mechanical Turk over the course of a few hours on either April 10 or 11 of 2018. Only US-based participants (as qualified by Amazon) were allowed (though it is of course possible that some small percentage spoofed their location). Each participant was paid $0.50 for completing their survey. I estimated that each participant would take ~5 minutes to complete their survey based on informal piloting on some graduate students at the Human Nature Lab. I spent a total of ~$220 (entirely my own personal funds) on these experiments, including Amazon's AMT levy and a small pilot test.

These surveys include the full text of the articles involved. These articles are are copyrighted by the New York Times and The Daily Beast respectively, and are included here in good faith under the fair use exemption for research such that this experiment may be replicated easily by any who so choose.

Soham Sankaran
https://soh.am/

You can’t perform that action at this time.