James, O., Jilke, S. & G. Van Ryzin. 2017. Experiments in Public Management Research: Challenges and Contributions. Cambridge: Cambridge University Press (pages 509-511).
Below is a checklist of important questions that researchers using experiments in public administration and management should consider when reporting their experimental studies and findings. The checklist is based on the core set of issues in James, Jilke and Van Ryzin (2017) guidance that synthesizes and adapts reporting guidelines for experiments from the social and medical sciences to public administration and management.1
i. What questions was the experiment designed to address?
ii. What specific hypotheses were tested? (e.g., the causal factor estimated, the expected sign and magnitude of expected effects on outcomes, and whether expected effects are expected to be homogenous or to vary by subgroups)
i. What was the exact setting and location of data collection? (e.g., a laboratory room at a university, online survey)
ii. Recruitment date(s)? (including follow-ups)
iii. Any eligibility or exclusion criteria for participants?
iv. How were participants recruited and selected, and who did the recruiting? (e.g., recruitment by researchers, survey recruitment firm)
v. What was the response rate? (if applicable)
i. What type of experimental design was used? (e.g., parallel, factorial, conjoint, within or between subjects)
i. Is there a detailed description of treatments? (e.g., provide all materials in main text or an appendix or supporting documents.)
ii. Was a control and/or placebo group used?
iii. Was deception used in treatments?
iv. Was the experiment incentivized for participants using monetary or other rewards?
v. Which method of treatment delivery was used? (e.g., pen-and-paper, computer, smartphone, face-to-face, telephone)
vi. Were treatments implemented as planned, what (if any) were the differences from the plan?
i. Which method and software were used to generate the randomization sequence?
ii. What type of randomization was used? (e.g., simple, clustered, blocked, etc.)
iii. What was the unit of randomization? (individuals, households, organisations, districts, etc.)
iv. Were participants, those administering the manipulations, and those assessing the outcomes unaware of condition assignment (i.e., blinding)?
v. Was the randomization process followed correctly? (e.g., check for imbalance of key characteristics across treatment groups)
i. What were the outcome measures (e.g, be sure to report all outcome measures, not just selected measures)?
ii. Is a full questionnaire provided in the appendix for outcomes measured by a survey (if applicable)?
iii. How and when were outcome measures collected? (e.g., report how long afer the treatment the outcomes were assessed)
i. How was the sample size of the experiment determined? (e.g., report any a priori power calculations)
ii. Have any stopping guidelines been used for the recruitment of subjects?
i. Is there a CONSORT participant flow diagram? (This diagram sets out the sampling of experimental participants, the allocation and delivery of treatment, and any attrition; if space precludes a diagram in the published paper, these details should be provided elsewhere.)
i. Does discussion of the results state clearly which of the outcome measures and subgroup analyses (if any) were determined prior to the conduct of the experiment and which are the result of exploratory analysis?
i. Is there a report of sample means and standard deviations (or proportions) of all experimental conditions for intention-to-treat analysis (the entire collection of subjects, whether or not the treatment was successfully delivered to them) prior to any further analysis?
ii. Was there any attrition? If yes, is there a discussion of reason(s) and examination if attrition is conditioned on treatment?
iii. Any other missing data?
iv. Any weighting procedures used?
v. Is any other statistical analysis of the data explained and justified?
i. Was the experiment reviewed and approved by an institutional review bard, ethics committee or other similar structure?
ii. Was an experimental protocol for the design of the experiment and analysis of the data registered prior to the conduct of the study? If yes, where can the preregistration be accessed?
iii. Did the design or analysis of the experiment deviate from the preregistration protocol (if applicable)? If yes, in what elements and why?
iv. If a replication dataset (and/or code) is available, is it available publicly? (e.g., provide a URL)
1 Boutron, John & Togerson (2010); Schulz, Altman & Moher (2010); Gerber et al. (2014).
Boutron, I., John, P. & Togerson, D.J. 2010. 'Reporting methodological items in randomized experiments', The Annals of the American Academy of Political and Social Science, 628(1), pp. 112-31.
Gerber, A., Arceneaux, K., Boudreau, C, Dowling, C, Hillygus, S., Palfrey, T., Biggers, S. & Henry, D. 2014. 'Reporting guidelines for experimental research: A report from the Experimental Research Section Standards Committee', Journal of Experimental Political Science, 1(1), pp. 81-98.
Schulz, K.F., Altman, D.G. & Mover, D. 2010. 'CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials', British Medical Journal, 340: c332.
Last change: 01/17/2020