| Week | Topics | Async Reading | Sync Reading | Assignment Due |
|---|---|---|---|---|
| 1 | Experimentation | FE 1, NYT | Feynman, News 1, 2, Predict or Cause | – |
| 2 | Apples to Apples | FE 2; Lewis & Reiley (p. 1-2.5, §1; §2A-B) | MTGI 1,5,8,9; Lakatos (O) Rubin, sections 1 & 2 | Essay 1 |
| 3 | Quantifying Uncertainty | FE 3.0, 3.1, 3.4 | Blackwell, Lewis and Rao 1, 3.1, 3.2 | PS 1, Revised Essay 1 |
| 4 | Blocking and Clustering | FE 3.6.1, 3.6.2, 4.4, 4.5 | (O): Cluster Estimator, BlockTools | Essay 2 |
| 5 | Covariates and Regression | MM 1, FE 4.1-3, MM 2, MHE p. 16-24 | Opower (O): FE Appendix B (p. 453) | PS 2, Revised Essay 2 |
| 6 | Regression; Multi-factor Experiments | MM 6.1, MM 95-97, FE 9.3.3, 9.4 | Montgomery Sections 1, 3.0, 3.1, 3.2, 3.5, 4.2, Skim 5 | Vote on Projects |
| 7 | HTE | FE 9, Multiple Comparisons, and Demo | Goodson (O): JLR 1, 2, 3.1, 4.3, Etsy | – |
| 8 | Incomplete Control of Delivery | FE 5 | G&G 2005; TD, Ch 7; TD, Ch 9 | PS 3 |
| 9 | Spillover | FE 8 and lyft and (O) uber | Miguel and Kremer; Blake and Cohey 2, 3 | Progress Report |
| 10 | Problems, Diagnostics and the Long View | FE 11.3 | DiNardo and Pischke, Simonsohn (O): Robinson | – |
| 11 | Causality from Observation? | MM 3.1, 4.1, 5.1 | Incinerators, Glynn, Dee (O): Glassberg Sands, Lalive, Rubin, Section 3 | PS 4 |
| 12 | Attrition, Mediation, Generalizabilty | FE 7, 10, Bates 2017 | Alcott and Rogers | Peer Eval 1 |
| 13 | Creative Experiments | FE 12, (O): Ny Mag, Science, FE 13 | Broockman Irregularities, Hughes et al. (O): Uber Platform | |
| 14 | Final Thoughts | Freedman | PS 5 | |
| 15 | – | (O): Retracted LaCour, (tl;dr), Podcat (audio)) | Final Paper, Peer Eval 2 | |
This course introduces students to experimentation in data science. Particular attention is paid to the formation of causal questions, and the design and analysis of experiments to provide answers to these questions. This topic has increased considerably in importance since 1995, as researchers have learned to think creatively about how to generate data in more scientific ways, and developments in information technology has facilitated the development of better data gathering.
This course begins with a discussion of the issues with causal inference based on observational data. We recognize that many of the decisions that we care about, whether they be business related or theoretically motivated, are essentially causal in nature.
The center of the course builds out an understanding of the mechanics of estimating a causal quantity. We present two major inferential paradigms, one new and one you are likely familiar with. We first present randomization inference as a unifying, intuitive inferential paradigm. We then demonstrate how this paradigm sits in complement to the classical frequentist inferential paradigm. These concepts in hand, we turn focus to the design of experiments and place particular focus both answering the question that we set out to answer, and achieving maximally powered experiments through design.
The tail of the course pursues two parallel tracks. In the first, students form a research question that requires a causal answer and design and implement the experiment that best answers this question. At the same time, new content presented in the course focuses on the practical stumbling blocks in running an experiment and the tests to detect these stumbling blocks.
We hope that each student who completes the course will:
- Become skeptical about claims of causality. When faced with a piece of research on observational data, you should be able to tell stories that illustrate possible flaws in the conclusions.
- Understand why experimentation (generating one’s own data by doing deliberate interventions) solves the basic causal-inference problem. You should be able to describe several examples of successful experiments and what makes you feel confident about their results.
- Appreciate the difference between laboratory experiments and field experiments.
- Appreciate how information systems and websites can be designed to make experimentation easy in the modern online
- Understand how to quantify uncertainty, using confidence intervals and statistical power calculations.
- Understand why control groups and placebos are both important.
- Design, implement, and analyze your own field experiment.
- Appreciate a few examples of what can go wrong in experiments. Examples include administrative glitches that undo random assignment, inability to fully control the treatment (and failure to take this inability into account), and spillovers between subjects.
Computing is conducted primarily in R.
If you are looking to work on something over the break, between semesters, I recommend this course on `data.table`, created by the package author, and available for free at datacamp.
We use two books in this course, and read a third book in the second week. We recommend that you buy a paper copy of the two textbooks (we’ve chosen textbooks that have a fair price), and would understand if you digitally read the third book.
- Field Experiments: Design and Analysis is the core textbook for the cousre. It is available at Amazon for $40.
- Mastering Metrics is the secondary textbook for the course. It is available at Amazon for $20.
- More than Good Intentions is the third book for the cousre. It is available at Amazon for $10, new, or $3 used. But, you could also read this digitally.
- We have made all the articles we read in the couse available in the repository. However, it is a great practice to get used to establishing a VPN to gain access to all the journal articles that are available through the library subscription service. Instructions for connecting are here. Journal access is one of the greatest benefits to belonging to a University, we suggest you use it.
- David has made a great resource that has suggestions for further reading. You can access this here.
| Day | Time | Instructor | Link |
|---|---|---|---|
| Monday | 12:30-1:30p | Alex | |
| Tuesday | 5:30-6:30 | Carson | |
| Tuesday | 6:30-8:30p | Daniel | |
| Wednesday | 5:30-6:30p | Alex & Carson | |
| Thursday | 5:30-6:30 | Micah | |
| Friday | TBD. | Ross |
- Problem Sets (50%, 10% each) A series of problem sets, mostly drawn from FE, many requiring programming or analysis in R.
- We encourage you to work together on problem sets, because great learning can come out of helping each other get unstuck. We ask that each person independently prepare his or her own problem-set writeup, to demonstrate that you have thought through the ideas and calculations and can explain them on your own. This includes making sure you run any code yourself and can explain how it works. Collaboration is encouraged, but mere copying will be treated as academic dishonesty.
- At this point, the course has lived for a number of semesters, and we have shared solution sets each semester. We note in particular that struggling with the problems is a key part of the learning in this course. Copying from past solutions constitutes academic dishonesty and will be punished as such; you should know that we have included language in the solutions that will make it clear when something has been merely copied rather than understood.
- Essays (20%, 10% each)
- Class Experiment (30%) In teams of 3-5 studetns, carry out a pilot experiment that measures a causal effect of interest.