-
Notifications
You must be signed in to change notification settings - Fork 3
/
PSAboot.Rmd
executable file
·282 lines (207 loc) · 17.1 KB
/
PSAboot.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
---
title: "Bootstrapping for Propensity Score Analysis"
author: "Jason Bryer, Ph.D."
date: "`r Sys.Date()`"
abstract: "As the popularity of propensity score methods for estimating causal effects in observational studies increase, the choices researchers have for which methods to use has also increased. Rosenbaum (2012) suggested that there are benefits for testing the null hypothesis more than once in observational studies. With the wide availability of high power computers resampling methods such as bootstrapping (Efron, 1979) have become popular for providing more stable estimates of the sampling distribution. This paper introduces the `PSAboot` package for R that provides functions for bootstrapping propensity score methods. It deviates from traditional bootstrapping methods by allowing for different sampling specifications for treatment and control groups, mainly to ensure the ratio of treatment-to-control observations are maintained. Additionally, this framework will provide estimates using multiple methods for each bootstrap sample. Two examples are discussed: the classic National Work Demonstration and PSID (Lalonde, 1986) study and a study on tutoring effects on student grades."
output:
html_document:
toc: true
toc_depth: 3
fig_crop: no
citation_package: biblatex
number_sections: true
pkgdown:
as_is: false
fontsize: 11pt
geometry: = 2in
# Custom YAML Pandoc Variables
line-numbers: true
list-tables: true
list-figures: true
# Package indexing
vignette: >
%\VignetteIndexEntry{Bootstrapping for Propensity Score Analysiss}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
```{r setup, echo=FALSE, results='hide', message=FALSE, warning=FALSE}
require(knitr)
opts_chunk$set(fig.width = 12,
fig.height = 8,
fig.align = 'center',
out.width = '100%',
echo = TRUE,
warning = FALSE,
message = FALSE)
require(PSAboot)
```
Set the number of bootstrap samples. This should be set to at least 100 but kept small to reduce the execution time for CRAN submissions.
```{r}
boot.M <- 10
```
```{r BootSizeWarning, echo=FALSE, results='asis'}
if(boot.M < 100) {
cat(paste0('**NOTE: This vignette uses ', boot.M, ' bootstrap samples. It is ',
'generally recommended to use at least 100 bootstrap samples or more for final publication.**'))
}
```
# Introduction
The latest version of the `PSAboot` package can be downloaded from Github using the `devtools` package.
```{r install, eval=FALSE}
devtools::install_github('jbryer/PSAboot')
```
The `PSAboot` function will perform the actual bootstrapping. It has a number of parameters for to specify how the bootstrap samples are drawn.
* `Tr` - a numeric (0 for control and 1 for treatment) or logical vector of treatment indicators.
* `Y` - a numeric vector with the outcome of interest.
* `X` - a data frame of covaraites.
* `M` (default is 100) - the number of bootstrap samples to draw.
* `formu` - the formula for estimating the propensity scores in phase I. Note that the dependent variable does not need to be specified as it will be replaced when combining the `Tr` vector and `X` data frame.
* `control.ratio` (default is 5) - This specifies the sample size of control units as a ratio of treatment units. For example, with a default value of 5 and 100 treatment units, this methods will sample 500 control units for each sample, or the number of control units if smaller than 500. When the ratio of treatment-to-control units increases, the range of propensity scores (using logistic regression) shrinks. Randomly selecting a subset of control units often results in wider and better overlapping distribution of propensity scores. See the the [`PSranges`](https://github.com/jbryer/multilevelPSA) function in the [`multilevelPSA`](https://github.com/jbryer/multilevelPSA) package for more information.
* `control.sample.size` (default is 5 times the number of treatment units) - The number of control units to sample for each bootstrap sample. If specified, this overrides the `control.ratio` parameter.
* `control.replace` (default is `TRUE`) - Specify whether random sampling of control units does so with replacement.
* `treated.sample.size` (default is the number of treatment units) - The number of treatment units to sample for each bootstrap sample.
* `treated.replace` (default is `TRUE`) - Specify whether random sampling of treatment units does so with replacement.
* `methods` - A list of functions to perform a propensity score analysis for each bootstrap sample. See the methods section below.
* `parallel` (default is `TRUE`) - Whether the bootstrapping procedure should be run in parallel.
* `seed` - Seed used for the random number generator. If specified, the random seed will be set to `seed + i` where `i` is the current bootstrap sample in (1, M).
Other parameters can be passed to `methods` using the `...` parameter.
## Methods
The `methods` parameter on the `PSAboot` function specifies the different propensity score methods that will be used. Specifically, for each bootstrap sample drawn, each function will be called. This allows for a comparison of methods across all bootstrap samples. Five methods are included, they are:
* `boot.strata` - This method estimates propensity scores using logistic regression and stratifies using quintiles on the propensity scores. Effects within each strata are estimated and aggregated.
* `boot.ctree` - This method creates strata using conditional inference trees vis-a-vis the `ctree` function in the `party` package. Effects within each strata (i.e. leaf node) are estimated and aggregated.
* `boot.rpart` - This method creates strata using classification trees vis-a-vis the `rpart` function. Effects within each strata (i.e. leaf node) are estimated and aggregated.
* `boot.matching` - This method finds matched pairs using the `Match` function in the `Matching` package. A paired dependent sample t-test is used to estimate effect sizes.
* `boot.matchit` - This method finds match pairs using the `matchit` function in the `MatchIt` package. A paired dependent sample t-ttest is used to estimate effect sizes.
### Defining Custom Methods
It is possible to define a custom method. Each method is a function with, at minimum, the following six parameters:
* `Tr` - A logical or integer (0 and 1) vector with treatment indicators.
* `Y` - A numeric vector representing the outcome.
* `X` - A data frame with the covariates.
* `X.trans` - A data frame with factor levels dummy coded.
* `formu` - A formula for estimating propensity scores in phase one.
* `...` - Other parameters passed through from the `PSAboot` function.
Each method must return a `list` with three elements:
* `summary` - This must be a named numeric vector with at minimum `estimate`, `ci.min`, and `ci.max`, however other values allowed.
* `balance` - This must be a named numeric vector with one element per covariate listed in `X.trans` representing a balance statistic. It is recommended, and the implementation for the built-in methods, to use an absolute standardized effect size. As will be shown below, the summary and plotting functions will include an adjusted balance statistic (i.e. effect size) before adjustment for comparison.
* `details` - This can be an arbitrary object, typically the result of the underlying method used.
For example, the `boot.matching.1to3` function below wraps the built-in `boot.matching` method but sets the `M` parameter to 3, thereby performing 1-to-3 matching instead of the default 1-to-1 matching. This framework simplifies the process of using, and comparing, slight variations of different propensity score methods.
```{r defineCustomFunction}
boot.matching.1to3 <- function(Tr, Y, X, X.trans, formu, ...) {
return(boot.matching(Tr=Tr, Y=Y, X=X, X.trans=X.trans, formu=formu, M=3, ...))
}
```
The `PSAboot` function returns an object of class `PSAboot`. The following S3 methods are implemented: `print`, `summary`, `plot`, `boxplot`, and `matrixplot`.
-----------------------------
# Example One: National Work Demonstration and PSID
The `lalonde` (Lalonde, 1986) has become the *de defacto* teaching dataset in PSA since Dehejia and Wahba's (1999) re-examination of the National Supported Work Demonstration (NSW) and the Current Population Survey (CPS).
The `lalonde` data set is included in the `Matching` package. The contingency table shows that there are 429 control units and 185 treatment units.
```{r lalonde.load}
data(lalonde, package='Matching')
table(lalonde$treat)
```
```{r lalonde.boot, cache=FALSE}
lalonde.formu <- treat~age + I(age^2) + educ + I(educ^2) + black +
hisp + married + nodegr + re74 + I(re74^2) + re75 + I(re75^2) +
u74 + u75
boot.lalonde <- PSAboot(Tr = lalonde$treat,
Y = lalonde$re78,
X = lalonde,
formu = lalonde.formu,
M = 100,
seed = 2112)
```
The `summary` function provides numeric results for each method including the overall estimate and confidence interval using the complete sample as well as the pooled estimates and confidence intervals with percentages of the number of confidence intervals that do not span zero.
```{r lalondeSummary}
summary(boot.lalonde)
```
The `plot` function plots the estimate (mean difference) for each bootstrap sample. The default is to sort from largest to smallest estimate for each method separately. That is, rows do not correspond across methods. The `sort` parameter can be set to `none` for no sorting or the name of any `method` to sort only based upon the results of that method. In these cases the rows then correspond to matching bootstrap samples. The blue points correspond to the the estimate for each bootstrap sample and the horizontal line to the confidence interval. Confidence intervals that do not span zero are colored red. The vertical blue line and green lines correspond to the overall pooled estimate and confidence for each method, respectively.
```{r lalonde.plot}
plot(boot.lalonde)
```
The `hist` function plots a histogram of the estimates across all bootstrap samples for each method.
```{r lalonde.histogram, warning=FALSE, message=FALSE}
hist(boot.lalonde)
```
The `boxplot` function depicts the distribution of estimates for each method along with confidence intervals in green. Additionally, the overall pooled estimate and confidence interval across all bootstrap samples and methods are represented by the vertical blue and green lines, respectively.
```{r lalonde.boxplot}
boxplot(boot.lalonde)
```
The `matrixplot` summarizes the estimates across methods for each bootstrap sample. The lower half of the matrix are scatter plots where each point represents the one bootstrap sample. The red line is a Loess regression line. The main diagonal depicts the distribution of effects and the upper half provides the correlation of estimates. In the following example we see that the `ctree` and `Stratification` methods have the strongest agreement with a correlation of 0.63 whereas the `rpart` and `MatchIt` methods have the least agreement with a correlation of 0.22.
```{r lalonde.matrixplot, warning = FALSE}
matrixplot(boot.lalonde)
```
## Evaluating Balance
The strength of propensity score analysis relies on achieving good balance. Typically one would evaluate each covariate separately to ensure that sufficient balance has been achieved. We recommend Helmreich and Pruzek (2009) for a more complete discussion of visualizations to evaluate balance. Given the large number of samples and methods used, it is desirable to have a single metric to evaluate balance. Drawing on the principles of the multiple covariate balance assessment plot (Helmreich & Pruzek, 2009), the `balance` function will estimate the effect of each covariate before and after adjustment. Moreover, to provide a single metric for each sample and method, the mean standardized effect size will be used.
```{r lalonde.balance, cache=FALSE}
lalonde.bal <- balance(boot.lalonde)
lalonde.bal
```
The `plot` function provides density plots of the balance statistics across all bootstrap samples for each method. The mean overall balance is represented by the vertical black lines, the overall balance for the complete dataset is represented by the vertical blue line, and the adjusted balance is represented by the vertical red line. Although no specific guidelines are recommended in the literature, achieving a balance of less than 0.1 is desirable. In this example we see that PSA has reduce the bias in all methods although `rpart`, and to a lesser extent `MatchIt`, did not reduce the bias by as much as would be typically desired.
```{r lalonde.balance.plot}
plot(lalonde.bal)
```
The `boxplot` function provides a more nuanced depiction of the balance by separating on the distribution of balance statistics by individual covariate. In addition to the boxplot of balance statistics, the mean balance statistic is represented by the blue point and the unadjusted balance statistic by the red point. We see that the largest imbalance was in the `black`, `married`, and `re74` (real earnings in 1974) before adjustment. All of the estimates reduced the bias in these covariates although it is worth noting that the `MatchIt` method did not reduce the bias in the `black` covariate to a desirable level.
This figure, in conjunction with the density plot above, show that the relatively large mean balance for `rpart` is likely due to some outlier samples where the adjusted balance for `educ`, `re75`, and `hispan` are fairly large. It should be noted that with these exceptions, balance for each covariates are generally less than the unadjusted balance.
```{r lalonde.balance.boxplot}
boxplot(lalonde.bal)
```
-----------------------------
# Example Two: Effectiveness of Tutoring on Course Grades
This example utilizes a dataset in the `TriMatch` package that provides student data in online English courses. The original study examined the effects of tutoring on English course grades. In the original study two treatments were defined, those who used tutoring services once and those who used tutoring services two or more times during the course term. The outcome of interest was course grade (0=F, 1=D, 2=C, 3=B, 4=A). For our purposes here we will consider only one treatment group: students who used tutoring services at least once during the course.
```{r tutoring.setup}
require(TriMatch)
require(PSAboot)
data(tutoring, package='TriMatch')
tutoring$treatbool <- tutoring$treat != 'Control'
covs <- tutoring[,c('Gender', 'Ethnicity', 'Military', 'ESL', 'EdMother', 'EdFather',
'Age', 'Employment', 'Income', 'Transfer', 'GPA')]
table(tutoring$treatbool)
```
```{r tutoring.psaboot, cache=FALSE}
tutoring.boot <- PSAboot(Tr=tutoring$treatbool,
Y=tutoring$Grade,
X=covs,
seed=2112,
M=boot.M,
control.sample.size=918, control.replace=TRUE,
treated.sample.size=224, treated.replace=TRUE,
methods=c('Stratification'=boot.strata,
'ctree'=boot.ctree,
'rpart'=boot.rpart,
'Matching'=boot.matching,
'Matching-1-to-3'=boot.matching.1to3,
'MatchIt'=boot.matchit)
)
summary(tutoring.boot)
```
```{r tutoring.plot}
plot(tutoring.boot)
```
```{r tutoring.histogram, message=FALSE, warning=FALSE}
hist(tutoring.boot)
```
```{r tutoring.boxplot}
boxplot(tutoring.boot)
```
```{r tutoring.matrixplot, fig.width = 12, fig.height = 12}
matrixplot(tutoring.boot)
```
```{r tutroing.balance}
tutoring.bal <- balance(tutoring.boot)
tutoring.bal
plot(tutoring.bal)
boxplot(tutoring.bal)
```
-----------------------------
# References
Bryer, J.M. (2013). TriMatch: Propensity Score Matching of Non-Binary Treatments. R package version 0.9.1. https://github.com/jbryer/TriMatch
Helmreich, J.E., Pruzek, R.M. (2009). PSAgraphics: An R Package to Support Propensity Score Analysis. Journal of Statistical Software 29(6), 1-23. https://www.jstatsoft.org/v29/i06/.
Ho, D.E., Imai, K., King, G., & Stuart, E.A. (2011). MatchIt: Nonparametric Preprocessing for Parametric Causal Inference. Journal of Statistical Software, (42) 8, 1-28. URL https://www.jstatsoft.org/v42/i08/
Hothorn, T., Hornik, K., & Zeileis, A. (2006). Unbiased Recursive Partitioning: A Conditional Inference Framework. Journal of Computational and Graphical Statistics, 15(3), 651-674.
Lalonde, R. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review 76: 604-620.
Dehejia, R.H. and Wahba, S. (1999). Causal Effects in Nonexperimental Studies: Re-Evaluating the Evaluation of Training Programs. Journal of the American Statistical Association 94: 1053-1062.
R Core Team (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.r-project.org/.
Rosenbaum, P.R. (2012). Testing one hypothesis twice in observational studies. Biometrika, 99, 4, 763-774.
Rosenbaum, P.R., & Rubin, D.B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 1, 41-55.
Sekhon, J.S. (2011). Multivariate and Propensity Score Matching Software with Automated Balance Optimization: The Matching Package for R. Journal of Statistical Software, 42(7), 1-52. URL https://www.jstatsoft.org/v42/i07/.
Therneau, T., Atkinson, B., & Ripley, B. (2013). rpart: Recursive Partitioning. R package version 4.1-3. https://cran.r-project.org/package=rpart