Dissertation: A National Study Comparing Charter and Traditional Public Schools Using Propensity Score Analysis

Author: Jason M. Bryer <jason@bryer.org>

Division of Educational Psychology and Methodology, University at Albany.

Committee:

• Robert Pruzek, Ph.D. (co-chair)
• Bruce Dudek, Ph.D. (co-chair)
• Heidi Andrade, Ed.D.
• Kathryn Schiller, Ph.D.

Abstract

Unlike their private school counterparts, charter schools receive public funding but are relieved of some of the bureaucratic and regulatory constraints of public schools in exchange for being held accountable for student performance. Studies provide mixed results with regard to charter school performance. Charter schools are, by definition, schools of choice, and this means that observational data methods are required for comparing such schools with others. In observational data contexts, simple comparisons of two groups such as traditional public and charter schools typically ignore the inherent and systematic differences between the two groups. However, given well-designed observational studies and appropriate analysis methods, the effects of the selection bias can be reduced, if not eliminated. The result is that the usual simple comparisons of two independent groups are replaced by comparisons that make adjustments for covariate differences. This study includes development of new methods, largely graphic in form, designed for observational data to compare two groups. These methods are then used to investigate the question of whether students who attend charter schools perform differently than their traditional public school counterparts on two key academic domains: reading and mathematics. The new methods represent extensions of propensity score analysis (Rosenbaum & Rubin, 1983) by aiding descriptions and aim in reducing selection bias in the context of clustered data.

Using data from the 2009 National Assessment of Educational Progress (NAEP) for mathematics and reading at grades four and eight, estimates of the differences between charter and traditional public schools were calculated at the state and national levels. This study finds that there is wide variability in math and reading performance for charter schools. But in aggregate, charter schools do not perform any differently than their traditional public school counterparts. The new methods were used to examine potential relationships between the “quality” of state charter laws as determined by the National Alliance for Public Charter Schools (NAPCS; 2010a) and aggregate differences in charter and traditional public school student NAEP scores produced by the new methods are explored. Analyses suggested that these relationships were either absent or modest across the two grades and subjects.

Contents

• Dissertation

• Slides

• Handouts (these are best printed on 13x19 photo paper): Overall Results, Grade 4 Math, Grade 4 Reading, Grade 8 Math, Grade 8 Reading

• The multilevelPSA package is available on CRAN, hosted on Github, and documented on my website. Abstract of a talk given at the 2010 Society of Multivariate Experimental Psychology annual meeting was published in Multivariate Behavior Research, 46(6).

• The naep package is hosted on Github and documented on my website. It can be installed directly from Github using the devtools package using the devtools::install_github('jbryer/naep') command.

• All the R scripts used to conduct the analysis are available in the R directory. Note that the use of NAEP data requires a restricted use license and as such, the data is not publicly available. However, researchers with a license and access to the 2009 NAEP data should be able to replicate my analysis using these scripts.

• Individual figures and tables (in LaTeX format) are located in the Figures2009 and Tables2009 directories, respectively.

• A paper and slides from a 2012 talk at AERA are available.

• Dissertation Proposal and slides.