Last edits to CRVE-FE paper.

paper_ClusterRobustTesting/ClusterRobustTesting_FE_models.Rnw

@@ -10,7 +10,7 @@
 \usepackage{multirow}
 
 % NOTE: To produce blinded version, replace "0" with "1" below.
-\newcommand{\blind}{0}
+\newcommand{\blind}{1}
 
 % DON'T change margins - should be 1 inch all around.
 \addtolength{\oddsidemargin}{-.5in}%
@@ -725,6 +725,10 @@ We join \citet{Imbens2015robust} in arguing that the CR2 estimator and correspon
 Because the performance of standard CRVE methods depends on features of the study design, the total number of clusters in the data is an insufficient guide to whether small-sample corrections are needed. 
 Instead, the clearest way to determine whether small-sample corrections are needed is simply to calculate them.
 
+The idea of developing small-sample adjustments based on a working model may seem strange to analysts accustomed to using CRVE---after all, the whole point of clustering standard errors is to avoid making assumptions about the error structure.
+However, simulation studies reported here and elsewhere \maskcitep{Tipton2015small-t, Tipton2015small-F} have demonstrated that the approach is actually robust to a high degree of misspecification in the working model. 
+Furthermore, while the working model provides necessary scaffolding when the number of clusters is small, its  influence tends to fall away as the number of clusters increases, so that the CR2 estimator and AHT maintain the same asymptotic robustness as standard CRVE methods. 
+
 The proposed AHT test involves two adjustments: use of the CR2 adjustment for the variance estimator and use of estimated degrees of freedom. 
 Our simulation results and empirical examples illustrate that the degrees of freedom adjustment has a relatively larger influence on small-sample performance.
 Even when used with the CR1 adjustment matrices, the degrees of freedom adjustment leads to much more accurate rejection rates, although using the CR2 estimator (and accounting for absorption of fixed effects) appears to be necessary to fully maintain the nominal level of the test. 
@@ -733,20 +737,16 @@ The estimated degrees of freedom are indicative of the precision of the standard
 We therefore recommend that the degrees of freedom be reported along with standard errors and $p$-values whenever the method is applied.
 
 It is interesting to note that the consequences of accounting for fixed effects estimation with CR2 runs counter to how fixed effects enter into other small-sample corrections. 
-In particular, including the fixed effect parameter dimension in the CR1S small sample adjustment can lead to bias when the per-cluster sample size is small \citep{Cameron2015practitioners}, whereas accounting for the fixed effects with the CR2 estimator \textit{improves} the accuracy of test rejection rates, although the differences are comparatively minor. 
-Accounting for fixed effects also makes hypothesis test results invariant to how the regression is calculated (whether by LSDV or after absorption), which we see as a useful feature for ensuring the replicability of an analysis. 
-
-The idea of developing small-sample adjustments based on a working model may seem strange to analysts accustomed to using CRVE---after all, the whole point of clustering standard errors is to avoid making assumptions about the error structure.
-However, simulation studies reported here and elsewhere \maskcitep{Tipton2015small-t, Tipton2015small-F} have demonstrated that the approach is actually robust to a high degree of misspecification in the working model. 
-Furthermore, while the working model provides necessary scaffolding when the number of clusters is small, its  influence tends to fall away as the number of clusters increases, so that the CR2 estimator and AHT maintain the same asymptotic robustness as standard CRVE methods. 
+In particular, including the fixed effect parameter dimension in the CR1S small sample adjustment can lead to bias when the per-cluster sample size is small \citep{Cameron2015practitioners}, whereas accounting for the fixed effects with the CR2 estimator \textit{improves} the accuracy of test rejection rates, although the difference is comparatively minor. 
+Accounting for fixed effects also makes hypothesis test results invariant to how the regression is calculated (whether by LSDV or after absorption), which we see as a useful feature for ensuring the replicability of one's analysis. 
 
 One limitation of our approach is that the rejection rates of the AHT tests tend to fall below nominal levels when the number of clusters is very small and the design is unbalanced. 
 The under-rejection can be severe for tests of multi-dimensional constraints (e.g., $q = 3$-dimensional hypotheses with $m = 15$ clusters in an unbalanced difference-in-differences design). 
-In principle, this problem could arise either because the proposed degrees of freedom estimator becomes less accurate or because the sampling distribution of the test statistic is no longer well-approximated by Hotelling's $T^2$ distribution at all. 
+In principle, this problem could arise either because the proposed degrees of freedom estimator loses accuracy under these conditions or because the sampling distribution of the test statistic is no longer well-approximated by Hotelling's $T^2$ distribution at all. 
 In previous work, we investigated several other methods of approximating the degrees of freedom, but found none that were more accurate than the method describe in the present paper \citep{Tipton2015small-F}. 
-Thus, future work may need to focus on other approximations to the reference distribution itself, such as an F distribution in which both the numerator and denominator degrees of freedom are estimated \citep[cf.][]{Mehrotra1997improving} or computationally intensive approximations such as the cluster-wild bootstrap \citep{MacKinnon2016wild}.   
+Thus, future work may need to focus on other approximations to the reference distribution itself, such as an F distribution in which both the numerator and denominator degrees of freedom are estimated \citep[cf.][]{Mehrotra1997improving} or computational approximations such as the cluster-wild bootstrap \citep{MacKinnon2016wild}.   
 
-Another outstanding limitation of the CR2 variance estimator is that it can become computationally costly (or even infeasible) when the within-cluster sample sizes are large \citep{Mackinnon2015wildCIs}. 
+Another outstanding limitation of the CR2 variance estimator is that it is costly (or even infeasible) to compute when the within-cluster sample sizes are large \citep{Mackinnon2015wildCIs}. 
 For example, \citet{Bertrand2004how} analyzed micro-level data from a 21-year panel of current population survey data, with clustering by state. Their data included some state-level clusters with over $n_i = 10,000$ individual observations. 
 The CR2 adjustment matrices have dimension $n_i \times n_i$, and would be very expensive to compute in this application. 
 Methods for improving the computational efficiency of the CR2 variance estimator (or alternative estimators that have similar performance to CR2), should be investigated further. 

paper_ClusterRobustTesting/ClusterRobustTesting_FE_models.log

@@ -1,4 +1,4 @@
-This is pdfTeX, Version 3.1415926-2.5-1.40.14 (MiKTeX 2.9) (preloaded format=pdflatex 2014.1.2)  20 SEP 2016 23:10
+This is pdfTeX, Version 3.1415926-2.4-1.40.13 (MiKTeX 2.9) (preloaded format=pdflatex 2013.9.5)  21 SEP 2016 14:56
 entering extended mode
 **ClusterRobustTesting_FE_models.tex
 
@@ -8,10 +8,10 @@ LaTeX2e <2011/06/27>
 Babel <v3.8m> and hyphenation patterns for english, afrikaans, ancientgreek, ar
 abic, armenian, assamese, basque, bengali, bokmal, bulgarian, catalan, coptic, 
 croatian, czech, danish, dutch, esperanto, estonian, farsi, finnish, french, ga
-lician, german, german-x-2013-05-26, greek, gujarati, hindi, hungarian, iceland
+lician, german, german-x-2012-05-30, greek, gujarati, hindi, hungarian, iceland
 ic, indonesian, interlingua, irish, italian, kannada, kurmanji, latin, latvian,
  lithuanian, malayalam, marathi, mongolian, mongolianlmc, monogreek, ngerman, n
-german-x-2013-05-26, nynorsk, oriya, panjabi, pinyin, polish, portuguese, roman
+german-x-2012-05-30, nynorsk, oriya, panjabi, pinyin, polish, portuguese, roman
 ian, russian, sanskrit, serbian, slovak, slovenian, spanish, swedish, swissgerm
 an, tamil, telugu, turkish, turkmen, ukenglish, ukrainian, uppersorbian, usengl
 ishmax, welsh, loaded.
@@ -174,7 +174,7 @@ This is `epsf.tex' v2.7.4 <14 February 2011>
 Package: enumerate 1999/03/05 v3.00 enumerate extensions (DPC)
 \@enLab=\toks26
 )
-("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\booktabs\booktabs.sty"
+(C:\Users\jep2963\AppData\Roaming\MiKTeX\2.9\tex\latex\booktabs\booktabs.sty
 Package: booktabs 2005/04/14 v1.61803 publication quality tables
 \heavyrulewidth=\dimen126
 \lightrulewidth=\dimen127
@@ -212,7 +212,7 @@ Package: longtable 2004/02/01 v4.11 Multi-page Table package (DPC)
 \c@LT@chunks=\count110
 \LT@p@ftn=\toks27
 )
-("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\apacite\apacite.sty"
+(C:\Users\jep2963\AppData\Roaming\MiKTeX\2.9\tex\latex\apacite\apacite.sty
 Package: apacite 2013/07/21 v6.03 APA citation
 \c@BibCnt=\count111
 \bibleftmargin=\skip54
@@ -221,7 +221,7 @@ Package: apacite 2013/07/21 v6.03 APA citation
 \bibitemsep=\skip57
 \biblabelsep=\skip58
 )
-("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\natbib\natbib.sty"
+(C:\Users\jep2963\AppData\Roaming\MiKTeX\2.9\tex\latex\natbib\natbib.sty
 Package: natbib 2010/09/13 8.31b (PWD, AO)
 \bibhang=\skip59
 \bibsep=\skip60
@@ -230,10 +230,10 @@ LaTeX Info: Redefining \cite on input line 694.
 )
 LaTeX Info: Redefining \BBA on input line 59.
 
-("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\caption\caption.sty"
+(C:\Users\jep2963\AppData\Roaming\MiKTeX\2.9\tex\latex\caption\caption.sty
 Package: caption 2013/05/02 v3.3-89 Customizing captions (AR)
 
-("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\caption\caption3.sty"
+(C:\Users\jep2963\AppData\Roaming\MiKTeX\2.9\tex\latex\caption\caption3.sty
 Package: caption3 2013/05/02 v1.6-88 caption3 kernel (AR)
 Package caption3 Info: TeX engine: e-TeX on input line 57.
 \captionmargin=\dimen140
@@ -247,7 +247,7 @@ Package caption3 Info: TeX engine: e-TeX on input line 57.
 \c@ContinuedFloat=\count113
 Package caption Info: longtable package is loaded.
 
-("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\caption\ltcaption.sty"
+(C:\Users\jep2963\AppData\Roaming\MiKTeX\2.9\tex\latex\caption\ltcaption.sty
 Package: ltcaption 2013/02/03 v1.3-62 longtable captions (AR)
 ))
 ("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\multirow\multirow.sty"
@@ -264,7 +264,7 @@ Package: float 2001/11/08 v1.3d Float enhancements (AL)
 ("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\rotfloat\rotfloat.sty"
 Package: rotfloat 2004/01/04 v1.2 Combining float+rotating package (AS)
 
-("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\rotating\rotating.sty"
+(C:\Users\jep2963\AppData\Roaming\MiKTeX\2.9\tex\latex\rotating\rotating.sty
 Package: rotating 2009/03/28 v2.16a rotated objects in LaTeX
 
 ("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\base\ifthen.sty"
@@ -291,7 +291,7 @@ Package: xkeyval 2012/10/14 v2.6b package option processing (HA)
 \XKV@depth=\count116
 File: xkeyval.tex 2012/10/14 v2.6b key=value parser (HA)
 ))
-("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\xcolor\xcolor.sty"
+(C:\Users\jep2963\AppData\Roaming\MiKTeX\2.9\tex\latex\xcolor\xcolor.sty
 Package: xcolor 2007/01/21 v2.11 LaTeX color extensions (UK)
 
 ("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\00miktex\color.cfg"
@@ -608,7 +608,7 @@ LaTeX Info: Redefining \addtolength on input line 77.
 \c@thm=\count140
 \c@lem=\count141
 
-("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\upquote\upquote.sty"
+(C:\Users\jep2963\AppData\Roaming\MiKTeX\2.9\tex\latex\upquote\upquote.sty
 Package: upquote 2012/04/19 v1.3 upright-quote and grave-accent glyphs in verba
 tim
 )
@@ -627,7 +627,7 @@ LaTeX Font Info:    ... okay on input line 90.
 LaTeX Font Info:    Checking defaults for U/cmr/m/n on input line 90.
 LaTeX Font Info:    ... okay on input line 90.
 
-("C:\Program Files (x86)\MiKTeX 2.9\tex\context\base\supp-pdf.mkii"
+(C:\Users\jep2963\AppData\Roaming\MiKTeX\2.9\tex\context\base\supp-pdf.mkii
 [Loading MPS to PDF converter (version 2006.09.02).]
 \scratchcounter=\count142
 \scratchdimen=\dimen196
@@ -642,7 +642,7 @@ LaTeX Font Info:    ... okay on input line 90.
 \everyMPtoPDFconversion=\toks46
 )
 \c@maskedRefs=\count148
- ("C:\Program Files (x86)\MiKTeX 2.9\tex\latex\apacite\english.apc"
+ (C:\Users\jep2963\AppData\Roaming\MiKTeX\2.9\tex\latex\apacite\english.apc
 File: english.apc 2013/07/21 v6.03 apacite language file
 LaTeX Info: Redefining \BPBI on input line 129.
 LaTeX Info: Redefining \BHBI on input line 130.
@@ -655,12 +655,13 @@ Package caption Info: End \AtBeginDocument code.
 ABD: EveryShipout initializing macros [1
 Non-PDF special ignored!
 
-{C:/ProgramData/MiKTeX/2.9/pdftex/config/pdftex.map}]
-[2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17]
+{C:/Users/jep2963/AppData/Local/MiKTeX/2.9/pdftex/config/pdftex.map}] [2] [3] [
+4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]
+[15] [16] [17]
 
 Package xcolor Warning: Incompatible color definition on input line 526.
 
-<CR_fig/overview-1.pdf, id=73, 722.7pt x 361.35pt>
+<CR_fig/overview-1.pdf, id=72, 722.7pt x 361.35pt>
 File: CR_fig/overview-1.pdf Graphic file (type pdf)
  <use CR_fig/overview-1.pdf>
 Package pdftex.def Info: CR_fig/overview-1.pdf used on input line 528.
@@ -671,7 +672,7 @@ CR_fig/overview-1.pdf>]
 
 Package xcolor Warning: Incompatible color definition on input line 553.
 
-<CR_fig/balance-1.pdf, id=89, 722.7pt x 397.485pt>
+<CR_fig/balance-1.pdf, id=88, 722.7pt x 397.485pt>
 File: CR_fig/balance-1.pdf Graphic file (type pdf)
  <use CR_fig/balance-1.pdf>
 Package pdftex.def Info: CR_fig/balance-1.pdf used on input line 555.
@@ -680,7 +681,7 @@ Package pdftex.def Info: CR_fig/balance-1.pdf used on input line 555.
 
 Package xcolor Warning: Incompatible color definition on input line 574.
 
-<CR_fig/df-1.pdf, id=90, 578.16pt x 237.88875pt>
+<CR_fig/df-1.pdf, id=89, 578.16pt x 237.88875pt>
 File: CR_fig/df-1.pdf Graphic file (type pdf)
  <use CR_fig/df-1.pdf>
 Package pdftex.def Info: CR_fig/df-1.pdf used on input line 576.
@@ -697,21 +698,20 @@ Missing character: There is no 
 Missing character: There is no � in font cmr12!
 Missing character: There is no � in font cmr12!
 Missing character: There is no � in font cmr12!
- [32]
 Missing character: There is no � in font cmr12!
 Missing character: There is no � in font cmr12!
 Missing character: There is no � in font cmr12!
- [33]) [34]
+ [32] [33]) [34]
 (C:\Users\jep2963\Documents\R\clubSandwich\paper_ClusterRobustTesting\ClusterRo
 bustTesting_FE_models.aux) ) 
 Here is how much of TeX's memory you used:
- 14760 strings out of 493921
- 284953 string characters out of 3144871
+ 14755 strings out of 493921
+ 285174 string characters out of 3144871
  321322 words of memory out of 3000000
- 17756 multiletter control sequences out of 15000+200000
- 14754 words of font info for 52 fonts, out of 3000000 for 9000
+ 17751 multiletter control sequences out of 15000+200000
+ 13461 words of font info for 47 fonts, out of 3000000 for 9000
  841 hyphenation exceptions out of 8191
- 62i,13n,75p,697b,455s stack positions out of 5000i,500n,10000p,200000b,50000s
+ 62i,8n,75p,697b,455s stack positions out of 5000i,500n,10000p,200000b,50000s
 <C:/Program Files (x86)/MiKTeX 2.9/fonts/type1/publ
 ic/amsfonts/cm/cmbx10.pfb><C:/Program Files (x86)/MiKTeX 2.9/fonts/type1/public
 /amsfonts/cm/cmbx12.pfb><C:/Program Files (x86)/MiKTeX 2.9/fonts/type1/public/a
@@ -723,13 +723,12 @@ ts/cm/cmmi12.pfb><C:/Program Files (x86)/MiKTeX 2.9/fonts/type1/public/amsfonts
 cmr10.pfb><C:/Program Files (x86)/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmr
 12.pfb><C:/Program Files (x86)/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmr8.p
 fb><C:/Program Files (x86)/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmsy10.pfb
-><C:/Program Files (x86)/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmsy7.pfb><C
-:/Program Files (x86)/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmsy8.pfb><C:/P
-rogram Files (x86)/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmti12.pfb><C:/Pro
-gram Files (x86)/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmtt12.pfb>
-Output written on ClusterRobustTesting_FE_models.pdf (34 pages, 313289 bytes).
+><C:/Program Files (x86)/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmsy8.pfb><C
+:/Program Files (x86)/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmti12.pfb><C:/
+Program Files (x86)/MiKTeX 2.9/fonts/type1/public/amsfonts/cm/cmtt12.pfb>
+Output written on ClusterRobustTesting_FE_models.pdf (34 pages, 299657 bytes).
 PDF statistics:
- 201 PDF objects out of 1000 (max. 8388607)
+ 197 PDF objects out of 1000 (max. 8388607)
  0 named destinations out of 1000 (max. 500000)
  28 words of extra memory for PDF output out of 10000 (max. 10000000)