Allow sandwich/bootstrap/jackknife standard errors as options

[nalimilan]

This is more a feature request or policy question than a bug report.

I'm wondering whether you would like to add an argument allowing to easily compute sandwich (heteroskedasticity-robust), bootstrap, jackknife and possibly other types of variance-covariance matrix and standard errors, instead of the asymptotic ones. Or whether you would like to provide a function to compute these after the model has been fitted.

R makes it relatively cumbersome to get these common standard errors (see e.g. [1] or [2]), which either means that researchers with limited programming skills are incited to keep using Stata, or to keep relying only on asymptotic standard errors. I see no good reason why such an option shouldn't be provided in the standard modeling package.

1: http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-bootstrapping.pdf
2: http://cran.r-project.org/web/packages/sandwich/vignettes/sandwich-OOP.pdf

[gragusa]

On this issue a I have written a basic package to calculate autocorrelation and heteroskedasticity robust standard error https://github.com/gragusa/VCOV.jl.

There are methods that work for GLM.jl models (at the moment tested only for models without weights).

I am planing to finish it in the next couple of weeks. However, it would be nice to have some infrastructure setup at the StatsBase level.

For instance, for models such that the parameters solve \sum_{i=1}^n g(W_i, \hat{\beta}) = 0, we could think of methods (for StatisticalModel type) that returns, \sum_{i=1}^n g(W_i, \hat{\beta}) and \sum_{i=1}^n \partial g(W_i, \hat{\beta})/\partial \beta'. This would make immediately available HAC type standard errors available for all models.

For showing the output would be only a matter to extend the show method to accept an argument defining the type of variance.

[nalimilan]

@gragusa I had missed your comment, but it sounds cool. I guess open an issue against StatsBase with suggestions about the API to discuss this.

[gragusa]

I wrote a (registered) package CovarianceMatrices.jl that is already pretty
exhaustive in terms of features, but I feel the api can be further
improved.
On Fri 6 Mar 2015 at 15:29 Milan Bouchet-Valat notifications@github.com
wrote:

@gragusa https://github.com/gragusa I had missed your comment, but it
sounds cool. I guess open an issue against StatsBase with suggestions about
the API to discuss this.

—
Reply to this email directly or view it on GitHub
#42 (comment).

[matthieugomez]

I also think one should be able to specify standard errors in fit (rather than using a vcov function on the result of fit as in R) for several reasons:
(i) The estimates may depend on the method chosen to specify errors. In particular, if I specify clustered errors wrt some variable, and this variable is missing for some observations, the estimate should be computed based only on those observations. This looks like an weird case but this happens in a lot with the data I'm working with
(ii) more friendly API for the user (one gets the relevant summary table right away).
From a simple google search, the issue is discussed here, here, here and here, etc.
(iii) faster (since a lot of computations are generally redundant between estimates and error estimations)
(iv) potentially lighter output of regression results since models don't need to carry every matrix to compute errors (it matters when working with datasets > Go, and obviously when working with datasets out of memory). The huge size of lm and glm models in R is discussed here, here, here here (and for absurd consequences, here and there).

A nice solution, which relies on multiple dispatch, would be to define an AbstractVcov type, and use it in the signature of fit. Any child type would then a particular way to compute errors. For instance, one child type is VcovCluster. An instance of the type can be constructed withVcovCluster(:State), where State is the variable I want to cluster on.

Errors method could then be specified as an argument to fit

fit(LinearModel, formula, df, VceCluster(:State))
fit(LinearModel, formula, df, VceHAC(:Year, lag = 2, kernell = Bartlett))

I've tried to do such an implementation in my fixed effects package

[nalimilan]

Interesting arguments.

The interface you suggest sounds good, but what about the implementation? How, when and with what arguments should the AbstractVcov constructor be called, so that we can handle not only clustered standard errors, but also e.g. various kinds of bootstrap? I guess for bootstrap the fitting method should pass a function which would be called again to fit the replicates. This might deserve some common infrastructure in StatsBase, as the procedure is essentially similar for many kinds of models.

[gragusa]

I think in principle having the fitting method dispatch on AbstractVcov
is a good idea, but it is a nightmare from a implementation point of view.
Also, you should have also Vcov methods for the fitted model (for
instance you estimate GLM with a Vcov and then you want to see how the
standard errors change under different assumptions on the covariance of the
errors. But this would amounts of having lots of duplication.

That's why in CovarianceMatrices I went the other way. I agree that for
bootstrap we can have a common infrastructure.

I think he would be enough to have a summary object that dispatch on the
RobustVariance.

Giuseppe Ragusa

On Thu, Jun 25, 2015 at 2:04 PM, Milan Bouchet-Valat <
notifications@github.com> wrote:

Interesting arguments.

The interface you suggest sounds good, but what about the implementation?
How, when and with what arguments should the AbstractVcov constructor be
called, so that we can handle not only clustered standard errors, but also
e.g. various kinds of bootstrap? I guess for bootstrap the fitting method
should pass a function which would be called again to fit the replicates.
This might deserve some common infrastructure in StatsBase, as the
procedure is essentially similar for many kinds of models.

—
Reply to this email directly or view it on GitHub
#42 (comment).

[nalimilan]

@gragusa Why do you think there should be code duplication? The AbstractVcov constructor could be passed the same arguments during model fitting as after (i.e. when called on the resulting object).

The problems with only choosing the standard errors type when calling summary have been detailed by @matthieugomez. It doesn't look like it would work very well for many cases.

[matthieugomez]

Exactly, computing errors as part of the fit does not require code duplication. I actually also define a vcov method in my package. The only difference with your package is that my vcov function (i) is not directly called by the user (ii) acts on a hat matrix, a matrix of regressor, and a vector of residuals, rather than the result of a glm model.

In fact, every type child of AbstractVcov has two methods, a allvars (that get all the variables needed to estimate the errors, which makes sure missing rows are removed before the estimation), and a vcov method

The fit function in just written

function fit(formula, df, vcov_method::AbstractVcov)
...
# instead of the df[complete_cases(df[allvars(formula)] step in ModelFrame
df[complete_cases(df[allvars(formula), allvars(vcov_method])
.....
# instead of returning the usual covariance matrix
vcov(vcov_method, df, Xhat, X, residuals)
end

This is actually similar to how Stata work: the avar package in Stata acts on matrices rather than models & is used under the hood by very different fit models https://ideas.repec.org/c/boc/bocode/s457689.html

[gragusa]

The problem is functional redundancy (for luck of a better word). In stata you have to refit the model to get new standard error. In julia you could

reg_1 = fit(::formula, df, ::AbstractVcov)

and then have

vcov(reg_1, ::AbstractVcov)

to get new standard errors. So you have two different interfaces to do the same thing.

[nalimilan]

I don't see why that would be an issue. It works the same in R. Also, these are not really the same since vcov only gives you the matrix, while passing the argument to fit would mean all actions on the model would use the chosen standard errors: summary, confidence intervals, plots...

[gragusa]

Thinking more about it, I do not agree to include variance information in the fit method. The reason is that logically the variance info should be relevant to fit, but here it is only relevant to the summary.

If the variance of the error vector is not a multiple of the identity matrix, then the efficient estimator is different. So I would expect a different fitting method.

Conceptually, i would expect

glm_1 = fit(::GLM, df, ::WorkingVar) 

and WorkingVar would be, for instance, Exchangeable, AR. In this case I would expect a fitting by Estimating Equations. Then, even in this case, I could ask what the Robust variance is

summary(glm_1, ::AbstractVcov)

What i am trying to say is that a variance in fit is logically a different object than a variance in summary.

[nalimilan]

@gragusa Sorry, I'm not familiar with the methods you're talking about. But it seems to me that in many cases several variance estimation methods can be used with a given fitting method. The simplest example is OLS regression, which can be combined with asymptotic (the default), heteroskedasticy-robust/sandwich, jackknife, bootstrap, and several kinds of survey design-based methods.

Anyway, we're not saying you shouldn't be allowed to pass another method to summary if needed: only that you should also be able to pass it to fit. For models where it might not make sense, raising an error is fine.

Finally, I don't think you addressed @matthieugomez's point that for clustered standard errors, one needs to get rid of missing values on the clustering variable before the fit.

[Nosferican]

I had a vcov(obj::MyPkg.Struct, vce::AbstractVCE) method in my package, but eventually moved to specifying it at the moment of building the model in part because the cluster variables are a defining feature. In order to request a different vcov one needs to refit the model, but as a benefit the implementation is a lot smoother. Basically packages only have to implement,

StatsBase.modelmatrix(obj::StatsBase.RegressionModel) =
    error("modelmatrix is not defined for $(typeof(obj)).")
StatsBase.residuals(obj::StatsBase.RegressionModel) =
    error("residuals is not defined for $(typeof(obj)).")
StatsBase.dof(obj::RegressionModel) =
    error("dof is not defined for $(typeof(obj)).")
StatsBase.dof_residual(obj::RegressionModel) =
    error("dof_residual is not defined for $(typeof(obj)).")

In addition, it has to provide the following methods,

getvce(obj::StatsBase.RegressionModel) = :OLS
function getclusters(obj::StatsBase.RegressionModel)
    û = StatsBase.residuals(obj)
    output = Matrix{Int64}(length(û),1)
    output[:,1] = eachindex(û)
    return output
end
function getΓ(obj::StatsBase.RegressionModel)
    n = size(StatsBase.modelmatrix(obj), 2)
    output = zeros(n, n)
    return output
end

A variance covariance package should be able to compute any variance covariance estimator for linear models with that information. If packages save in their structs the inverse of the normal matrix or the pseudo-inverse (bread), these could be accessed and avoid computing the inverse twice.
The API would just be two functions

function vcov!(obj::StatsBase.RegressionModel, fieldname::Symbol)
	setfield(obj, fieldname, vcov(obj, vce(obj)))
end
function vce(obj::StatsBase.RegressionModel)
	Bread = getbread(obj)
	ũ = getũ(obj)
	G = getG(obj)
	Meat = ũ * ũ.' .* G
	fsa = getγ(obj)
	output = fsa * Bread * Meat * Bread.'
	return output
end

[nalimilan]

@Nosferican The best way to show that this design suits people's needs is to put them in a package and write implementations using it for the various packages that can take advantage of it. (BTW, in Julia we usually omit the get prefix to functions.)

[Nosferican]

Here is a prototype VarianceCovarianceEstimators.jl. It provides OLS, and the multi-way clustering generalized estimators for HC1, HC2, and HC3. It relies on the StatsBase.RegressionModel methods.