Allow matrix structure in covariance matrices to be exploited

[kshedden]

This PR provides adds a new method covariance_matrix_solve to the CovStruct class that solves a system of equations whose coefficient matrix is the covariance matrix represented by the CovStruct instance.

Currently, we construct this covariance matrix explicitly, and solve these systems with a general-purpose solver. However, for many of the dependence structures, the linear algebra can be optimized to exploit the structure of the matrix.

We provide a default implementation of covariance_matrix_solve that uses a general-purpose solver.

We override this with optimized methods for the independence, autoregressive, and exchangeable cases.

The speed improvement would be most noticeable when the cluster sizes are large.

Tests for the independence and exchangeable cases match results obtained from R (as did the earlier code).

We don't have a comparator for the autoregressive case, so I added a regression test. It agrees with the results from the previous (non-optimized) implementation.

[coveralls]

Coverage remained the same when pulling 45728d3 on kshedden:gee-linalg-refactor2 into b52bc09 on statsmodels:master.

[josef-pkt]

rslt might be None
?

[coveralls]

Coverage remained the same when pulling 078eaae on kshedden:gee-linalg-refactor2 into b52bc09 on statsmodels:master.

[josef-pkt]

np.squeeze would be safer I think.
np.squeeze will cause shape error later if y.shape[1] > 1
IIUC

[josef-pkt]

M-dependent is the only case I really worked my way through it. So I might postpone understanding this and just do a superficial review before merge.
Frank's m-dependent PR #1495 will need an update after this is merged.

[coveralls]

Coverage remained the same when pulling 4bf5117 on kshedden:gee-linalg-refactor2 into b52bc09 on statsmodels:master.

[coveralls]

Coverage remained the same when pulling 69e1166 on kshedden:gee-linalg-refactor2 into b52bc09 on statsmodels:master.

[coveralls]

Coverage remained the same when pulling 69e1166 on kshedden:gee-linalg-refactor2 into b52bc09 on statsmodels:master.

[kshedden]

Thanks for the comments. I think I've addressed everything.

The m-dependent structure needs a bit of cleanup. I will work with Frank on that.

[josef-pkt]

I'm not a big fan of rhs, but I don't see an alternative. Now with the docstring it's easy to understand.
We don't have a good terminology for this kind of linear equations. Linear restrictions use r_matrix and q_matrix or something like this, R b = q.
rhs is a bit confusing, because before the docstring I thought of y = x b and x is the "rhs", not y.

This looks like a clean branch according to the network tree, and can be merged with green button.

[josef-pkt]

side comment, doesn't need to be changed

in general, I still prefer floating point integers to signal that we are working with floats not integers.
(1. - self.dep_params)

(I still have a left-over habit to hunt for bugs caused by integer division.)

[josef-pkt]

one more: can you inherit the docstring for the methods in the subclasses? for update, covariance_matrix_solve, ...
e.g. generalized linear model has this
622: remove_data.__doc__ = base.LikelihoodModelResults.remove_data.__doc__

I mentioned this in Frank's PR

[josef-pkt]

I think this is about ready to merge.
Kerby, whenever you think it's ready, I will hit the merge button.

[kshedden]

Ready to merge now. Thanks for your help.

[josef-pkt]

merged