Two Stage or Adaptive Estimator Class

[mnarayan]

There are several methods to estimate the inverse covariance matrix (MLE, Pseudolikelihood, Dtrace, CLIME, ...). However, the MLE produces positive semi-definite/symmetric estimate with the standard minimum variance benefits (potentially? asymptotically efficient) and thus appropriate as a final estimator.

Therefore we want an adaptive estimator class that

• Takes an initial estimate as an input
• Takes a few different strategies to create adaptive weights and refits the MLE with a weighted penalty matrix.
  • These could take the form of 1/|coefficient| or 1/|coefficient|^2 (classic adaptive glasso)
  • Or these could be incorporated into a non-convex SCAD or MCP penalty.

See adaptivity/two-stage example in the GELATO (ftp://ess.r-project.org/Manuscripts/buhlmann/gelato.pdf) estimator (Section 3.3). Here the weights are very naive and basically just binary.

[jasonlaska]

Transferring the PR comment to here as I will close out that PR before addressing.

Gelato is a very obscure name. I would just use the 'refitted' or 'relaxed' which is a more widely understood name.

Note: In any documentation we can add in a reference to gelato as well as a paper called "Relaxed lasso" by Meinshausen.

[mnarayan]

I think the implementation of this (as I see in develop branch) will have to be changed, it works out in the binary case but not in the others.

For example, the glasso and the inverse cannot use the standard MLE in the following line of code, they all have to go through making a call to quic_graph_lasso with the appropriate weights.

https://github.com/jasonlaska/scikitquic/blob/develop/inverse_covariance/two_stage_adaptive.py#L62

[jasonlaska]

I havent worked on the adapative routines yet, the branch name was
incorrect

On Sun, Aug 21, 2016 at 11:33 AM mnarayan notifications@github.com wrote:

I think the implementation of this (as I see in develop branch) will have
to be changed, it works out in the binary case but not in the others.

For example, the glasso and the inverse cannot use the standard MLE in the
following line of code, they all have to go through making a call to
quic_graph_lasso with the appropriate weights.

https://github.com/jasonlaska/scikitquic/blob/develop/inverse_covariance/two_stage_adaptive.py#L62

—
You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub
#27 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABk3vP2V2aKFLAuNviUFmC-OKiRpwbxjks5qiJn5gaJpZM4Jj80r
.

[jasonlaska]

For coefficients that are zero, do we just ignore them? How can we do 1 / | coefficient| otherwise?

[mnarayan]

For coefficients that are already 0, put some value much higher than 1 (if
correlation matrix) or the rescaled equivalent. That should ensure that
corresponding entry will be forced to 0.

• Manjari

On Sun, Aug 21, 2016 at 6:24 PM, Jason Laska notifications@github.com
wrote:

For coefficients that are zero, do we just ignore them? How can we do 1 /
| coefficient| otherwise?

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#27 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAov955yOTCHKryOnFI1gutpSFm-LF67ks5qiPo0gaJpZM4Jj80r
.

[jasonlaska]

I guess I kinda have the binary case working;

The last (adaptive) plot is with BIC as the first estimate (ignore lam=-1);

you can see the colors are closer to the original precision matrix. For
the other variants (1/c^2, 1/|c|), it blows away all the coeffs. For
GraphLassoCV, it doesn't appear to reduce the false positives in this
example (if the implementation is correct)

[image: Screen Shot 2016-08-21 at 6.46.47 PM.png]

On Sun, Aug 21, 2016 at 6:29 PM mnarayan notifications@github.com wrote:

For coefficients that are already 0, put some value much higher than 1 (if
correlation matrix) or the rescaled equivalent. That should ensure that
corresponding entry will be forced to 0.

• Manjari

On Sun, Aug 21, 2016 at 6:24 PM, Jason Laska notifications@github.com
wrote:

For coefficients that are zero, do we just ignore them? How can we do 1 /
| coefficient| otherwise?

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<
#27 (comment)
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AAov955yOTCHKryOnFI1gutpSFm-LF67ks5qiPo0gaJpZM4Jj80r

.

—
You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub
#27 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABk3vNFtBlkOlJ2rA_5XunaIjyvP3iToks5qiPtlgaJpZM4Jj80r
.

[mnarayan]

I can't see the screenshot.

for the (1/c^2 and 1/|c|) approaches, i suspect there are scaling issues (there usually are)). It might be necessary to do these in path mode only.

If initial estimate gives us c values then

• set lam to use 1/|c| in the matrix penalty component, and
• put in a grid of lambda values (standard range) in the path parameter.
• Re-run graphlasso or graphlassocv in this way.

This is one way of assessing what the right "scaling" should be as long as the grid of values covers the necessary range.

[jasonlaska]

Some working code; will close this issue once I've added tests.

• Unit tests