-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pspecdata.I now accounts for flags in data by taking time avg #94
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just checking: @acliu: Should the weights really be included here? They are used in the empirical covariance calculation because the estimator for C_emp uses the (inverse variance-)weighted data by definition. The definition of I
is just I
though, regardless of what the data look like.
In general: Should our covariance methods return w^T C w
, or just C
?
Ok, I just checked some of the math, to see if having the non-trivial weights messes up anything with our normalization. It turns out that it does, but it can be accommodated within our current architecture. I can do this on a separate branch, or we can do it on this one if you're not in a hurry to get this done. I'll include the revised mathematical expressions in the notes. (It makes sense that there's a difference---consider a situation where the first half of the band is all flagged. Then you've effectively halved the bandwidth, and the normalization needs to know about that. For more complicated flagging/weights it's not as simple as just changing the bandwidth, but the expression can still be written down). @nkern @philbull Are the weights here purely binary weights to indicate flagged vs. not flagged? Or are we intending to use this for more general weights as well? My preference is that If the weights are intended to only be binary, I could be receptive to having them incorporated into |
@acliu to answer your first question: I think this is a must-have feature but not totally time critical, so we should wait on this PR until we figure out how exactly we want to incorporate data flags. If you want to push your edits to this branch you should feel free to do so. To answer your next question: I think weights should be binary, so that we can think of them as flags. I agree we should think of covariance matrices as either |
previously,
pspecdata.I()
was not accounting for data flags. this PR accounts for data flags by taking the time average of the data weights and multiplying that into the diagonal of identity.addresses #72 and partially addresses #55