-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mismatch between posteriors from polychord vs importance sampler #124
Comments
I haven't been able to replicate this directly. Could you confirm from looking at the chain headers that the exact same cosmosis-standard-library version was used in each case? And perhaps give me some more detail on exactly what you ran to replicate? |
Hm. I just attempted to reproduce the vary-m1-only result on my laptop, and am finding matching polychord and IS posteriors. Since I'd previously ran the test on my office computer, I won't have access to those output files for another week or so (I'm traveling). I'm relatively sure that in that case I was using the same CSL version since I ran polychord then IS in succession with no code updates between them. Previously I'd seen mismatches when using IS to save extra derived parameters for DES Y3 chains. I'll aim to hunt down the details on those within the next few days. |
Sorry for the delayed response to this! I was actually running with Nautilus rather than Polychord (an on NERSC). I was worried about different versions so ran a new Nautilus chain and then ran the Importance Sampler using the exact same params.ini, just changing the sampler, with only a few hours in between and no code updates. Here's a portion of the Nautilus output header:
And here is the corresponding IS one:
Interestingly the cosmosis and CSL git versions are blank, but like I said they should be identical. |
Could you send me the full chains, or just the full ini files that generated them? |
Just to follow up: I'm now unable to replicate the mismatch I noted in my post above where I said
I don't know if I had an error in my initial test, or some update to my workspace or to cosmosis in the meantime affected this. In any case, now on my simple desktop tests I'm unable to replicate the polychord-vs-IS mismatch. Apologies for adding confusion to the discussion! |
Hello! I've been investigating something that was initially flagged by Noah (@nweaverd), where the posterior reported by the importance sampler and polychord sampler slightly differ in some cases even when they are run with the exact same cosmosis pipeline.
The context is that occasionally we've been running an importance sampler on chains as an approach to save model predictions for datavectors that can be used for making plots, PPD calculations, and some validation tests. In principle, doing this with the exact same cosmosis pipeline, same data, same scale cuts, should produce importance sampler output where all the log_weight entries are 0, meaning the IS
post
is identical to the original sampler'sold_post
. We're finding that this isn't always the case.(There's a thread on this in the cosmosis channel of the DES slack team; let me know if you want to be pointed to that.)
We're still not sure what's happening, but compiling notes on what we've checked so far:
extra_output
columns in either the original or IS chain doesn't affect this.log_weight
correlates with anything; it doesn't have any obvious correlation with post, like, prior, old_post, or the handful of individual parameters we checked.We have a couple more ideas of things to check. I'll add notes here as we get to them.
I know that running a test sampler handles priors differently that running a sampler with varying parameters, so it seems plausible that there might be worth scrutinizing how the importance and list samplers do this, but I don't have a good enough understanding to be sure.
The text was updated successfully, but these errors were encountered: