Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gaussian_mixture true_theta / observation have shifted with version #45

Closed
bkmi opened this issue Apr 5, 2022 · 1 comment
Closed

Comments

@bkmi
Copy link
Contributor

bkmi commented Apr 5, 2022

I was running the benchmark and found that no method was producing accurate posteriors (according to C2ST) for the gaussian_mixture task. I wondered if the simulator has somehow changed, thereby introducing a different ground truth posterior for each saved observation.

Indeed, this simple check shows that there has been some drift in the simulator

task = sbibm.get_task("gaussian_mixture")
num_observation = 5
true_theta = task.get_true_parameters(num_observation)
sbibm_obs = task.get_observation(num_observation)
new_obs = task.get_simulator()(true_theta)
obss = torch.concat([task.get_simulator()(true_theta) for _ in range(100)])
print(
    (torch.linalg.norm(sbibm_obs - obss)).mean(),
    (torch.linalg.norm(new_obs - obss)).mean(),
)

This typically returns tensor(115.6793) tensor(16.9946).


To fix the issue, either the simulator can be returned to its previous state or we could generate new ground truth parameters and observations; however, this runs the issue of not being backwards compatible with previous versions of sbibm.

@bkmi
Copy link
Contributor Author

bkmi commented Apr 5, 2022

Hmm, now I've rerun things and I'm no longer sure that what I'm saying is correct.

@bkmi bkmi closed this as completed Apr 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant