Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow MetropolisHastings samples to be dropped as they are generated #163

Closed
mhutti1 opened this issue Aug 13, 2018 · 1 comment
Closed

Comments

@mhutti1
Copy link
Contributor

mhutti1 commented Aug 13, 2018

Is your feature request related to a problem? Please describe.
I am sampling each layer of a tree consecutively and then downsampling to a every complete iteration of the tree. The issue I am facing is that to take a decent number of samples I first need to store each vertex and every sample for it before downsampling. This uses a lot of memory and once I hit about 80GB memory pressure rises and the system becomes unusable.

Describe the solution you'd like
I would like to be able to downsample as we go. E.g Take around 140 samples (one sample for each tree layer), then drop all but the last one and repeat for the desired sample count. One option would be to add downsampling as an optional parameter to the getPosteriorSamples method.

Describe alternatives you've considered
Other possible solutions would be to allow users to create their own drop selector and pass it in, or to use streams to filter in real time (although this might be more complex).

@christophernorth
Copy link
Contributor

Fixed by PR #172

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants