Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add max_samples argument to calculate() #236

Closed
goldingn opened this issue Oct 15, 2018 · 2 comments
Closed

add max_samples argument to calculate() #236

goldingn opened this issue Oct 15, 2018 · 2 comments

Comments

@goldingn
Copy link
Member

When running calculate() with an mcmc.list object, all of the samples are pushed through the model simultaneously. This is usually the most efficient approach, but can run into memory constraints when the number of samples is large, and the model already had very large data objects.

If there were a max_samples argument (defaulting to Inf), the user could just process the samples in batches instead.

We could also try to catch the out-of memory error, and return a nicer message to the user.

@goldingn goldingn mentioned this issue Jan 8, 2020
23 tasks
@goldingn
Copy link
Member Author

goldingn commented Jan 8, 2020

Doesn't look like catching an out of memory error will be reliable - at least on my mac it crashes R Studio (segfault?) and predicting the memory to be used by an operation doesn't seem feasible - allowing users to tweak it, and using a sensible default value seems like thee best option.

@goldingn
Copy link
Member Author

goldingn commented Jan 8, 2020

Here's a cludgy user-level version of batching calculate, to adapt:

window_greta <- function (x, start, finish) {
  mi <- attr(x, "model_info")
  mi <- rlang:::env_clone(mi)
  x <- coda:::window.mcmc.list(x, start, finish)
  mi$raw_draws <- coda:::window.mcmc.list(mi$raw_draws, start, finish)
  attr(x, "model_info") <- mi
  x
}

library(greta)
x <- normal(0, 1)
m <- model(x)
draws <- mcmc(m)
y <- x ^ 2

draws_1 <- window_greta(draws, 1, 500)
draws_2 <- window_greta(draws, 501, 1000)
y_out_1 <- calculate(y, draws_1)
y_out_2 <- calculate(y, draws_2)
y_mat <- rbind(as.matrix(y_out_1),
               as.matrix(y_out_2))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant