Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce peak memory usage during performance evaluation #450

Merged
merged 5 commits into from
Sep 13, 2023

Conversation

fweber144
Copy link
Collaborator

Closes #440 by implementing the suggestion from #440 (comment). I can confirm that the reprex from #440 (comment) now runs through on my machine (the same machine that I was using originally for that reprex) and only errors at the end when the projection onto the full model takes place (which is probably due to #323; the error message is Error in if (any(edgevals <- 0 < bdiff & bdiff < boundary.tol)) { :\nmissing value where TRUE/FALSE needed). Running the reprex with nterms_max = 3 and peakRAM::peakRAM() around the varsel() expression (twice; takes long enough) confirms the reduction of peak memory usage:

# Branch `master`:
peak_old <- replicate(2, peakRAM(eval(vs_expr)), simplify = FALSE)
quantile(sapply(peak_old, "[[", "Peak_RAM_Used_MiB"))
#       0%      25%      50%      75%     100%
# 10659.80 10668.08 10676.35 10684.62 10692.90

# Branch `merge_getters` (this PR):
peak_new <- replicate(2, peakRAM(eval(vs_expr)), simplify = FALSE)
quantile(sapply(peak_new, "[[", "Peak_RAM_Used_MiB"))
#       0%      25%      50%      75%     100%
# 5297.800 5304.425 5311.050 5317.675 5324.300

…y usage

(before, the submodel fits (`outdmin` elements stored in `submodl` objects)
corresponding to the increasing submodel sizes along the predictor ranking were
stored at the same time, which led to a high peak memory usage; this is avoided
now).
…nto `get_submodls()`

in the second-to-last commit.
… into `get_submodls()`

in the third-to-last commit.
@fweber144 fweber144 merged commit 7186e6f into stan-dev:master Sep 13, 2023
@fweber144 fweber144 deleted the merge_getters branch September 13, 2023 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Memory issues in multilevel models
1 participant