Memory usage, tidyverse copy-on-modify and mclapply

[whtns]

Thanks for this package. I have run the development release successfully on a few tumor samples. I have lately been running into persistent memory issues especially in phylogeny steps. I am running on our lab's local server with 64 Gb RAM and 8 cores. I have begun to suspect that the issue is the use of tidyverse in mclapply routines. I have recently converted every mclapply step into a basic for loop assigning to a pre-allocated list--ignoring any parallelization speed benefits. That has helped a little bit. I have considered trying to use dtplyr to ease some memory issues.

Do you think this suspicion is reasonable? Have you already made efforts to reduce the memory footprint?

[evanbiederstedt]

Hi @whtns

I was thinking about trying something like this, but I didn't have the time.

The correct approach above is if you create a pull request with those changes for us to review----please provide statistics on memory usages improvements as well if you could.

After forking the repo, please use the develop branch for these changes, and create a PR against that branch.

Thanks, Evan

[teng-gao]

Hello @whtns,

In the most recent update 0.1.3, we have completely replaced mclapply in the phylogeny part using RcppParallel. This led to a 10-20x speedup and the memory usage should now stay constant with respect to the number of threads. Please let us know if that helps with your memory issue!

Thanks,
Teng