-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
parallel support #39
Comments
And progress bars of course. Apparently we can do this in |
I think a nice middle ground here might be to support parallel computation the same way One special case could possibly be done for if(parallel > 1){
if(!require('doParallel')){
warning("Parallel computation requested but doParallel not available")
}else{
if(isnull(getDoParName)){
message(paste0("Parallel computation requested, starting doParallel with ",parallel," cores))
registerDoParallel(cores=parallel)
}
} For Exporting Random seeds are quite difficult and work very differently in the different backends. Moreover, there is some interaction between the random seed for the master and the seeds for the workers (even with setting the worker random seed, you get different results for different master random seeds). Progress bars will be difficult no matter what -- |
Thinking about this more: I think |
Is there a simple way in which users can make use of multiple cores as long as parallel support isn't part of simr? |
In my fork, there is some parallel support, but until it's integrated with upstream, the API is subject to change. There's also support for testing multiple coefficients / terms per simulation iteration, which is a huge boost for larger models.
I haven't had a chance to work on it in a while, but if somebody else is using it and thus "helping" work out any kinks/bugs/awkwardness, even just by using it on additional datasets, then that would certainly help move development along....
Finally, you can abuse simr a bit and extract the results of individual simulations and then compute your power (including CIs) from that. So that means you can run n simr runs of m/n iterations simultaneously on m cores to compute m simulated iterations. There's two things to watch out for here:
1. Make sure to set your random seed appropriately so that you don't have m identical copies of the n runs, which would be misleading.
2. Parallel operation on a mulitcore machine may not speed things up as much as you expect if your BLAS is properly tuned for mulitcore operations because the parallel instances then compete with each other. (Now, if you have a cluster then you can easily do both levels of parallel operation with SNOW.)
The biggest boost I've seen for my stuff comes from the optimised omnibus testing for large models in my fork.There does however seem to be a weird interaction with some of that code and certain lme4 versions though that kills performance, at least on Linux. As far as I can tell, it's something related to memory allocation. But again, if someone is willing to use the code on their own stuff and report any issues, this will certainly help me fix them!
|
Hi @palday thanks for your input. This is really useful. One clarification question, though, because I'm not an expert for BLAS: If the calculations can be parallelized on the level of BLAS, why is parallel support needed in simr at all? |
You're starting to grasp why this project hasn't been high priority for me. :-)
Parallel support would still be useful for clusters and single machines with many processors, e. g. those with multiple processors, each with multiple cores. This all ties into issues with memory performance and locality, which is a much larger topic.
Using a good BLAS will also speed up many other things in R.
The default BLAS on macOS is pretty good. Revolution R is a bit controversial as a commercial enterprise, but their version of R is built against the Intel MKL, which is perhaps the fastest BLAS for Intel processors. On Linux, I find OpenBLAS to be quite good, but you need to make sure that R is using it via update-alternatives or otherwise setting your library path.
|
Clusters -- of course. Thanks again. This info is going to be useful for a lot of people in my research group. |
@pitakakariki @palday @tmalsburg ; Given the discussion above, I would like to share some example code, which perhaps could be useful when considering how to implement parallelization in simr. Below is an example illustrating the solution I used, which also shows the speedup on my MacBook Pro. suppressMessages({library(lme4)
library(simr)
library(future)
library(future.apply)
library(parsimr)
library(purrr) # for map()
library(Hmisc) # for binconf()
library(dplyr) # For tibble, %>%, ...
})
# Number of simulations
nsim <- 100L
# Example mixed model
fm1 <- lmer(Reaction ~ Days + (Days | Subject), sleepstudy)
fixef(fm1)
#> (Intercept) Days
#> 251.40510 10.46729
fixef(fm1)["Days"] <- 10
# User powerSim function
system.time(test0 <- powerSim(fm1, nsim = nsim, progress = FALSE))
#> singular fit
#> user system elapsed
#> 12.325 0.255 12.594
# Function for repeatedly calling powerSim with nsim = 1
ps_function <- function(...) powerSim(..., nsim = 1)
# Run in sequential using future
plan(sequential)
system.time(test1 <- future_replicate(nsim, ps_function(fit = fm1, progress = FALSE), simplify = FALSE))
#> user system elapsed
#> 12.903 0.217 13.126
# Run in parallel using future
plan(multiprocess)
system.time(test2 <- future_replicate(nsim, ps_function(fit = fm1, progress = FALSE), simplify = FALSE))
#> singular fit
#> user system elapsed
#> 0.106 0.088 2.985
# test0 contains one powerSim object.
# test1 and test2 contain lists of powerSim objects.
# These must be combined in order to get the right results
# Here is what we get for test0
summary(test0)
#> successes trials mean lower upper
#> 1 100 100 1 0.9637833 1
# Let us reproduce it
# Set significance threshold
alpha <- 0.05
# Extract p-values
pvals <- map_dbl(test2, ~ .x$pval)
# Get the confidence interval
tibble(
successes = sum(pvals < alpha),
trials = length(pvals),
mean = successes / trials
) %>%
bind_cols(as_tibble(matrix(binconf(sum(.$successes), n = length(pvals), method = "exact")[, c("Lower", "Upper")],
nrow = 1, dimnames = list(NULL, c("Lower", "Upper")))))
#> # A tibble: 1 x 5
#> successes trials mean Lower Upper
#> <int> <int> <dbl> <dbl> <dbl>
#> 1 100 100 1 0.964 1 Created on 2019-02-18 by the reprex package (v0.2.1) |
I had also recently run across the |
A bit of indirection might help, but would require more effort to implement: the call |
|
So @osorensen solution using future is very interesting and helpful. However, I keep getting a weird error that I have no idea why it arises. Any insights are most welcome. I am fitting the following model model.of.interest.1 <- glmer(Y ~ X * Z + (1 | G), offset = log(offset), And then following @osorensen suggestion on using the package future number of simulationsnsim <- 1000L Function for repeatedly calling powerSim with nsim = 1ps_function <- function(...) powerSim(..., nsim = 1) plan(multisession) And the interesting thing is that all the powerSim in my list test1 compute a power of 0 and return the following error
Any idea why it is not finding the dataset df2 to simulate over? Thank you |
Haven't done much parallel work but I think |
@pitakakariki is correct. @lesegura lme4 creates a reference to the dataset in question using the magic of "environments" in R, so that it has a local copy that only actually copies if there is a change to the original dataset. But this type of reference isn't passed cleanly when passing the fitted model, which is why manipulations of that model in a future fail. I had worked on parallelizing this a while back and had a branch in my fork, but I haven't updated it in a while (and don't do much R development these days because I've been focused on MixedModels.jl, where oddly enough, I've again been involved in parallelizing code ....). Maybe there's a low effort way I can get these changes back into the main line of |
Thanks @palday. Going to link to https://github.com/palday/simr here so I know where to look when I make time for this. simr is well overdue for an overhaul, and I'm going to request a substantial amount of time to dedicate to this next FY. |
Just wondering: is parallelization still under development? |
Not under active development - it's reasonably high on the "wish list" but I wouldn't want to hazard a guess to when I'll find the time. |
Thank you for the quick response. Fwiw, if there are others with similar questions, here's what I did to work around that as a quick hack for my specific GLMM analysis:
|
Hi all, All the best,
|
Hi @victorshiramizu, I'll have a proper look soon-ish. I haven't used futures before (I like the look of it though) but with that caveat my guess is that you're right and If you don't use Edit: that shouldn't explain different sample sizes giving the same power though ... did you intend to use |
Looks like there are three issues:
|
There's a bunch of things that will have to be sorted out to get polished parallel support.
parallel=TRUE
and it should Just Work.parallel=TRUE
will have to choose a default number of cores somehow.parallel=n
?parallel=cl
?simr
in this casesimr
in each worker will use the installed version, not the version we're working on (will this mess withtestthat
too?)lme4
versus other packages and user defined functions.The text was updated successfully, but these errors were encountered: