parallel::clusterEvalQ(cl, Sys.sleep(1.0)) stalls on WSL2

[mhahsler]

This is quite exciting and might also be useful for my seriation package. I have tried some simple futurize examples on WSL2. Here is an example:

library(futurize)
plan(multisession) 

slow_fcn <- function(x) {
     Sys.sleep(0.1)  # emulate work
     x^2
}

xs <- 1:1000
ys <- lapply(xs, slow_fcn) |> futurize()

It first spawns some R-session instances, but then hangs and does not finish the job. It falls back to a single process with little CPU utilization, but does not finish. After esc/CTRL-C I get:

Warning: Caught FutureInterruptError. Canceling all iterations ...
Error: future_lapply() interrupted at 2026-03-01T10:36:49, while running on ‘25-11427’ (pid 195644)

doparallel works on WSL2. Any ideas.

Version used:

wsl --version
WSL version: 2.6.3.0
Kernel version: 6.6.87.2-1
WSLg version: 1.0.71
MSRDC version: 1.2.6353
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26100.1-240331-1435.ge-release
Windows version: 10.0.22631.6649

> sessionInfo()
R version 4.5.2 (2025-10-31)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 24.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

time zone: America/Chicago
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] futurize_0.1.0 future_1.68.0  TSP_1.2.6     

loaded via a namespace (and not attached):
 [1] digest_0.6.39     codetools_0.2-20  doParallel_1.0.17 iterators_1.0.14  parallel_4.5.2    cli_3.6.5         foreach_1.5.2     parallelly_1.46.0
 [9] pkgload_1.4.1     compiler_4.5.2    globals_0.18.0    rstudioapi_0.17.1 tools_4.5.2       listenv_0.10.0    rlang_1.1.6      

[HenrikBengtsson]

Thanks for bringing this over here. I completely missed this issue when I continued our thread over at mhahsler/TSP#22 (comment):

You also wrote (mhahsler/TSP#22 (comment)):

Yes, plan(multisession, workers = 2) works for me as well.

If I use the default plan(multisession) as used in some of the examples, it consistently hangs for me.

That's very useful information. How many CPU cores does your WSL2 system have/see, e.g. what does the following report?

> parallelly::availableCores(which = "all")

One hypothesis now is that it's over-parallelizing, but still, that should not cause it to freeze.

Looking at your sessionInfo(), I noticed a few things:

• It does not reflect the code you ran, e.g. it's missing future.apply and it lists many other packages not used, e.g. foreach and doParallel.

• You're using an outdated version of future - I don't think it matters, but just in case, could you please make sure to update all your packages.

• It lists rstudioapi, which suggests your running this in the RStudio Console. To rule out RStudio here, could you please retry with plan(multisession) running R directly from the terminal (not the RStudio GUI)?

[mhahsler]

> parallelly::availableCores(which = "all")
           system /proc/self/status             nproc 
               14                14                14  

When I use

library(futurize)
plan(multisession, workers = 14)
ys <- sapply(1:3, identity) |> futurize()
print(ys)

Then it hangs as well. It seems to work fine up to 8 workers.

[HenrikBengtsson]

Informative, but really odd. Did you run this in RStudio or from the terminal?

FYI, plan(multisession) is basically equivalent to:

nworkers <- parallelly::availableCores()
cl <- parallelly::makeClusterPSOCK(nworkers)
plan(cluster, workers = cl)

which is nearly the same as:

nworkers <- parallelly::availableCores()
cl <- parallel::makeCluster(nworkers)
plan(cluster, workers = cl)

The corresponding doParallel setup would be:

nworkers <- parallelly::availableCores()
cl <- parallel::makeCluster(nworkers)
doParallel::registerDoParallel(cl)

You're saying "doparallel works on WSL2", but it's not clear to me exactly how you set it up and with how many workers. doParallel is actually providing two completely different types on parallel workers. Knowing the details will help narrow in on the cause - I suspect your problem can be reproduced with vanilla parallel code, but hard to tell right now.

PS. Please try to update your R packages - it helps to rule out stuff.

[mhahsler]

I tried it using both. I wonder if it is some configuration issue on my machine only.

[HenrikBengtsson]

I see. It's really hard for me to help out any further here, if I don't fully understand your set up (e.g. if you use RStudio or vanilla R, if your packages are up-to-date, and exactly how you use registerDoParallel()). Seeing exact code is more or less essential.

Another thing to try is with forked workers, e.g.

library(futurize)
plan(multicore) ## forked workers

slow_fcn <- function(x) {
     Sys.sleep(0.1)  # emulate work
     x^2
}

xs <- 1:1000
ys <- lapply(xs, slow_fcn) |> futurize()

There's also plan(future.mirai::mirai_multisession) and plan(future.callr::callr) to try and see if the problem is there too, or specific to the multisession backend. Knowing will rule out a lot of things.

One can also see if it can be reproduced without futurize (which I doubt matters here) and future.apply (which I give a slight probability to be involved) and see if it is happens with just future, e.g.

library(future)
plan(multisession)
futureSessionInfo()

library(future)
plan(multisession, workers = 8)
futureSessionInfo()

library(future)
plan(multisession, workers = 9)
futureSessionInfo()

[mhahsler]

I think the issue is that setting up the session gets stuck. I get:

$ R --vanilla

R version 4.5.2 (2025-10-31) -- "[Not] Part in a Rumble"
Copyright (C) 2025 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(future)
plan(multisession)
futureSessionInfo()
*** Package versions
future 1.69.0, parallelly 1.46.1, parallel 4.5.2, globals 0.19.0, listenv 0.10.0

*** Allocations
availableCores():
           system /proc/self/status             nproc
               14                14                14
availableWorkers():
$nproc
 [1] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
 [7] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
[13] "localhost" "localhost"

$system
 [1] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
 [7] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
[13] "localhost" "localhost"


*** Settings
- future.plan=<not set>
- future.fork.multithreading.enable=<not set>
- future.globals.maxSize=<not set>
- future.globals.onReference=<not set>
- future.resolve.recursive=<not set>
- future.rng.onMisuse=<not set>
- future.wait.timeout=<not set>
- future.wait.interval=<not set>
- future.wait.alpha=<not set>
- future.startup.script=<not set>

*** Backends
Number of workers: 14
List of future strategies:
1. multisession:
   - args: function (..., workers = availableCores(constraints = "connections-16"), rscript_libs = .libPaths())
   - tweaked: FALSE
   - call: plan(multisession)

*** Basic tests
Main R session details:
    pid     r sysname                          release
1 23931 4.5.2   Linux 6.6.87.2-microsoft-standard-WSL2
                                              version nodename machine   login
1 futureverse/futurize#1 SMP PREEMPT_DYNAMIC Thu Jun  5 18:30:46 UTC 2025  host001  x86_64 user002
     user effective_user
1 user001        user001

It stops right there and does not report the other nodes. I waited for more than a minute.

It works reliably with 2 or 4 workers and sometimes with 6.

> library(future)
plan(multisession, workers = 4)
futureSessionInfo()
*** Package versions
future 1.69.0, parallelly 1.46.1, parallel 4.5.2, globals 0.19.0, listenv 0.10.0

*** Allocations
availableCores():
           system /proc/self/status             nproc
               14                14                14
availableWorkers():
$nproc
 [1] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
 [7] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
[13] "localhost" "localhost"

$system
 [1] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
 [7] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
[13] "localhost" "localhost"


*** Settings
- future.plan=<not set>
- future.fork.multithreading.enable=<not set>
- future.globals.maxSize=<not set>
- future.globals.onReference=<not set>
- future.resolve.recursive=<not set>
- future.rng.onMisuse=<not set>
- future.wait.timeout=<not set>
- future.wait.interval=<not set>
- future.wait.alpha=<not set>
- future.startup.script=<not set>

*** Backends
Number of workers: 4
List of future strategies:
1. multisession:
   - args: function (..., workers = 4)
   - tweaked: TRUE
   - call: plan(multisession, workers = 4)

*** Basic tests
Main R session details:
    pid     r sysname                          release
1 28373 4.5.2   Linux 6.6.87.2-microsoft-standard-WSL2
                                              version nodename machine   login
1 futureverse/futurize#1 SMP PREEMPT_DYNAMIC Thu Jun  5 18:30:46 UTC 2025  host001  x86_64 user002
     user effective_user
1 user001        user001
Worker R session details:
  worker   pid     r sysname                          release
1      1 28441 4.5.2   Linux 6.6.87.2-microsoft-standard-WSL2
2      2 28440 4.5.2   Linux 6.6.87.2-microsoft-standard-WSL2
3      3 28442 4.5.2   Linux 6.6.87.2-microsoft-standard-WSL2
4      4 28443 4.5.2   Linux 6.6.87.2-microsoft-standard-WSL2
                                              version nodename machine   login
1 futureverse/futurize#1 SMP PREEMPT_DYNAMIC Thu Jun  5 18:30:46 UTC 2025  host001  x86_64 user002
2 futureverse/futurize#1 SMP PREEMPT_DYNAMIC Thu Jun  5 18:30:46 UTC 2025  host001  x86_64 user002
3 futureverse/futurize#1 SMP PREEMPT_DYNAMIC Thu Jun  5 18:30:46 UTC 2025  host001  x86_64 user002
4 futureverse/futurize#1 SMP PREEMPT_DYNAMIC Thu Jun  5 18:30:46 UTC 2025  host001  x86_64 user002
     user effective_user
1 user001        user001
2 user001        user001
3 user001        user001
4 user001        user001
Number of unique worker PIDs: 4 (as expected)

Anything more, and it gets stuck. Maybe a weird interaction between the Windows host and the Ubuntu WSL2 instance?

-Michael

[mhahsler]

Btw: plan(future.callr::callr) and plan(multicore) work and create 14 workers.

[HenrikBengtsson]

Thanks. This is all very helpful and definitely a big step forward.

plan(multisession) uses a PSOCK cluster of the parallel package. Such clusters rely on socket connections for the main R process to communicate with the parallel workers. Given that plan(future.callr::callr) and plan(multicore) work, it could be that there is something conflicting the R connections (e.g. firewall rules, but shouldn't happen under Linux WSL2).

We've now excluded both futurize and future.apply from the equation (and RStudio). I suspect that this is also independent of the future package, but before we can rule that out, the next thing to test would be:

library(future)
cl <- parallel::makeCluster(14)  ## Set up 14 vanilla PSOCK cluster workers
print(cl)
plan(cluster, workers = cl)
futureSessionInfo()

What is the output and does that also freeze? If not, what about:

library(future)
cl <- parallel::makeCluster(14, type = parallelly::RPSOCK)  ## Set up 14 "enhanced" PSOCK cluster workers
print(cl)
plan(cluster, workers = cl)
futureSessionInfo()

[mhahsler]

Both versions freeze when I request more than 2 workers. It looks like the connections are open.

> library(future)
cl <- parallel::makeCluster(14)  ## Set up 14 vanilla PSOCK cluster workers
print(cl)
socket cluster with 14 nodes on host ‘localhost’
> showConnections()
   description         class      mode  text     isopen   can read can write
4  "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"
5  "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"
6  "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"
7  "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"
8  "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"
9  "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"
10 "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"
11 "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"
12 "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"
13 "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"
14 "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"
15 "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"
16 "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"
17 "<-localhost:11202" "sockconn" "a+b" "binary" "opened" "yes"    "yes"

but then futureSessionInfo() hangs.

[HenrikBengtsson]

Perfect. To peel off more, next would be to see if we can reproduce it with manually created futures; does the following also hang?

library(future)
cl <- parallel::makeCluster(14)
plan(cluster, workers = cl)

fs <- lapply(seq_along(cl), FUN = function(ii) {
  future({
    Sys.sleep(1.0)
    data.frame(worker = ii, pid = Sys.getpid(), r = getRversion(), as.list(Sys.info()))
  })
})
vs <- value(fs)
str(vs)

If so, what about:

library(future)
cl <- parallel::makeCluster(14)
plan(cluster, workers = cl)

fs <- lapply(seq_along(cl), FUN = function(ii) {
  future({
    Sys.sleep(1.0)
    data.frame(worker = ii, pid = Sys.getpid(), r = getRversion(), as.list(Sys.info()))
  }, stdout = FALSE, conditions = character(0), globals = list(ii = ii))
})
vs <- value(fs)
str(vs)

?

[mhahsler]

both freeze. Using parallel directly (clusterApply) works as expected:

cl <- parallel::makeCluster(14)

fs <- parallel::clusterApply(cl, seq_along(cl), function(ii) {
    data.frame(worker = ii, pid = Sys.getpid(), r = getRversion(), as.list(Sys.info()))
  })
vs <- value(fs)
str(vs)

However, if I add a Sys.sleep() to the parallelized function, the processes never seem to wake up, and it hangs. Looks like there is an issue with how nanosleep() interacts in WSL2.

[HenrikBengtsson]

Wow, interesting. There might be a bug in R here around how Sys.sleep() is implemented. Let's see.

To rule out more things, does the following also freeze on you?

cl <- parallel::makeCluster(14)
y <- parallel::clusterApply(cl, seq_along(cl), function(ii) {
    Sys.sleep(1.0)
})
str(y)

PS. You don't need vs <- value(fs) here or in your latest example - that's only for futures.

If that freezes, what about

cl <- parallel::makeCluster(14)
y <- parallel::clusterEvalQ(cl, Sys.sleep(1.0))
str(y)

?

BTW, what you found means that your original ys <- lapply(xs, slow_fcn) |> futurize() should work if you drop Sys.sleep() call - is that the case?

[HenrikBengtsson]

Hello, I haven't heard back from you here. Did you resolve this?

[mhahsler]

Hi, I was traveling and had no time to look into this. I wonder if it is related to a weird WSL2 setup that my university uses. I think we need to see if someone with a vanilla installation of Windows/WSL2 has the same issues.

[HenrikBengtsson]

Yes, maybe. But if you could try what I wrote in my last comment, then we can narrow in on the problem further, which then simplifies asking others in your organization to check what they get. So, first goal is to identify a minimal reproducible example.