Question on Connecting to Remote Servers for External R Processing

[danielra0091]

So I am having trouble wrapping my head around using the future package to connect to a remote linux server. Couldn't find much documentation to help me troubleshoot this error I am getting.

Error in socketConnection("localhost", port = port, server = TRUE, blocking = TRUE,  : 
  reached elapsed time limit

library("plotly")
library("future")
library("microbenchmark")

remote_ip <- "192.168.43.255"

## %<-% assignments will be resolved remotely
plan(remote, workers = remote_ip)

## Set up data (locally)
set.seed(100)
d <- diamonds[sample(nrow(diamonds), 1000), ]

## Generate ggplot2 graphics and plotly-fy (remotely)
gg %<-% {
p <- ggplot(data = d, aes(x = carat, y = price)) +
  geom_point(aes(text = paste("Clarity:", clarity)), size = 4) +
  geom_smooth(aes(colour = cut, fill = cut)) + facet_wrap(~ cut)
ggplotly(p)
}

## Display graphics in browser (locally)
gg <- microbenchmark(gg)
gg

[HenrikBengtsson]

For a starter, what is your operating system and your sessionInfo()?

[danielra0091]

My OS is Windows 10, server OS is Linux Ubuntu server 18.04 (Bionic Beaver)

sessionInfo():

R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] ssh_0.2

loaded via a namespace (and not attached):
[1] compiler_3.5.1 tools_3.5.1    yaml_2.2.0    

[HenrikBengtsson]

Ok, your on Windows. I don't see session info on the future package - please make sure your show that after loading future.

Can you SSH into the remote machine and launch Rscript? That is a minimum requirement. See ?future::makeClusterPSOCK for help and examples - that is the function used internally when setting up the connection to the remote machine.

[danielra0091]

I do have the ability to make those connections. I am testing the code on a virtual linux server to see how Future package works. It more HOW to establish those connections is where I am getting confused on. But I will follow your suggestion on ?future::makeClusterPSOCK and hopefully I can understand it lol.

R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] promises_1.0.1 future_1.9.0   ssh_0.2       

loaded via a namespace (and not attached):
 [1] compiler_3.5.1   magrittr_1.5     R6_2.2.2         later_0.7.3      parallel_3.5.1   tools_3.5.1     
 [7] listenv_0.7.0    yaml_2.2.0       Rcpp_0.12.18     codetools_0.2-15 digest_0.6.15    globals_0.12.1  
> 

[HenrikBengtsson]

I do have the ability to make those connections. I am testing the code on a virtual linux server to see how Future package works. It more HOW to establish those connections is where I am getting confused on.

Should I read this as you can indeed connect to your remote machine using cl <- makeClusterPSOCK(...) by using for instance PuTTY as illustrated in example("makeClusterPSOCK")?

If so, then just use:

plan(remote, workers = cl)

or, equivalently,

plan(cluster, workers = cl, persistent = TRUE)

Validate that future expressions are evaluated remotely by:

info %<-% Sys.info()
print(info)

[danielra0091]

Sorry I should have clarified, I have made the connection by using the putty application outside of Rstudio. Where I am having the trouble, is establishing the same connection in R with makeClusterPSOCK(...). I seem to not be putting in the correction information or I am missing something.

Also, I am trying to following the examples in makeClusterPSOCK() but can't seem to follow them. I think where I am lost is the SSH connection. Do I need to relay it through Putty? or can i do that all within R?

IP: 192.168.1.53
User: danielra0091@danny_remote_server
port:22

[HenrikBengtsson]

It's hard to give much more suggestions without knowing exactly what you've tried. See my suggestion on how to troubleshoot a similar problem over at futureverse/future#242 (comment) - that setup more complex that yours, but the gist of how to troubleshoot is them same. Using makeClusterPSOCK(..., verbose = TRUE) is your friend - it helps clarifying what's going on under the hood.

[...] Do I need to relay it through Putty? or can i do that all within R?

Yes, future::makeClusterPSOCK(), which is just a more powerful version of parallel::makePSOCKcluster(), requires an external SSH client that it calls via base::system(). I noticed that you had the 'ssh' package loaded, which I guess is related to this question - no, there is no support for using that package.

FYI, Windows 10 version 1803 (April 2018) and newer has an ssh client built-in that works the same as on Linux and macOS. The future::makeClusterPSOCK() function will find this if it is available. If you can use that, you don't have to worry about getting the PuTTY call correct.

[danielra0091]

So here is the code i ran and the message associated with it. Not entirely sure whether this was correct or not.

workers <- "192.168.1.52"
cl <- makeClusterPSOCK(
  workers, user = "danielra0091",port = 22, rshcmd = c("C:/Program Files/PuTTY/plink.exe", "-ssh"),
  dryrun = TRUE
)

----------------------------------------------------------------------
Manually start worker HenrikBengtsson/future#1 on ‘192.168.1.52’ with:
  "Rscript" --default-packages=datasets,utils,grDevices,graphics,stats,methods -e "parallel:::.slaveRSOCK()" MASTER=localhost PORT=22 OUT=/dev/null TIMEOUT=2592000 XDR=TRUE
Alternatively, start it from the local machine with:
  "C:/Program Files/PuTTY/plink.exe" "-ssh" -R 22:localhost:22 -l danielra0091 192.168.1.52 "\"Rscript\" --default-packages=datasets,utils,grDevices,graphics,stats,methods -e \"parallel:::.slaveRSOCK()\" MASTER=localhost PORT=22 OUT=/dev/null TIMEOUT=2592000 XDR=TRUE"

So I am brand new to SSH, private /public keys, moving them to servers etc etc. So I try to follow some examples but get lost in the meaning of how the PATHs are established or when dummy data is used as opposed to real data (i.e. fake ips, fake paths)

If it is possible, I would be more then happy to speak online via skype where I can share my screen or even a phone call to discuss further. I think my lack of experience in this, makes troubleshooting difficult since I can't explain well the problems I am experiencing.

email: danielra0091@gmail.com

[HenrikBengtsson]

Note that argument port is not specifying the port using by SSH, but the port used by the R master and the R worker to communicate. Specifying port=22 could be a problem. Try without specifying the port such that you instead get a random port assigned.

If that doesn't solve it, make sure to try HenrikBengtsson/future#242 (comment)

[danielra0091]

Will do, also I added a paragraph to my last comment about reaching out directly with my email. If you have the time or are willing. It will be much appreciated.

[danielra0091]

So here is the new error I have now. I am unsure how to complete this code correctly to get it working.

> system('"C:/Program Files/PuTTY/plink.exe" "-ssh" -R 11888:localhost:11888 -l root  -i C:/Users/danie/Desktop/RCore Keys/id_rsa 192.168.43.94 date')
Unable to open connection:
Host does not exist

[HenrikBengtsson]

It looks like it cannot reach 192.168.43.94. Verify by simplifying:

$ system('"C:/Program Files/PuTTY/plink.exe" -ssh 192.168.43.94')

If that doesn't work, try the same at the Windows Command line ("DOS"):

"C:/Program Files/PuTTY/plink.exe" -ssh 192.168.43.94

If that doesn't work either, you've got the incorrect IP number or don't have access.

[HenrikBengtsson]

@danielra0091, any progress on your end? Can I close this issue?

[danielra0091]

Still working to resolve it, had to switch gears to another issue but I will be coming back to this soon. Was having issues logging in to the server now. But will try a few more attempts before reaching out again.
Thank you!

[danielra0091]

Here is the latest progress I have had on getting the connection to work. I seem to be having trouble with the RSA file.

system('"C:/Program Files/PuTTY/plink.exe" -ssh 192.168.1.52 -l danielra0091 -i C:/Users/danie/Desktop/id_rsa')

Result:

[danielra0091@192.168.1.52's password: 
Unable to use key file "C:/Users/danie/Desktop/id_rsa" (OpenSSH SSH-2 private key (old PEM format))
Using username "danielra0091"

[HenrikBengtsson]

Please see ?future:makeClusterPSOCK on using PuTTY. The text and the example show how to use a *.ppk file - your code snippet does not use that. It might be that there is another way, but the one I verified when I wrote the help was using a *.ppk file.

Otherwise, to troubleshoot this, I suggest that you try to figure out what it should look like without using R, i.e. use the Windows Command line ("DOS"):

"C:/Program Files/PuTTY/plink.exe" -ssh 192.168.1.52 -l danielra0091 -i C:/Users/danie/Desktop/id_rsa
[danielra0091@192.168.1.52's password: 
Unable to use key file "C:/Users/danie/Desktop/id_rsa" (OpenSSH SSH-2 private key (old PEM format))
Using username "danielra0091"

Look at the PuTTY docs or query at the PuTTY forums for a solution to using SSH keys. When you've got that working, then it should work from within R.

[danielra0091]

Hello, So I am getting an notification to manually start my worker.

library(future)

workers <- "proxy21.rt3.io"
cl <- makeClusterPSOCK(workers,
 user = "pi", manual = TRUE,
#revtunnel = FALSE, 
  dryrun = TRUE ) #trusted_connection = true)
cl

Manually start worker HenrikBengtsson/future#1 on ‘proxy21.rt3.io’ with:
  "Rscript" --default-packages=datasets,utils,grDevices,graphics,stats,methods -e "parallel:::.slaveRSOCK()" MASTER=localhost PORT=11799 OUT=/dev/null TIMEOUT=2592000 XDR=TRUE
Alternatively, start it from the local machine with:
  "C:\WINDOWS\System32\OpenSSH\ssh.exe" -R 11799:localhost:11799 -l pi proxy21.rt3.io "\"Rscript\" --default-packages=datasets,utils,grDevices,graphics,stats,methods -e \"parallel:::.slaveRSOCK()\" MASTER=localhost PORT=11799 OUT=/dev/null TIMEOUT=2592000 XDR=TRUE"
> cl
socket cluster with 1 nodes on host ‘NULL’
> 

Not sure where to go from here in terms of establishing my connection. "proxy21.rt3.io" is a proxy to connect to a raspberry pi running R that has future packaged installed and ssh installed as well.

[HenrikBengtsson]

Ok. Let's forget about R on your local Windows machine for now. What happens if you call the following from the Windows Console/Terminal:

C:\Users\alice> "C:\WINDOWS\System32\OpenSSH\ssh.exe" -l pi proxy21.rt3.io Rscript --version

and if that works, what about:

C:\Users\alice> "C:\WINDOWS\System32\OpenSSH\ssh.exe" -R 11799:localhost:11799 -l pi proxy21.rt3.io Rscript --version

(Before anything else, I want to confirm that you can ssh into your remote machine and that the Rscript executable is available. When that has been established, we can retry from within R.)

[danielra0091]

Here are the results: I was able to establish my connection

C:\Users\danie>"C:\WINDOWS\System32\OpenSSH\ssh.exe" -l pi proxy21.rt3.io  -p 34383 Rscript --version
pi@proxy21.rt3.io's password:
R scripting front-end version 3.3.3 (2017-03-06)

C:\Users\danie>

C:\Users\danie>"C:\WINDOWS\System32\OpenSSH\ssh.exe" -R 11799:localhost:11799 -l pi proxy21.rt3.io -p 34383 Rscript --version
pi@proxy21.rt3.io's password:
R scripting front-end version 3.3.3 (2017-03-06)

C:\Users\danie>

[HenrikBengtsson]

Awesome, that confirms that you can connect to R on the remote machine over SSH. Go back and try the below from within R, which should do the same thing in the background, launch R on the remote machine which will try to connect back to your local R session:

> library(future)
> workers <- "proxy21.rt3.io"
> cl <- makeClusterPSOCK(workers, user = "pi", verbose = TRUE)
> print(cl)

What do you get/see?

BTW, what version of R do you run locally? I see you run R 3.3.3 on the remote machine.

[danielra0091]

Here are the specs of my remote Pi and my computer:

Raspberry Pi 
> version
_
platform       arm-unknown-linux-gnueabihf
arch           arm
os             linux-gnueabihf
system         arm, linux-gnueabihf
status
major          3
minor          3.3
year           2017
month          03
day            06
svn rev        72310
language       R
version.string R version 3.3.3 (2017-03-06)
nickname       Another Canoe

My Computer Specs
> version
platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          5.1                         
year           2018                        
month          07                          
day            02                          
svn rev        74947                       
language       R                           
version.string R version 3.5.1 (2018-07-02)
nickname       Feather Spray

Results from your code:

> library(future)
> workers <- "proxy21.rt3.io"
> cl <- makeClusterPSOCK(workers, user = "pi", verbose = TRUE)
Workers: [n = 1] ‘proxy21.rt3.io’
Base port: 11530
Creating node 1 of 1 ...
- setting up node
Starting worker HenrikBengtsson/future#1 on ‘proxy21.rt3.io’: "C:\WINDOWS\System32\OpenSSH\ssh.exe" -R 11530:localhost:11530 -l pi proxy21.rt3.io "\"Rscript\" --default-packages=datasets,utils,grDevices,graphics,stats,methods -e \"parallel:::.slaveRSOCK()\" MASTER=localhost PORT=11530 OUT=/dev/null TIMEOUT=2592000 XDR=TRUE"
Waiting for worker HenrikBengtsson/future#1 on ‘proxy21.rt3.io’ to connect back
Error in socketConnection("localhost", port = port, server = TRUE, blocking = TRUE,  : 
  reached elapsed time limit
> print(cl)
socket cluster with 1 nodes on host ‘NULL’

When I added the port: ( 34383)

> library(future)
> workers <- "proxy21.rt3.io"
> cl <- makeClusterPSOCK(workers, user = "pi", port = 34383, verbose = TRUE)
Workers: [n = 1] ‘proxy21.rt3.io’
Base port: 34383
Creating node 1 of 1 ...
- setting up node
Starting worker HenrikBengtsson/future#1 on ‘proxy21.rt3.io’: "C:\WINDOWS\System32\OpenSSH\ssh.exe" -R 34383:localhost:34383 -l pi proxy21.rt3.io "\"Rscript\" --default-packages=datasets,utils,grDevices,graphics,stats,methods -e \"parallel:::.slaveRSOCK()\" MASTER=localhost PORT=34383 OUT=/dev/null TIMEOUT=2592000 XDR=TRUE"
Waiting for worker HenrikBengtsson/future#1 on ‘proxy21.rt3.io’ to connect back
Error in socketConnection("localhost", port = port, server = TRUE, blocking = TRUE,  : 
  reached elapsed time limit
> print(cl)
socket cluster with 1 nodes on host ‘NULL’

[HenrikBengtsson]

Hmm... first another quick confirmation. Does it even work to set up a localhost worker?

> library(future)
> cl <- makeClusterPSOCK("localhost", verbose = TRUE)
> print(cl)

I assume so, but want to make sure.

Also, can you update your remote R version, to rule out that you're hitting some bug in the parallel package of R 3.3.3?

[HenrikBengtsson]

Oh, and using argument out = NULL will give you output from the remote worker too, i.e.

> cl <- makeClusterPSOCK(workers, user = "pi", out = NULL, verbose = TRUE)

[danielra0091]

Here is what I get running the code.

Also: I am trying to install the latest R, but its been a bit tricky. Running into some issues.

# Remote Worker Setup test   
> library(future)
> cl <- makeClusterPSOCK("localhost", out = NULL , verbose = TRUE)
Workers: [n = 1] ‘localhost’
Base port: 11998
Creating node 1 of 1 ...
- setting up node
Starting worker HenrikBengtsson/future#1 on ‘localhost’: "C:/PROGRA~1/R/R-3.5.1/bin/x64/Rscript" --default-packages=datasets,utils,grDevices,graphics,stats,methods -e "parallel:::.slaveRSOCK()" MASTER=localhost PORT=11998 OUT= TIMEOUT=2592000 XDR=TRUE
Waiting for worker HenrikBengtsson/future#1 on ‘localhost’ to connect back
Connection with worker HenrikBengtsson/future#1 on ‘localhost’ established
- assigning connection UUID
- collecting session information
Creating node 1 of 1 ... done
> print(cl)
socket cluster with 1 nodes on host ‘localhost’

[HenrikBengtsson]

Good, so it works when you use localhost workers. It's a much simpler use case (avoids SSH and most firewall issues), but at least we have a baseline.

For your remote use case, what is the output when you use out = NULL. It could be that there is an error when launching the remote worker, and using out = NULL could then provide us with some clues.

[danielra0091]

Here are the results: It failed to connect.
Also (cl) just printed the localhost variable. JUST FYI.

> library(future)
>  workers <- "proxy13.remot3.it"
>   cl <- makeClusterPSOCK(workers, user = "pi", out = NULL, verbose = TRUE)
Workers: [n = 1] ‘proxy13.remot3.it’
Base port: 11492
Creating node 1 of 1 ...
- setting up node
Starting worker HenrikBengtsson/future#1 on ‘proxy13.remot3.it’: "C:\WINDOWS\System32\OpenSSH\ssh.exe" -R 11492:localhost:11492 -l pi proxy13.remot3.it "\"Rscript\" --default-packages=datasets,utils,grDevices,graphics,stats,methods -e \"parallel:::.slaveRSOCK()\" MASTER=localhost PORT=11492 OUT= TIMEOUT=2592000 XDR=TRUE"
Waiting for worker HenrikBengtsson/future#1 on ‘proxy13.remot3.it’ to connect back
Error in socketConnection("localhost", port = port, server = TRUE, blocking = TRUE,  : 
  reached elapsed time limit
>   print(cl)
socket cluster with 1 nodes on host ‘localhost’

[HenrikBengtsson]

Interesting, with out = NULL, the lack of a line:

starting worker pid=32690 on localhost:11492 at 11:14:15.120

after Starting worker HenrikBengtsson/future#1 ... and before Error in socketConnection(...) suggests that the R worker on the remote machine does actually not start.

What happens if you log into the remote machine and call:

$ Rscript --default-packages=datasets,utils,grDevices,graphics,stats,methods -e "parallel:::.slaveRSOCK()" MASTER=localhost PORT=11492 OUT= TIMEOUT=2592000 XDR=TRUE
starting worker pid=631 on localhost:11492 at 11:48:19.096

Do you get an error or do you see that starting working pid=... output?

[danielra0091]

Here is the result I get for running the Rscript on my raspberry Pi:
It fails to establish the connection.

pi@raspberrypi:~ $ Rscript --default-packages=datasets,utils,grDevices,graphics,stats,methods -e "parallel:::.slaveRSOCK()" MASTER=localhost PORT=11492 OUT= TIMEOUT=2592000 XDR=TRUE
starting worker pid=1863 on localhost:11492 at 20:08:46.011
Error in socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :
  cannot open the connection
Calls: <Anonymous> ... doTryCatch -> recvData -> makeSOCKmaster -> socketConnection
In addition: Warning message:
In socketConnection(master, port = port, blocking = TRUE, open = "a+b",  :
  localhost:11492 cannot be opened
Execution halted

[HenrikBengtsson]

I've updated the develop version of the future package such that you will get a much more informative error message when it fails to connect. First, in a fresh R session, install it using (enough on your local machine):

> remotes::install_github("HenrikBengtsson/future@develop")

Then, retry, now by adding the new logfile = TRUE argument:

> library(future)
> workers <- "proxy13.remot3.it"
> cl <- makeClusterPSOCK(workers, user = "pi", logfile = TRUE, verbose = TRUE)

You will get a very detailed error message - please show all of the output.

PS. I learned that on Windows when running Rgui or RStudio Console, using outfile = NULL will still not show anything. I will if you run R from the Windows Console/terminal.

PS2. Could it be that you log in to the remote machine using a password? Only SSH-key logins are supported. If so, the new error message should reveal that is what's going on.

[HenrikBengtsson]

After testing on Windows 10 (version 1803, build 17134.523, 2018-07-10 with OpenSSH_for_Windows_7.6p1, LibreSSL 2.6.4), I believe there is a bug in "C:\WINDOWS\System32\OpenSSH\ssh.exe".

It works with PuTTY and with the SSH client that comes with RStudio. See ?makeClusterPSOCK in the develop version for how to use these instead.

UPDATE 2019-01-14: According to PowerShell/Win32-OpenSSH#1265, there have been some fixes in Windows 10 version 1809 (build 17763.253, 2018-11-13 with OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5) for reverse tunneling and some of the examples that didn't work before now work. However, one user PowerShell/Win32-OpenSSH#1265 (comment) still reports (2018-11-16) that it doesn't work. When I test under Windows 10 version 1809, I can confirm that although it no longer stalls, there is something else going on causing failures, e.g.

> cl <- future::makeClusterPSOCK("remote.server.org", outfile=NULL, verbose=TRUE)
Workers: [n = 1] 'remote.server.org'
Base port: 11780
Creating node 1 of 1 ...
- setting up node
Starting worker HenrikBengtsson/future#1 on 'remote.server.org': "C:\Windows\System32\OpenSSH\ssh.exe" -R 11780:localhost:11780 remote.server.org "\"Rscript\" --default-packages=datasets,utils,grDevices,graphics,stats,methods -e \"parallel:::.slaveRSOCK()\" MASTER=localhost PORT=11780 OUT= TIMEOUT=2592000 XDR=TRUE"
Waiting for worker HenrikBengtsson/future#1 on 'remote.server.org' to connect back
starting worker pid=29014 on localhost:11780 at 22:00:56.665
Error in unserialize(node$con) : error reading from connection
Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCK0node -> unserialize
[ ... stalls ...]

Maybe it works with text but not binary streaming?!?

UPDATE 2019-05-26: This error is still there with Windows 10 version 1903 (build 18362.116, 2019-05-23 with OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5), which is probably because SSH hasn't been updated since last time.

[HenrikBengtsson]

Hmm... not sure what happened here; this issue is still reported as being transferred (for two days now) and it's content looks very much Issue #20 but incomplete and different timestamps. Closing.