-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
QUESTION: is SSH + BSUB (LSF) available? #105
Comments
This is not (yet) possible. Sending commands to a remote system is no problem, |
Short comment regarding sshfs: the few times I used this to ensure a shared fs when there was none I was surprised how well this worked. I had no problems, just saying this if somebody needa a workaround. |
@gabora: is the main reason you want to do this because of the editor? R studio? |
Because years ago I was in the same situation before I began using vim. For me the main hurdle was to avoid manually copying files from the local machine to the server. So when you use sshfs here, in a different way, to mount your server project code dir on your local machine, not many problems remain
Does this help? I like using vim on the server more now. But this approach was really totally OK for me some time ago. |
Further side note: Getting reliable exit codes and output for remote commands seems to be possible with the subprocess package. Unfortunately, C++11 is a requirement. |
Note that I implemented something like that on a ZeroMQ-backend (clustermq) back in the day when I was frustrated with The main difference to It also supports sending remote jobs via SSH (so first SSH, then the job submission system). Downside is that it relies on the SSH forwarding not getting disconnected while the jobs run, which is good enough for my purposes but may not be for everyone. |
Thanks a lot for the reply @mllg @berndbischl and @mschubert . I used @mschubert 's clustermq, which works really well, but we had problem with long runs. @mllg thanks for considering for possible implementation!
I will try |
Sorry if this is a naive questions, but I'm still new to Background: |
This was already implemented for Slurm, and now there is a prototype for SGE. It is kind of buggy though, getting the quoting right is not straight forward. |
NB: LSF's bsub needs the template provided via STDIN instead of a file. I currently see no way to accomplish this in a reliable way with SSH. |
I'm not sure exactly what you mean by "getting the quoting right". Are there any guidelines/docs to help users avoid problems? |
This was just a comment on the implementation ...
Not yet. I still have to write it down in full detail, but here are the required steps in a TLDR style:
|
I'm quite close to making this work, but it seems like there is a path being expanded in the slurm SSH submission still. I get the following error:
where The setup is as follows:
Any suggestions on where to look? The template seems to be generating a reasonable job file with a relative path for the Rscript call. Thanks. |
Hi,
I am curious if I could use
batchtools
on my local machine and submit jobs through SSH (usingBSUB
with LSF queueing system) on a remote cluster?This would be a combination of
clusterFunctionLSF
andclusterFunctionSSH
, but I haven't found such thing implemented yet, right?I really would like to work on my local machine (in Rstudio) and do the computation on the cluster.
ClusterFunctionSSH
is not a solution, because I am not allowed to do computation/memory heavy tasks on the front end (as far as I understoodclusterFunctionsSSH
together with theworker
does not use the queue system, but I might missed something here. ).Looking forward to your suggestion,
Thanks and Kind Regards,
Attila
The text was updated successfully, but these errors were encountered: