Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BACKEND: ClusterMQ as a new backend #204

Open
alexvorobiev opened this issue Mar 10, 2018 · 9 comments
Open

BACKEND: ClusterMQ as a new backend #204

alexvorobiev opened this issue Mar 10, 2018 · 9 comments

Comments

@alexvorobiev
Copy link

@alexvorobiev alexvorobiev commented Mar 10, 2018

I have recently discovered ClusterMQ which can run R code in SLURM/LSF/etc. jobs. The biggest advantage over batchtools is it uses ZMQ to transfer data directly to the distributed jobs. In my experience the most serious bottleneck in batchtools is using shared file system (NFS) for data transfer - especially if the data is large.

@HenrikBengtsson
Copy link
Owner

@HenrikBengtsson HenrikBengtsson commented Mar 10, 2018

Yes, @mschubert's ClusterMQ is a great candidate for a future backend. I don't have the resources myself right now to work also on that. Having said that, and without having worked with ClusterMQ myself, I don't think it should be too much work to wrap it all up in a ClusterMQFuture - a future backend is mostly a thin layer on top of an existing API.

Related: I'm working on setting up a conformation test suite (e.g. future.tests pkg) that can be used by all future backend pkgs to make sure they got it correct. That is my number one priority before working on new backends.

@mschubert
Copy link

@mschubert mschubert commented Mar 10, 2018

I fully support this, but unfortunately my time is also quite limited these days.

@HenrikBengtsson HenrikBengtsson changed the title ClusterMQ as a new backend BACKEND: ClusterMQ as a new backend Mar 11, 2018
@wlandau
Copy link

@wlandau wlandau commented Jun 28, 2018

Given that clustermq::Q() is synchronous, I am wondering what it would take to make an asynchronous ClusterMQFuture. Do we need local background processes to collect the results?

@wlandau
Copy link

@wlandau wlandau commented Dec 12, 2019

Will future.clustermq somehow allow for heterogeneous transient workers? Some drake users such as @jennysjaarda prefer transient future-based workers over persistent clustermq-based workers, e.g. ropensci/drake#1083 (comment), but there is still the snag that batchtools is slower than clustermq.

@jennysjaarda
Copy link

@jennysjaarda jennysjaarda commented Dec 17, 2019

Yes, this would be great if it somehow clustermq could allow for transient workers!

@wds15
Copy link

@wds15 wds15 commented Aug 12, 2020

May I ask what the status of the backend is? Is it still planned to include clustermq as a backend to future or is there already a way to get that functionality via some workaround.

clustermq is quite a bit more efficient as pointed out in this thread and is thus very interesting for cluster usage.

@HenrikBengtsson
Copy link
Owner

@HenrikBengtsson HenrikBengtsson commented Aug 12, 2020

Still on my wishlist to get to, so, yes, certainly on the todo list. Resources/time is the limiting factor. Indirectly, a big step forward has actually been made since automatic validation of new backends is now in place, cf. future.tests.

PS. I invite anyone to have a look at the very rudimentary first prototype future.clustermq and see if they can give it a push forward (PRs welcome).

@mschubert
Copy link

@mschubert mschubert commented Oct 14, 2020

@HenrikBengtsson When I wanted to check out the future.clustermq link I got a 404.

Is this still set to private?

@HenrikBengtsson
Copy link
Owner

@HenrikBengtsson HenrikBengtsson commented Oct 16, 2020

It would be awesome to get this going. I just made it public, but please note that it's very rudimentary/prototypical and I have not touched it for a very long time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants