New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BACKEND: ClusterMQ as a new backend #204

Open
alexvorobiev opened this Issue Mar 10, 2018 · 3 comments

Comments

Projects
None yet
4 participants
@alexvorobiev
Copy link

alexvorobiev commented Mar 10, 2018

I have recently discovered ClusterMQ which can run R code in SLURM/LSF/etc. jobs. The biggest advantage over batchtools is it uses ZMQ to transfer data directly to the distributed jobs. In my experience the most serious bottleneck in batchtools is using shared file system (NFS) for data transfer - especially if the data is large.

@HenrikBengtsson

This comment has been minimized.

Copy link
Owner

HenrikBengtsson commented Mar 10, 2018

Yes, @mschubert's ClusterMQ is a great candidate for a future backend. I don't have the resources myself right now to work also on that. Having said that, and without having worked with ClusterMQ myself, I don't think it should be too much work to wrap it all up in a ClusterMQFuture - a future backend is mostly a thin layer on top of an existing API.

Related: I'm working on setting up a conformation test suite (e.g. future.tests pkg) that can be used by all future backend pkgs to make sure they got it correct. That is my number one priority before working on new backends.

@mschubert

This comment has been minimized.

Copy link

mschubert commented Mar 10, 2018

I fully support this, but unfortunately my time is also quite limited these days.

@HenrikBengtsson HenrikBengtsson changed the title ClusterMQ as a new backend BACKEND: ClusterMQ as a new backend Mar 11, 2018

@wlandau

This comment has been minimized.

Copy link

wlandau commented Jun 28, 2018

Given that clustermq::Q() is synchronous, I am wondering what it would take to make an asynchronous ClusterMQFuture. Do we need local background processes to collect the results?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment