Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question: Where to start future implementation for Redis? #151

Open
rstub opened this issue Jun 12, 2017 · 3 comments
Open

Question: Where to start future implementation for Redis? #151

rstub opened this issue Jun 12, 2017 · 3 comments
Labels

Comments

@rstub
Copy link

@rstub rstub commented Jun 12, 2017

In doFuture's vignette it says:

There is currently no known future implementation and therefore no known [doFuture] alternative to the [doRedis] packages.

I think a future implementation based on Redis would be very interesting. Could you give me a hint where to start such an implementation?

@HenrikBengtsson
Copy link
Owner

@HenrikBengtsson HenrikBengtsson commented Jan 5, 2018

Hi, this one slipped for many reasons - one being I'm not a regular/savvy user of Redis. One solution would be to implement a future backend on top of the rrqueue package, which builds upon Redis.

The very rough outline would be to create a RrqueueFuture class that extends the Future class (or possibly the MultiprocessFuture class) and implements a run() that launch/spawn to rrqueue, a resolved() that checks with rrqueue whether the task is done, and a value() that collects the value from rrqueue. It looks like rrqueue already has functions for those mechanisms, so it could be that a first prototype would only require a very shallow set of wrappers.

@richfitz
Copy link

@richfitz richfitz commented Apr 1, 2021

Hi @HenrikBengtsson - we're in the process of tidying up the successor to rrqueue at the moment (https://github.com/mrc-ide/rrq/) which will go to CRAN later in the year. Where would one start looking for implementing a future backend, and I can either look at this or see if I can put pieces in place to make it straightforward

@HenrikBengtsson
Copy link
Owner

@HenrikBengtsson HenrikBengtsson commented Apr 4, 2021

@richfitz - it could be that the future.callr backend is a good example, where the core of the action is in https://github.com/HenrikBengtsson/future.callr/blob/develop/R/CallrFuture-class.R.

I'd be interested in learning more on Redis and how it can be used for orchestrating parallel processing, and would be willing to help with the future end of the task. I know you've used it for a long time. We could start with a call to synchronize expectations on how futures and Redis can work together. I'll ping you by email.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants