Question: Where to start future implementation for Redis?

[rstub]

In doFuture's vignette it says:

There is currently no known future implementation and therefore no known [doFuture] alternative to the [doRedis] packages.

I think a future implementation based on Redis would be very interesting. Could you give me a hint where to start such an implementation?

[HenrikBengtsson]

Hi, this one slipped for many reasons - one being I'm not a regular/savvy user of Redis. One solution would be to implement a future backend on top of the rrqueue package, which builds upon Redis.

The very rough outline would be to create a RrqueueFuture class that extends the Future class (or possibly the MultiprocessFuture class) and implements a run() that launch/spawn to rrqueue, a resolved() that checks with rrqueue whether the task is done, and a value() that collects the value from rrqueue. It looks like rrqueue already has functions for those mechanisms, so it could be that a first prototype would only require a very shallow set of wrappers.

[richfitz]

Hi @HenrikBengtsson - we're in the process of tidying up the successor to rrqueue at the moment (https://github.com/mrc-ide/rrq/) which will go to CRAN later in the year. Where would one start looking for implementing a future backend, and I can either look at this or see if I can put pieces in place to make it straightforward

[HenrikBengtsson]

@richfitz - it could be that the future.callr backend is a good example, where the core of the action is in https://github.com/HenrikBengtsson/future.callr/blob/develop/R/CallrFuture-class.R.

I'd be interested in learning more on Redis and how it can be used for orchestrating parallel processing, and would be willing to help with the future end of the task. I know you've used it for a long time. We could start with a call to synchronize expectations on how futures and Redis can work together. I'll ping you by email.