Does purrr need to support backends?

[hadley]

Potential backends:

• In memory list
• Sharded list, spread over multiple cores
• NoSQL databases?

[hadley]

Implies we need to be looking at specified verbs as commonly implemented by noSQL dbs

[hadley]

First place to start would with ddR

[russellpierce]

Is there a way in which building this out for ddR is trying to run before we can walk? A minimal case seems like it would be to leverage fork-based parallelism for an in-memory list. Or do you not want to tackle it until you've got the right pattern built out all of the way?

[1danjordan]

If this splits out into a package 'purrrallel' would be a great name...

[hadley]

It would be natural to build these out for the modify() family, since that verb implies modification "in place", i.e. leaving the data on the remote compute engines. That will likely require making modify() an S3 generic, and only using [<- for the default method.

[lorenzwalthert]

Are there any plans to implement any of the potential backends listed at the top of this issue in the near future?

[lionel-]

Yes we'll start working on this soon(ish).

[lorenzwalthert]

Awesome.