New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does purrr need to support backends? #121

Open
hadley opened this Issue Nov 5, 2015 · 8 comments

Comments

Projects
None yet
5 participants
@hadley
Member

hadley commented Nov 5, 2015

Potential backends:

  • In memory list
  • Sharded list, spread over multiple cores
  • NoSQL databases?
@hadley

This comment has been minimized.

Show comment
Hide comment
@hadley

hadley Nov 5, 2015

Member

Implies we need to be looking at specified verbs as commonly implemented by noSQL dbs

Member

hadley commented Nov 5, 2015

Implies we need to be looking at specified verbs as commonly implemented by noSQL dbs

@hadley

This comment has been minimized.

Show comment
Hide comment
@hadley

hadley Jul 3, 2016

Member

First place to start would with ddR

Member

hadley commented Jul 3, 2016

First place to start would with ddR

@russellpierce

This comment has been minimized.

Show comment
Hide comment
@russellpierce

russellpierce Feb 9, 2017

Is there a way in which building this out for ddR is trying to run before we can walk? A minimal case seems like it would be to leverage fork-based parallelism for an in-memory list. Or do you not want to tackle it until you've got the right pattern built out all of the way?

russellpierce commented Feb 9, 2017

Is there a way in which building this out for ddR is trying to run before we can walk? A minimal case seems like it would be to leverage fork-based parallelism for an in-memory list. Or do you not want to tackle it until you've got the right pattern built out all of the way?

@hadley hadley added the feature label Mar 3, 2017

@dandermotj

This comment has been minimized.

Show comment
Hide comment
@dandermotj

dandermotj Mar 8, 2017

Contributor

If this splits out into a package 'purrrallel' would be a great name...

Contributor

dandermotj commented Mar 8, 2017

If this splits out into a package 'purrrallel' would be a great name...

@hadley

This comment has been minimized.

Show comment
Hide comment
@hadley

hadley Mar 12, 2017

Member

It would be natural to build these out for the modify() family, since that verb implies modification "in place", i.e. leaving the data on the remote compute engines. That will likely require making modify() an S3 generic, and only using [<- for the default method.

Member

hadley commented Mar 12, 2017

It would be natural to build these out for the modify() family, since that verb implies modification "in place", i.e. leaving the data on the remote compute engines. That will likely require making modify() an S3 generic, and only using [<- for the default method.

@lorenzwalthert

This comment has been minimized.

Show comment
Hide comment
@lorenzwalthert

lorenzwalthert Mar 8, 2018

Are there any plans to implement any of the potential backends listed at the top of this issue in the near future?

lorenzwalthert commented Mar 8, 2018

Are there any plans to implement any of the potential backends listed at the top of this issue in the near future?

@lionel-

This comment has been minimized.

Show comment
Hide comment
@lionel-

lionel- Mar 8, 2018

Member

Yes we'll start working on this soon(ish).

Member

lionel- commented Mar 8, 2018

Yes we'll start working on this soon(ish).

@lorenzwalthert

This comment has been minimized.

Show comment
Hide comment
@lorenzwalthert

lorenzwalthert commented Mar 8, 2018

Awesome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment