Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does purrr need to support backends? #121

Open
hadley opened this issue Nov 5, 2015 · 11 comments
Open

Does purrr need to support backends? #121

hadley opened this issue Nov 5, 2015 · 11 comments
Labels
Milestone

Comments

@hadley
Copy link
Member

@hadley hadley commented Nov 5, 2015

Potential backends:

  • In memory list
  • Sharded list, spread over multiple cores
  • NoSQL databases?
@hadley
Copy link
Member Author

@hadley hadley commented Nov 5, 2015

Implies we need to be looking at specified verbs as commonly implemented by noSQL dbs

@hadley
Copy link
Member Author

@hadley hadley commented Jul 3, 2016

First place to start would with ddR

@russellpierce
Copy link

@russellpierce russellpierce commented Feb 9, 2017

Is there a way in which building this out for ddR is trying to run before we can walk? A minimal case seems like it would be to leverage fork-based parallelism for an in-memory list. Or do you not want to tackle it until you've got the right pattern built out all of the way?

@hadley hadley added the feature label Mar 3, 2017
@1danjordan
Copy link
Contributor

@1danjordan 1danjordan commented Mar 8, 2017

If this splits out into a package 'purrrallel' would be a great name...

@hadley
Copy link
Member Author

@hadley hadley commented Mar 12, 2017

It would be natural to build these out for the modify() family, since that verb implies modification "in place", i.e. leaving the data on the remote compute engines. That will likely require making modify() an S3 generic, and only using [<- for the default method.

@lorenzwalthert
Copy link

@lorenzwalthert lorenzwalthert commented Mar 8, 2018

Are there any plans to implement any of the potential backends listed at the top of this issue in the near future?

@lionel-
Copy link
Member

@lionel- lionel- commented Mar 8, 2018

Yes we'll start working on this soon(ish).

@lorenzwalthert
Copy link

@lorenzwalthert lorenzwalthert commented Mar 8, 2018

Awesome.

@yimingli
Copy link

@yimingli yimingli commented Jan 21, 2019

While waiting for the native support in purrr, I was wondering if there are any caveats of using the furrr package, which is purrr + future.

@pat-s
Copy link

@pat-s pat-s commented Oct 10, 2019

@lionel- Meanwhile its fall 2019 - could you give an update on the current status and plans?

@lionel-
Copy link
Member

@lionel- lionel- commented Oct 10, 2019

Other things have higher priority currently. We're working on switching the tidyverse to vctrs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.