Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

map_partitions with review comments #6776

Merged
merged 4 commits into from
Jan 15, 2021

Conversation

kumarprabhu1988
Copy link
Contributor

@kumarprabhu1988 kumarprabhu1988 commented Oct 29, 2020

This PR adds functionality to return partition_info in a call to map_partitions. It is useful for distributed implementations of many algorithms. This PR includes changes for review comments in this PR: #6755.

@kumarprabhu1988 kumarprabhu1988 mentioned this pull request Oct 29, 2020
2 tasks
@jsignell
Copy link
Member

Thanks for keeping at it @kumarprabhu1988 - I think this is getting close. The missing piece now is docs. Could you add a section to the examples here:

def map_partitions(self, func, *args, **kwargs):

@kumarprabhu1988
Copy link
Contributor Author

@jsignell Apologies for the delay. I've updated the docs. Let me know if this looks good.

@jsignell
Copy link
Member

jsignell commented Dec 1, 2020

I really think the doc string should include an example using partition_info. Would you be able to add that?

@jsignell
Copy link
Member

I'll merge this when it passes.

@jrbourbeau
Copy link
Member

Just merged master to bring in some CI fixes (cc @jsignell)

@jrbourbeau
Copy link
Member

Just checking in @jsignell, is this still good to go?

@jsignell jsignell merged commit d9fbc42 into dask:master Jan 15, 2021
abduhbm pushed a commit to abduhbm/dask that referenced this pull request Jan 19, 2021
* map_partitions with review comments

* update doc

* Add example to docstring

Co-authored-by: Julia Signell <jsignell@gmail.com>
Co-authored-by: James Bourbeau <jrbourbeau@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add magic partition_info keyword to dd.map_partitions function
3 participants