Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto merge small chunks when df.groupby().apply(func) is doing aggregation #2708

Merged
merged 1 commit into from
Feb 12, 2022

Conversation

qinxuye
Copy link
Collaborator

@qinxuye qinxuye commented Feb 11, 2022

What do these changes do?

This PR introduces mechanism that auto merge small chunks when df.groupby().apply(func) is doing aggregation.

Related issue number

Fixes #2692 .

Check code requirements

  • tests added / passed (if needed)
  • Ensure all linting tests pass, see here for how to run them

Copy link
Member

@wjsi wjsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@hekaisheng hekaisheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hekaisheng hekaisheng merged commit df1492c into mars-project:master Feb 12, 2022
@qinxuye qinxuye deleted the enh/auto-merge branch February 12, 2022 03:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Chunks of df.groupby().apply(func) can be auto merged if the func is actually an aggregation func
3 participants