(enhancement): Apply modin repartitioning where required only

[jaidisido]

Feature or Bugfix

• Enhancement

Detail

Currently modin repartitioning is applied as soon as a write method (to_csv, to_parquet) is called. This is not only a wasteful operation performance-wise but also does not prevent the need to repartition once more if the dataframe is altered within the method (e.g. if a column is cast to a new type).

Instead, repartitioning should be applied when it's currently required, that is before a modin group by only.

Relates

• BUG: KeyError when UDF in groupby.apply accesses data from another column partition modin-project/modin#3435
• Pandas-esque groupby apply modin-project/modin#4165 (comment)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[malachi-constant]

AWS CodeBuild CI Report

• CodeBuild project: GitHubLoadTests5656BB24-s6u9F3qN9oFy
• Commit ID: 01934e2
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[malachi-constant]

AWS CodeBuild CI Report

• CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
• Commit ID: 01934e2
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[malachi-constant]

AWS CodeBuild CI Report

• CodeBuild project: GitHubStandardCodeBuild8C06-llutOAimTATs
• Commit ID: 01934e2
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[malachi-constant]

AWS CodeBuild CI Report

• CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
• Commit ID: 72c3903
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[malachi-constant]

AWS CodeBuild CI Report

• CodeBuild project: GitHubStandardCodeBuild8C06-llutOAimTATs
• Commit ID: 72c3903
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[malachi-constant]

AWS CodeBuild CI Report

• CodeBuild project: GitHubLoadTests5656BB24-s6u9F3qN9oFy
• Commit ID: 72c3903
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[LeonLuttenberger]

LGTM

[csnatarajan]

@jaidisido This would be a good addition to Modin as well, would you be willing to contribute this back to Modin?

[kukushking]

@csnatarajan do you mean @modin_repartition decorator?

[mvashishtha]

@kukushking is there anything related to that decorator that would be good to include in Modin? Is the purpose of the decorator to work around bugs like modin-project/modin#3435?

[mvashishtha]

@kukushking also, would it help if modin allowed users to configure the column and row partitioning separately as here? That way you wouldn't have to keep undoing the column partitioning in modin_repartition (from a cursory look I think that's what that decorator does).

[kukushking]

Yes @mvashishtha the purpose of the decorator was precisely to work around the issues like when accessing columns from another column-axis partition. We also noticed that we need to have the df in this shape before operations like groupby - perhaps @jaidisido can explain better