Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evaluate the pros and cons of lazy functions submission (via partition.add_to_apply_calls) #5809

Closed
dchigarev opened this issue Mar 17, 2023 · 1 comment · Fixed by #6731
Closed
Labels
Performance 🚀 Performance related issues and pull requests.

Comments

@dchigarev
Copy link
Collaborator

Do we want to make this lazy? Since split_row_partitions is in effect the properly partitioned dataframe, we can transform to col partitions, and then add_to_apply_calls the sort instead, and defer metadata materialization till it's needed?

Originally posted by @RehanSD in #5780 (comment)

Modin's partition class has a feature that seems to not be used at all. The feature allows forming a queue of remote calls making it possible to combine them as a single remote call to the execution engine when the materialization is actually needed, thus reducing the number of remote calls being made to the engine and potentially optimizing the workflow:

def add_to_apply_calls(self, func, *args, length=None, width=None, **kwargs):
"""
Add a function to the call queue.

We once had an attempt to use this feature (#1838) which however resulted in performance regression for certain cases and thus the changes were reverted by #2471. We never tried to re-iterate the changes and figure out the root cause of the regression.

So opening this issue as a reminder that we may want to do so as IMO it has a great potential for optimizing the way we operate with the execution engine.

@dchigarev dchigarev added the Performance 🚀 Performance related issues and pull requests. label Mar 17, 2023
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 7, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 7, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 7, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 8, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 8, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 8, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 8, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 9, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 9, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 9, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 9, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 9, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 9, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 10, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 16, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 17, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 20, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 21, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 27, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 27, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 27, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 28, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 29, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Nov 30, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 1, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 1, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 3, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 6, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 7, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 8, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 8, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 8, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 8, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 8, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 11, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 19, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 21, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Dec 22, 2023
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Jan 18, 2024
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
@YarShev
Copy link
Collaborator

YarShev commented Jan 19, 2024

Related to #6731.

AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Jan 25, 2024
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Jan 26, 2024
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Jan 26, 2024
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Jan 30, 2024
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Feb 2, 2024
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Feb 2, 2024
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko added a commit to AndreyPavlenko/modin that referenced this issue Feb 2, 2024
… queue

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
YarShev added a commit that referenced this issue Feb 5, 2024
Co-authored-by: Iaroslav Igoshev <Poolliver868@mail.ru>
Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Performance 🚀 Performance related issues and pull requests.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants