New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Evaluate the pros and cons of lazy functions submission (via partition.add_to_apply_calls
)
#5809
Labels
Performance 🚀
Performance related issues and pull requests.
Comments
7 tasks
7 tasks
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 7, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 7, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 7, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 8, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 8, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 8, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 8, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 9, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 9, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 9, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
7 tasks
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 9, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 9, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 9, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 10, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 16, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 17, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 20, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 21, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 27, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 27, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 27, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 28, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 29, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Nov 30, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 1, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 1, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 3, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 6, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 7, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 8, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 8, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 8, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 8, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 8, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 11, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 19, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 21, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Dec 22, 2023
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Jan 18, 2024
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
Related to #6731. |
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Jan 25, 2024
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Jan 26, 2024
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Jan 26, 2024
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Jan 30, 2024
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Feb 2, 2024
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Feb 2, 2024
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
AndreyPavlenko
added a commit
to AndreyPavlenko/modin
that referenced
this issue
Feb 2, 2024
… queue Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
YarShev
added a commit
that referenced
this issue
Feb 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Originally posted by @RehanSD in #5780 (comment)
Modin's partition class has a feature that seems to not be used at all. The feature allows forming a queue of remote calls making it possible to combine them as a single remote call to the execution engine when the materialization is actually needed, thus reducing the number of remote calls being made to the engine and potentially optimizing the workflow:
modin/modin/core/dataframe/pandas/partitioning/partition.py
Lines 117 to 119 in 94b2a0e
We once had an attempt to use this feature (#1838) which however resulted in performance regression for certain cases and thus the changes were reverted by #2471. We never tried to re-iterate the changes and figure out the root cause of the regression.
So opening this issue as a reminder that we may want to do so as IMO it has a great potential for optimizing the way we operate with the execution engine.
The text was updated successfully, but these errors were encountered: