New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FEAT-#6398: Improved performance of list-like objects insertion into HDK DataFrames #6412
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AndreyPavlenko
force-pushed
the
issue-6398
branch
from
July 24, 2023 19:35
db868ed
to
d5aaffa
Compare
...n/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition.py
Fixed
Show fixed
Hide fixed
...n/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition.py
Fixed
Show fixed
Hide fixed
AndreyPavlenko
force-pushed
the
issue-6398
branch
from
July 24, 2023 20:35
d5aaffa
to
5559bae
Compare
...n/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition.py
Fixed
Show fixed
Hide fixed
AndreyPavlenko
force-pushed
the
issue-6398
branch
6 times, most recently
from
July 27, 2023 22:26
6dfdfc4
to
6e33d36
Compare
modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/utils.py
Fixed
Show fixed
Hide fixed
...n/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition.py
Fixed
Show fixed
Hide fixed
AndreyPavlenko
force-pushed
the
issue-6398
branch
6 times, most recently
from
July 29, 2023 19:47
9d6575b
to
eb7ade3
Compare
AndreyPavlenko
force-pushed
the
issue-6398
branch
from
August 9, 2023 12:59
eb7ade3
to
347d15f
Compare
7 tasks
AndreyPavlenko
force-pushed
the
issue-6398
branch
from
August 22, 2023 18:00
347d15f
to
e166a1b
Compare
AndreyPavlenko
force-pushed
the
issue-6398
branch
3 times, most recently
from
September 25, 2023 21:07
2dfdf73
to
48f2a81
Compare
AndreyPavlenko
force-pushed
the
issue-6398
branch
from
September 27, 2023 13:06
48f2a81
to
521fddc
Compare
7 tasks
anmyachev
reviewed
Jan 9, 2024
modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/utils.py
Outdated
Show resolved
Hide resolved
anmyachev
reviewed
Jan 9, 2024
modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/utils.py
Outdated
Show resolved
Hide resolved
AndreyPavlenko
changed the title
FEAT-#6398: Improved performance of list-like objects insertion into DataFrames
FEAT-#6398: Improved performance of list-like objects insertion into HDK DataFrames
Jan 22, 2024
AndreyPavlenko
force-pushed
the
issue-6398
branch
from
January 22, 2024 18:48
6b073ef
to
8c55fc1
Compare
anmyachev
reviewed
Jan 23, 2024
modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
Outdated
Show resolved
Hide resolved
anmyachev
reviewed
Jan 23, 2024
...n/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition.py
Show resolved
Hide resolved
anmyachev
reviewed
Jan 23, 2024
modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/utils.py
Outdated
Show resolved
Hide resolved
anmyachev
reviewed
Jan 23, 2024
modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
Show resolved
Hide resolved
...n/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition.py
Fixed
Show fixed
Hide fixed
AndreyPavlenko
force-pushed
the
issue-6398
branch
from
January 23, 2024 22:20
5480e62
to
ca4f315
Compare
anmyachev
previously approved these changes
Jan 24, 2024
anmyachev
reviewed
Jan 24, 2024
modin/experimental/core/execution/native/implementations/hdk_on_native/dataframe/dataframe.py
Outdated
Show resolved
Hide resolved
@dchigarev any other comments? |
...n/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition.py
Fixed
Show fixed
Hide fixed
AndreyPavlenko
force-pushed
the
issue-6398
branch
from
January 24, 2024 12:53
0deed6c
to
ba8afb7
Compare
anmyachev
previously approved these changes
Jan 24, 2024
@anmyachev, @dchigarev, can we merge this? |
@YarShev let's merge after #6887 if @dchigarev agree with that changes |
dchigarev
previously approved these changes
Jan 26, 2024
@AndreyPavlenko please rebase |
…sertion into HDK DataFrames If the partition contains either pandas DataFrame or pyarrow Table, insert the object directly into the frame/table, otherwise create a single-column frame and join the frames by rowid. Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
Co-authored-by: Dmitry Chigarev <dmitry.chigarev@intel.com> Co-authored-by: Anatoly Myachev <anatoly.myachev@intel.com>
…n_native/dataframe/dataframe.py Co-authored-by: Anatoly Myachev <anatoliimyachev@mail.com>
AndreyPavlenko
dismissed stale reviews from dchigarev and anmyachev
via
January 26, 2024 18:26
2e73aea
AndreyPavlenko
force-pushed
the
issue-6398
branch
from
January 26, 2024 18:26
ba8afb7
to
2e73aea
Compare
anmyachev
approved these changes
Jan 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do?
If the partition contains either pandas DataFrame or pyarrow Table, insert the object directly into the frame/table, otherwise create a single-column frame and join the frames by rowid.
flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
git commit -s
docs/development/architecture.rst
is up-to-date