-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[data] Removing unnecessary data copy in convert_udf_returns_to_numpy #39188
Conversation
Signed-off-by: Hao Chen <chenh1024@gmail.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
@@ -488,7 +488,10 @@ def _concat_same_type( | |||
[e for a in to_concat for e in a] | |||
) | |||
else: | |||
storage = pa.concat_arrays([c.storage for c in to_concat]) | |||
if len(to_concat) == 1: | |||
storage = to_concat[0].storage |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we add a comment for why this is needed, for reader in the future.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It turns out that the PyArrow Block copy code path expects this function to copy data. I reverted this change in this PR, as it doesn't impact the final benchmark perf.
This reverts commit 75b7581. Signed-off-by: Hao Chen <chenh1024@gmail.com>
…ray-project#39188) --------- Signed-off-by: Hao Chen <chenh1024@gmail.com>
…ray-project#39188) --------- Signed-off-by: Hao Chen <chenh1024@gmail.com> Signed-off-by: Jim Thompson <jimthompson5802@gmail.com>
…ray-project#39188) --------- Signed-off-by: Hao Chen <chenh1024@gmail.com> Signed-off-by: Victor <vctr.y.m@example.com>
Why are these changes needed?
This increases the perf of
image_loader_microbenchmark.py
by ~10%.Related issue number
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/
under thecorresponding
.rst
file.