-
Notifications
You must be signed in to change notification settings - Fork 154
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use CuPy array in pip_bitmap_column_to_binary_array
#1418
Conversation
pip_bitmap_column_to_binary_array
to cupy array
Looks like the test needs to be updated as well:
|
@@ -31,7 +32,7 @@ def apply_binarize(in_col, width): | |||
if out.size > 0: | |||
out[:] = 0 | |||
binarize.forall(out.size)(in_col, out, width) | |||
return out | |||
return cp.asarray(out) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this is 2D, you could use asfortranarray
so when sliced into 1D columns the memory is contiguous. (I'll also ensure this on the cudf side)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
asfortranarray
will reorder the data to F-contiguous with new allocation and increases peak memory usage. It would be nice if we don't have to assume memory order for return values? Cupy support fast indexing for any memory order anyway.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we would like to change the order
of the array, this could be done by passing the order
flag to reshape
That said, given this wasn't done before with the DeviceNDArray
(and may involve other subtle changes in associated code), maybe this should be converted to a new issue/PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe this should be converted to a new issue/PR?
Yes - again, unless explicitly required by downstream libraries, I would rather cuspatial not having to dictate a memory order here but just stick to the defaults. Switching to cupy gives the flexibility not having to be explicit on memory orders, and cudf is adding back support to different memory order arrays. This relieves the responsibility from cuspatial.
pip_bitmap_column_to_binary_array
to cupy arraypip_bitmap_column_to_binary_array
Have retitled the PR to reflect the current state. Hope that is ok 🙂 |
/merge |
#1407 skipped the taxi dropoffs notebook due to performance regression fixed in #1418, so this PR re-enables the notebook in CI by removing it from SKIPNBS in ci/test_noteboks.sh. Authors: - Mark Harris (https://github.com/harrism) Approvers: - https://github.com/jakirkham - Michael Wang (https://github.com/isVoid) - James Lamb (https://github.com/jameslamb) URL: #1422
Description
The performance regression in #1413 is due to numba's
DeviceNDArray
is slow in slicing. Recent cudf's DataFrame construction has simplified the construction and delegated construction
to similar logic that handles cuda_array_interface. Since the construction involves slicing the array, we need
this operation to be fast. In that sense, we should cast the use of DeviceNDArray to cupy array to support fast
slicing.
closes #1413
Checklist