Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Register partd encode dispatch in dask_cudf #14287

Merged
merged 108 commits into from
Nov 6, 2023
Merged

Register partd encode dispatch in dask_cudf #14287

merged 108 commits into from
Nov 6, 2023

Conversation

rjzamora
Copy link
Member

@rjzamora rjzamora commented Oct 16, 2023 •
edited
Loading

Description

This PR enables "disk"-based shuffling of cudf-backed Dask-DataFrame collections, but does not yet add the shuffle="disk" option to the dask_cudf.DataFrame.shuffle/sort_values APIs.

We now use basic (slow) pickle logic to convert cudf.DataFrame objects to/from bytes here, so I'd like to consider further optimizations before making the shuffle="disk" option "official".

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

raydouglass and others added 30 commits March 30, 2020 11:03
@raydouglass
Merge pull request rapidsai#4714 from rapidsai/branch-0.13
19f5174
@GPUtester
REL v0.13.0 release
a2804c3
@mike-wendt
REL v0.13.0 CHANGELOG Updates
fef2a2b 
Closes rapidsai#4775
@raydouglass
Merge pull request rapidsai#5310 from rapidsai/branch-0.14
ab00eb0
@GPUtester
REL v0.14.0 release
b34b838
@ajschmidt8
update master references
9ff9cdb
@mike-wendt
REL DOC Updates for main branch switch
789d19b 
Merge pull request rapidsai#5690 from ajschmidt8/phase2
[skip ci] Update master references for main branch
@raydouglass
Merge pull request rapidsai#6079 from rapidsai/branch-0.15
819f514
@GPUtester
REL v0.15.0 release
3a0f214
@raydouglass
Merge pull request rapidsai#6101 from rapidsai/branch-0.15
f947393 
[RELEASE] Re-release v0.15 cudf [skip-ci]
@GPUtester
REL v0.15.0 release
71cb8c0
@raydouglass
Merge pull request rapidsai#6547 from rapidsai/branch-0.16
7ef8174
@GPUtester
REL v0.16.0 release
2b8298f
@ajschmidt8
Merge pull request rapidsai#6935 from rapidsai/branch-0.17
d72b1eb 
[RELEASE] cudf v0.17
@GPUtester
REL v0.17.0 release
f56ef85
@raydouglass
Merge pull request rapidsai#7405 from rapidsai/branch-0.18
b7e1a85 
[RELEASE] cudf v0.18
@GPUtester
REL v0.18.0 release
20778e5
@raydouglass
Merge pull request rapidsai#7585 from rapidsai/branch-0.18
042c20f 
[RELEASE] Release v0.18.1 cudf
@raydouglass
REL v0.18.1 release
999be56
@raydouglass
Merge pull request rapidsai#7969 from rapidsai/branch-0.18
2391864 
[RELEASE] v0.18.2 `cudf` release [skip-ci]
@raydouglass
REL v0.18.2 release
3341561
@raydouglass
Merge pull request rapidsai#7626 from rapidsai/branch-0.19
6573759
@GPUtester
REL v0.19.0 release
f07b251
@ajschmidt8
REL Changelog update
61e5a20
@raydouglass
Merge pull request rapidsai#8037 from rapidsai/branch-0.19
a13e8dc 
[RELEASE] v0.19.1 cudf
@GPUtester
REL v0.19.1 release
a9f3453
@raydouglass
Merge pull request rapidsai#8100 from rapidsai/branch-0.19
2089fc9 
[RELEASE] v0.19.2 cudf [skip-ci]
@GPUtester
REL v0.19.2 release
ab3b3f6
@raydouglass
Merge pull request rapidsai#8418 from rapidsai/branch-21.06
f9d5e2e
@GPUtester
REL v21.06.00 release
ae44046
wence-
wence- previously approved these changes Oct 18, 2023
View reviewed changes
Copy link
Contributor

@wence- wence- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, thanks!

@wence- wence- dismissed their stale review October 18, 2023 11:49

Implementation not yet fully working

@wence-
Copy link
Contributor

wence- commented Oct 18, 2023

Hmm, looks like the disk shuffle in dask wants some internal property of an Index that our indices do not provide:

FAILED python/dask_cudf/dask_cudf/tests/test_sort.py::test_disk_shuffle - AttributeError: 'Int64Index' object has no attribute '_attributes'

@rjzamora
Copy link
Member Author

Hmm, looks like the disk shuffle in dask wants some internal property of an Index that our indices do not provide:

Ah, right - Sorry. The test requires a newer version of dask, so I'll need to add a version check to that test.

@rjzamora
Copy link
Member Author

/ok to test

@rjzamora
Copy link
Member Author

/ok to test

@rjzamora
Copy link
Member Author

/ok to test

@rjzamora
Copy link
Member Author

/ok to test

@rjzamora
Copy link
Member Author

/ok to test

@rjzamora
Copy link
Member Author

/ok to test

@rjzamora
Copy link
Member Author

rjzamora commented Nov 3, 2023

/ok to test

galipremsagar
galipremsagar reviewed Nov 3, 2023
View reviewed changes
python/dask_cudf/dask_cudf/backends.py
import pickle
from functools import partial

import partd
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add partd to our package requirements and conda recipes?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it comes in transitively through dask? Which is a dependency of dask-cudf.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right exactly, dask depends on partd. I think it should be safe for us to let dask worry about the partd dependency. If dask suddenly stops using partd for shuffle="disk", it will also stop using partd_encode_dispatch.

@rjzamora
Copy link
Member Author

rjzamora commented Nov 6, 2023

/ok to test

@rjzamora
Copy link
Member Author

rjzamora commented Nov 6, 2023

/ok to test

@rjzamora rjzamora added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 3 - Ready for Review Ready for review by team labels Nov 6, 2023
@rjzamora
Copy link
Member Author

rjzamora commented Nov 6, 2023

/merge

@rapids-bot rapids-bot bot merged commit 70c4283 into rapidsai:branch-23.12 Nov 6, 2023
61 checks passed
@rjzamora rjzamora deleted the register-partd_encode_dispatch branch November 6, 2023 19:45
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wence- wence- wence- left review comments

@galipremsagar galipremsagar galipremsagar approved these changes

Assignees
No one assigned
Labels
5 - Ready to Merge Testing and reviews complete, ready to merge dask Dask issue improvement Improvement / enhancement to an existing function non-breaking Non-breaking change Python Affects Python cuDF API.
Projects
cuDF/Dask/Numba/UCX
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@rjzamora @wence- @galipremsagar @raydouglass @GPUtester @mike-wendt @ajschmidt8