-
Notifications
You must be signed in to change notification settings - Fork 883
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[REVIEW] Adding to_parquet and write_partition definitions to dask_cudf #3369
Conversation
gddf.to_parquet(tmpdir) | ||
|
||
# NOTE: Need `.compute()` to resolve correct index | ||
# name after `from_dask_dataframe` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure I understand this. Is there something wrong with our metadata?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was also a bit confused by this. It seems that gddf = ddf.map_partitions(cudf.from_pandas)
will result in a dask_cudf dataframe with the _meta
not having the same index name ans the original dask dataframe. Is it reasonable for me to raise a separate issue?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you can find a quick resolution to that I think that that would be ideal. If it ends up being more complicated then sure, let's wait.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you can find a quick resolution to that I think that that would be ideal.
Okay - Looks like the index name is dropped in iloc
calls for cudf dataframes (and not for pandas). I'll see if the fix is simple
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mrocklin My idea for a simple fix doesn't seem to work - Lets address the index-name issue separately so I can focus on higher priority items in the short term.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, sounds good. Happy to merge on passed tests
Co-Authored-By: Matthew Rocklin <mrocklin@gmail.com>
Codecov Report
@@ Coverage Diff @@
## branch-0.11 #3369 +/- ##
===============================================
+ Coverage 87.15% 87.18% +0.02%
===============================================
Files 49 49
Lines 9213 9220 +7
===============================================
+ Hits 8030 8038 +8
+ Misses 1183 1182 -1
Continue to review full report at Codecov.
|
Closes #3365
This PR adds explicit definitions for
to_parquet
andwrite_partition
in dask_cudf. Previously, dask_cudf was falling back to upstream dask for this functionality. However, there is pandas-specific code inwrite_partition
that is causing problems for cudf-based dask dataframes. Since we will soon need to define aCudfEngine
implementation ofwrite_partition
when a GPU-acceleratedcudf.io.to_parquet
is supported, it probably makes sense to add an initial implementation to address the #3365 bug.