[REVIEW] Adding to_parquet and write_partition definitions to dask_cudf #3369
Conversation
rjzamora
changed the title
rjzamora
added
3 - Ready for Review
| gddf.to_parquet(tmpdir) | ||
|
|
||
| # NOTE: Need `.compute()` to resolve correct index | ||
| # name after `from_dask_dataframe` |
There was a problem hiding this comment.
I'm not sure I understand this. Is there something wrong with our metadata?
There was a problem hiding this comment.
I was also a bit confused by this. It seems that gddf = ddf.map_partitions(cudf.from_pandas) will result in a dask_cudf dataframe with the _meta not having the same index name ans the original dask dataframe. Is it reasonable for me to raise a separate issue?
There was a problem hiding this comment.
If you can find a quick resolution to that I think that that would be ideal. If it ends up being more complicated then sure, let's wait.
There was a problem hiding this comment.
If you can find a quick resolution to that I think that that would be ideal.
Okay - Looks like the index name is dropped in iloc calls for cudf dataframes (and not for pandas). I'll see if the fix is simple
There was a problem hiding this comment.
@mrocklin My idea for a simple fix doesn't seem to work - Lets address the index-name issue separately so I can focus on higher priority items in the short term.
There was a problem hiding this comment.
OK, sounds good. Happy to merge on passed tests
Co-Authored-By: Matthew Rocklin <mrocklin@gmail.com>
Codecov Report
@@ Coverage Diff @@
## branch-0.11 #3369 +/- ##
===============================================
+ Coverage 87.15% 87.18% +0.02%
===============================================
Files 49 49
Lines 9213 9220 +7
===============================================
+ Hits 8030 8038 +8
+ Misses 1183 1182 -1
Continue to review full report at Codecov.
|
vyasr
added
4 - Needs Review
Closes #3365
This PR adds explicit definitions for
to_parquetandwrite_partitionin dask_cudf. Previously, dask_cudf was falling back to upstream dask for this functionality. However, there is pandas-specific code inwrite_partitionthat is causing problems for cudf-based dask dataframes. Since we will soon need to define aCudfEngineimplementation ofwrite_partitionwhen a GPU-acceleratedcudf.io.to_parquetis supported, it probably makes sense to add an initial implementation to address the #3365 bug.