-
Notifications
You must be signed in to change notification settings - Fork 883
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Disable Zstandard decompression on nvCOMP 2.4 and Pascal GPus #11856
Disable Zstandard decompression on nvCOMP 2.4 and Pascal GPus #11856
Conversation
@GregoryKimball should I update the table at https://docs.rapids.ai/api/cudf/nightly/user_guide/io.html in this PR? It should indicate ZSTD as stable with an asterisk for the disabled case. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Codecov ReportBase: 87.50% // Head: 87.51% // Increases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## branch-22.10 #11856 +/- ##
=============================================
Coverage 87.50% 87.51%
=============================================
Files 133 133
Lines 21826 21826
=============================================
+ Hits 19099 19100 +1
+ Misses 2727 2726 -1
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
I think I reviewed this incorrectly. Having a closer look now. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 this looks good. Hopefully nvcomp-2.4.1 will fix this and we can update to allow it.
…bug-nvcomp-zstd-pascal
…into bug-nvcomp-zstd-pascal
cpp/src/io/comp/nvcomp_adapter.cpp
Outdated
// compute 6.1 | ||
#if NVCOMP_MAJOR_VERSION > 2 or (NVCOMP_MAJOR_VERSION == 2 and NVCOMP_MINOR_VERSION >= 4) | ||
// compute 6.x | ||
#if NVCOMP_MAJOR_VERSION == 2 and NVCOMP_MINOR_VERSION == 4 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't we need to check the patch version == 0 as well?
Otherwise this will disable it for 2.4.1 as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 looks good
Use this version to export zstd.parquet files. An exception occurs when reading with arrow while some page compressed size is larger than the size before. The similarity between these pages is that the last multiple elements are empty. Is this a usage problem?,Looking forward to your reply! |
Description
Zstandard decompression in nvCOMP 2.4 can produce incorrect results on compute 6.0 GPUs.
This PR disables the Zstandard decompression in this configuration.
Checklist