Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disable Zstandard decompression on nvCOMP 2.4 and Pascal GPus #11856

Merged
merged 9 commits into from
Oct 7, 2022
Merged

Disable Zstandard decompression on nvCOMP 2.4 and Pascal GPus #11856

merged 9 commits into from
Oct 7, 2022

Conversation

vuule
Copy link
Contributor

@vuule vuule commented Oct 4, 2022

Description

Zstandard decompression in nvCOMP 2.4 can produce incorrect results on compute 6.0 GPUs.
This PR disables the Zstandard decompression in this configuration.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@vuule vuule added bug Something isn't working cuIO cuIO issue breaking Breaking change labels Oct 4, 2022
@vuule vuule self-assigned this Oct 4, 2022
@github-actions github-actions bot added the libcudf Affects libcudf (C++/CUDA) code. label Oct 4, 2022
@vuule
Copy link
Contributor Author

vuule commented Oct 4, 2022

@GregoryKimball should I update the table at https://docs.rapids.ai/api/cudf/nightly/user_guide/io.html in this PR? It should indicate ZSTD as stable with an asterisk for the disabled case.

mythrocks
mythrocks approved these changes Oct 4, 2022
View reviewed changes
Copy link
Contributor

@mythrocks mythrocks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@vuule vuule changed the title Disable Zstandard decompression on nvCOMP 2.4 and compute 6.0 Disable Zstandard decompression on nvCOMP 2.4 and compute 6.1 Oct 4, 2022
@codecov
Copy link

codecov bot commented Oct 4, 2022 •
edited
Loading

Codecov Report

Base: 87.50% // Head: 87.51% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (c45ac9b) compared to base (59a3152).
Patch has no changes to coverable lines.

Additional details and impacted files 
@@              Coverage Diff              @@
##           branch-22.10   #11856   +/-   ##
=============================================
  Coverage         87.50%   87.51%           
=============================================
  Files               133      133           
  Lines             21826    21826           
=============================================
+ Hits              19099    19100    +1     
+ Misses             2727     2726    -1
Impacted Files Coverage Δ
python/cudf/cudf/core/column/numerical.py 95.49% <0.00%> (+0.28%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@vuule vuule marked this pull request as ready for review October 4, 2022 16:33
@vuule vuule requested a review from a team as a code owner October 4, 2022 16:33
@vuule vuule requested review from vyasr and upsj and removed request for a team October 4, 2022 16:33
@mythrocks mythrocks self-requested a review October 4, 2022 17:14
@mythrocks
Copy link
Contributor

I think I reviewed this incorrectly. Having a closer look now.

@vuule vuule requested a review from jbrennan333 October 4, 2022 18:21
jbrennan333
jbrennan333 approved these changes Oct 4, 2022
View reviewed changes
Copy link
Contributor

@jbrennan333 jbrennan333 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 this looks good. Hopefully nvcomp-2.4.1 will fix this and we can update to allow it.

cpp/src/io/comp/nvcomp_adapter.cpp Outdated Show resolved Hide resolved
@jbrennan333 jbrennan333 mentioned this pull request Oct 4, 2022
Closed
@vuule vuule changed the title Disable Zstandard decompression on nvCOMP 2.4 and compute 6.1 Disable Zstandard decompression on nvCOMP 2.4 and Pascal GPus Oct 6, 2022
jbrennan333
jbrennan333 reviewed Oct 6, 2022
View reviewed changes
cpp/src/io/comp/nvcomp_adapter.cpp Outdated
// compute 6.1
#if NVCOMP_MAJOR_VERSION > 2 or (NVCOMP_MAJOR_VERSION == 2 and NVCOMP_MINOR_VERSION >= 4)
// compute 6.x
#if NVCOMP_MAJOR_VERSION == 2 and NVCOMP_MINOR_VERSION == 4
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we need to check the patch version == 0 as well?
Otherwise this will disable it for 2.4.1 as well.

jbrennan333
jbrennan333 approved these changes Oct 6, 2022
View reviewed changes
Copy link
Contributor

@jbrennan333 jbrennan333 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 looks good

@raydouglass raydouglass merged commit 4c4bce9 into rapidsai:branch-22.10 Oct 7, 2022
@mythrocks mythrocks mentioned this pull request Oct 11, 2022
Closed
@vuule vuule deleted the bug-nvcomp-zstd-pascal branch August 10, 2023 03:25
@xingwenqiang
Copy link

Use this version to export zstd.parquet files. An exception occurs when reading with arrow while some page compressed size is larger than the size before. The similarity between these pages is that the last multiple elements are empty. Is this a usage problem?，Looking forward to your reply！

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mythrocks mythrocks mythrocks approved these changes

@galipremsagar galipremsagar galipremsagar left review comments

@vyasr vyasr vyasr approved these changes

@ttnghia ttnghia ttnghia approved these changes

@jolorunyomi jolorunyomi jolorunyomi approved these changes

@jbrennan333 jbrennan333 jbrennan333 approved these changes

@upsj upsj Awaiting requested review from upsj upsj is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

Assignees

@vuule vuule

Labels
breaking Breaking change bug Something isn't working cuIO cuIO issue libcudf Affects libcudf (C++/CUDA) code.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
9 participants
@vuule @mythrocks @xingwenqiang @vyasr @ttnghia @galipremsagar @jolorunyomi @jbrennan333 @raydouglass