Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the bounce buffer size in ORC writer #14947

Merged
merged 4 commits into from
Feb 7, 2024

Conversation

vuule
Copy link
Contributor

@vuule vuule commented Feb 1, 2024

Description

Closes #14932

ORC writer uses uncompressed stream sizes when allocating the bounce buffer. This can lead to issues when all uncompressed streams are larger than the GDS threshold, but compressed size is not. In this scenario, the bounce buffer is not allocated, and writing the compressed stream through the bounce buffer causes a crash.

This PR moves the computation of the bounce buffer size until after compression, so it works with correct stream sizes.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@vuule vuule added bug Something isn't working cuIO cuIO issue non-breaking Non-breaking change labels Feb 1, 2024
@vuule vuule self-assigned this Feb 1, 2024
@github-actions github-actions bot added the libcudf Affects libcudf (C++/CUDA) code. label Feb 1, 2024
@vuule vuule marked this pull request as ready for review February 1, 2024 19:46
@vuule vuule requested a review from a team as a code owner February 1, 2024 19:46
@ttnghia
Copy link
Contributor

ttnghia commented Feb 1, 2024

Thanks for working on it 😄

@vuule vuule changed the base branch from branch-24.04 to branch-24.02 February 6, 2024 21:47
@vuule vuule requested review from a team as code owners February 6, 2024 21:47
@vuule vuule requested review from vyasr and mroeschke and removed request for a team February 6, 2024 21:47
@vuule vuule changed the base branch from branch-24.02 to branch-24.04 February 6, 2024 21:48
@bdice bdice changed the base branch from branch-24.04 to branch-24.02 February 6, 2024 21:55
@raydouglass raydouglass merged commit 8d2b0ed into rapidsai:branch-24.02 Feb 7, 2024
67 of 68 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working cuIO cuIO issue libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

[BUG] ORC writer hit cudaErrorInvalidValue exception with ZSTD compression
4 participants