Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixes bug in temporary decompression space estimation before calling nvcomp #11879

Merged
merged 2 commits into from
Oct 8, 2022
Merged

Fixes bug in temporary decompression space estimation before calling nvcomp #11879

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
22beba9
Fixes bug in temporary decompression space estimation before calling …
abellina Oct 7, 2022
61e2499
Apply codestyle changes
abellina Oct 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions cpp/src/io/comp/nvcomp_adapter.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,12 +147,15 @@ size_t batched_decompress_temp_size(compression_type compression,
size_t max_uncomp_chunk_size,
size_t max_total_uncomp_size)
{
size_t temp_size = 0;
auto const nvcomp_status =
batched_decompress_get_temp_size_ex(
compression, num_chunks, max_uncomp_chunk_size, &temp_size, max_total_uncomp_size)
.value_or(batched_decompress_get_temp_size(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we inadvertently changing the semantics here?

The intention of the original code seemed to be to evaluate the "else" path only if std::nullopt. i.e. If batched_decompress_get_temp_size_ex() returned nvcompErrorInternal, it was not to call batched_decompress_get_temp_size().

In the new version, batched_decompress_get_temp_size() is called in both cases. @abellina , @vuule, is that ok?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like what we should be checking is if nvcomp_status is simply nullopt, then doing the second call.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like what we should be checking is if nvcomp_status is simply nullopt, then doing the second call.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah - I'm not sure how much value there is in calling the second if the first failed in the library. It will likely also fail.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reason against trying the old API if the new one failed. Agreed that it's unlikely to help.

compression, num_chunks, max_uncomp_chunk_size, &temp_size));
size_t temp_size = 0;
auto nvcomp_status = batched_decompress_get_temp_size_ex(
compression, num_chunks, max_uncomp_chunk_size, &temp_size, max_total_uncomp_size);

if (nvcomp_status.value_or(nvcompStatus_t::nvcompErrorInternal) !=
nvcompStatus_t::nvcompSuccess) {
nvcomp_status =
batched_decompress_get_temp_size(compression, num_chunks, max_uncomp_chunk_size, &temp_size);
}

CUDF_EXPECTS(nvcomp_status == nvcompStatus_t::nvcompSuccess,
mythrocks marked this conversation as resolved.
Show resolved Hide resolved
"Unable to get scratch size for decompression");
Expand Down