You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are seeing an issue where trying to read a zstd parquet file is causing a very large allocation to happen (something much higher than GPU memory like 50 or 60 GB).
We have bisected it to a change in this PR: #11652.
Specifically we see that value_or is evaluating the argument in this function batched_decompress_temp_size (changed in the PR above), where the author didn't mean this:
And so temp_size is getting set twice, the second time to the wrong value. This leads to an OOM when nvcomp tries to allocate the temporary space for the decompression.
The text was updated successfully, but these errors were encountered:
I dug up some numbers from when we first encountered the problem with nvcompBatchedZstdDecompressGetTempSize. This is before nvcomp was updated to add nvcompBatchedZstdDecompressGetTempSizeEx. For q23b from tpcds running at 3TB, I saw numbers like this:
We are seeing an issue where trying to read a zstd parquet file is causing a very large allocation to happen (something much higher than GPU memory like 50 or 60 GB).
We have bisected it to a change in this PR: #11652.
Specifically we see that
value_or
is evaluating the argument in this functionbatched_decompress_temp_size
(changed in the PR above), where the author didn't mean this:And so
temp_size
is getting set twice, the second time to the wrong value. This leads to an OOM when nvcomp tries to allocate the temporary space for the decompression.The text was updated successfully, but these errors were encountered: