[C++] arrow::util::Codec::Decompress() doesn't return decompressed data size

[asfimport]

We can't know decompressed data size when we only have compressed data. The current arrow::util::Codec::Decompress() doesn't return decompressed data size. So we can't know which data in output_buffer can be used.

FYI: arrow::util::Codec::Compress() returns compressed data size.

Reporter: Kouhei Sutou / @kou
Assignee: Kouhei Sutou / @kou

PRs and other links:

• GitHub Pull Request #3024

_{Note: This issue was originally created as ARROW-3831. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
The current use case is that you already know the decompressed size, because the information was recorded somewhere (in e.g. a Parquet file).

[asfimport]

Wes McKinney / @wesm:
Do compression libraries provide the decompressed size consistently? Knowing the decompressed size a priori is useful to be able to pre-allocate memory

[asfimport]

Kouhei Sutou / @kou:
I think that the actual decompressed size even when we know the expected decompressed size.
For example, we can validate the decompression result. If the actual decompressed size and the expected one is different, the compressed data will be broken (or decompress logic is broken).

As far as I know, zlib, LZ4 and Zstandard return the actual decompressed size.

[asfimport]

Wes McKinney / @wesm:
What do you think about adding a second Decompress virtual method that will return the output length?

https://github.com/apache/arrow/blob/master/cpp/src/arrow/util/compression.h#L106

The default implementation could be NotImplemented for codecs that do not support this

[asfimport]

Antoine Pitrou / @pitrou:
That sounds ok to me.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 3024
#3024