AVRO-4285: [c++] Enforce a maximum decompressed block size by iemejia · Pull Request #3854 · apache/avro

iemejia · 2026-07-11T13:04:06Z

What is the purpose of the change

DataFileReaderBase::readDataBlock bounds decompression across codecs: snappy and zstandard reject an over-large declared/frame size up front, while deflate and the zstd streaming path cap buffer growth and reject once the output would exceed the limit. Exceeding the limit throws an avro::Exception.

When reading a data file, each block is decompressed according to the file's codec. A block with a very high compression ratio (or a malformed block) could expand to far more memory than its compressed size (a decompression bomb). This enforces a configurable maximum decompressed size while reading each block, mirroring the Java SDK's decompression limit (AVRO-4247). The limit defaults to 200 MiB and can be overridden with the AVRO_MAX_DECOMPRESS_LENGTH environment variable.

This is part of the umbrella issue AVRO-4283.

Verifying this change

This change added tests and can be verified as follows:

Added test/DecompressionLimitTests.cc (registered via unittest(DecompressionLimitTests)) covering deflate/snappy/zstandard over-limit rejection and within-limit decoding.
Run: ctest -R DecompressionLimitTests

Documentation

Does this pull request introduce a new feature? (no — hardening/robustness)
If yes, how is the feature documented? (not applicable; the new AVRO_MAX_DECOMPRESS_LENGTH environment variable is documented in code comments)

When reading a data file, DataFileReaderBase::readDataBlock decompresses each block according to the file's codec. A block with a very high compression ratio (or a malformed block) could expand to far more memory than its compressed size. Enforce a configurable maximum decompressed size across the deflate, snappy and zstandard codecs, mirroring the Java SDK's decompression limit (AVRO-4247): snappy and zstd reject an over-large declared/frame size up front, while deflate and the zstd streaming path cap buffer growth and reject once the output would exceed the limit. The limit defaults to 200 MiB and can be overridden with the AVRO_MAX_DECOMPRESS_LENGTH environment variable. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

This PR hardens the Avro C++ data-file reader against decompression bombs by enforcing a configurable maximum decompressed block size (default 200 MiB, overridable via AVRO_MAX_DECOMPRESS_LENGTH) across supported codecs.

Changes:

Add AVRO_MAX_DECOMPRESS_LENGTH handling and enforce maximum decompressed size during Snappy/Zstandard/Deflate block decoding.
Update Zstandard decompression API to take an explicit maxLength and reject oversized frames.
Add a new Boost unit test suite to validate over-limit rejection and within-limit decoding, and register it in CMake.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
lang/c++/impl/DataFile.cc	Adds env-configured decompression limit and enforces it in Snappy/Zstd/Deflate decode paths.
lang/c++/impl/ZstdDecompressWrapper.hh	Changes Zstd wrapper API to require a `maxLength` for bounded decompression.
lang/c++/impl/ZstdDecompressWrapper.cc	Enforces max decompressed size for both known-size and streaming Zstd frames.
lang/c++/test/DecompressionLimitTests.cc	Adds tests for over-limit rejection and within-limit reads across codecs.
lang/c++/CMakeLists.txt	Registers the new `DecompressionLimitTests` test executable.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

iemejia · 2026-07-11T15:40:28Z

 #include "DataFile.hh"
 #include "Compiler.hh"
+#include <cstdlib>
 #include <cstring>
 #include "Exception.hh"


Fixed in 66809e0 — added and to DataFile.cc.

iemejia · 2026-07-11T15:40:29Z

+size_t maxDecompressLength() {
+    const char *env = std::getenv("AVRO_MAX_DECOMPRESS_LENGTH");
+    if (env != nullptr && *env != '\0') {
+        errno = 0;
+        char *end = nullptr;
+        unsigned long long value = std::strtoull(env, &end, 10);
+        if (errno == 0 && end != nullptr && *end == '\0' && value > 0) {
+            return static_cast<size_t>(value);
+        }
+    }
+    return defaultMaxDecompressLength;
+}


Fixed in 66809e0 — maxDecompressLength rejects a leading sign and clamps to std::numeric_limits<size_t>::max(), so a huge configured value cannot truncate on 32-bit builds.

iemejia · 2026-07-11T15:40:30Z

            uncompressed.append(tmp.data(), out.pos);
+            // Reject a block that decompresses to more than the allowed maximum.
+            if (uncompressed.size() > maxLength) {
+                throw Exception(
+                    "Decompressed block size exceeds the maximum allowed of {} bytes", maxLength);
+            }


Fixed in 66809e0 — the streaming zstd path now checks the limit before appending (out.pos > maxLength - uncompressed.size()), so the buffer never grows past the limit.

iemejia · 2026-07-11T15:40:32Z

+    try {
+        DataFileWriter<std::string> writer(path.c_str(), schema, 64 * 1024 * 1024, codec);
+        writer.write(big);
+        writer.close();
+    } catch (const Exception &) {
+        // Codec not available in this build; nothing to exercise.
+        std::filesystem::remove(path);
+        return;
+    }


Fixed in 66809e0 — the test no longer swallows writer exceptions (real failures surface); snappy/zstd cases are gated on the SNAPPY_CODEC_AVAILABLE / ZSTD_CODEC_AVAILABLE macros instead.

…ests - Include <cerrno> and <limits> in DataFile.cc (used by maxDecompressLength). - maxDecompressLength rejects a leading sign and clamps to size_t's max so a huge configured value cannot truncate on 32-bit builds. - The streaming (unknown content size) Zstandard path checks the limit before appending, so the buffer never grows past the configured maximum. - The decompression-limit test no longer swallows writer exceptions (real failures now surface); snappy/zstd cases are gated on the codec-availability macros instead. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

iemejia · 2026-07-11T17:07:49Z

+size_t maxDecompressLength() {
+    const char *env = std::getenv("AVRO_MAX_DECOMPRESS_LENGTH");
+    if (env != nullptr && *env != '\0') {
+        errno = 0;
+        char *end = nullptr;
+        unsigned long long value = std::strtoull(env, &end, 10);
+        // Reject a leading sign (strtoull would otherwise wrap it) and clamp to
+        // what size_t can represent so a huge value does not truncate on 32-bit
+        // (or smaller size_t) builds.
+        if (errno == 0 && end != nullptr && *end == '\0' && value > 0 && env[0] != '-') {
+            if (value > std::numeric_limits<size_t>::max()) {
+                return std::numeric_limits<size_t>::max();
+            }
+            return static_cast<size_t>(value);
+        }
+    }
+    return defaultMaxDecompressLength;
+}


Fixed. maxDecompressLength() now skips leading whitespace before rejecting an explicit sign, so a value like " -5" is caught (strtoull accepts leading whitespace and would otherwise wrap the negative). Pushed in 710bbe2.

iemejia · 2026-07-11T17:07:51Z

+        {
+            // Reject an over-large block before allocating for it, based on the
+            // uncompressed length declared in the Snappy block header.
+            size_t declared = 0;
+            if (snappy::GetUncompressedLength(reinterpret_cast<const char *>(compressed_.data()),
+                                              len - 4, &declared) &&
+                declared > maxDecompressLength()) {
+                throw Exception(
+                    "Decompressed block size {} exceeds the maximum allowed of {} bytes",
+                    declared, maxDecompressLength());
+            }
+        }


Fixed. The Snappy path now evaluates maxDecompressLength() once into a local and reuses it for both the check and the error message. Pushed in 710bbe2.

iemejia · 2026-07-11T17:07:52Z

+    setDecompressLimit("1048576"); // 1 MiB, smaller than the 4 MiB block
+
+    bool rejected = false;
+    try {
+        DataFileReader<std::string> reader(path.c_str(), schema);
+        std::string out;
+        reader.read(out); // triggers block decompression
+    } catch (const Exception &) {
+        rejected = true;
+    }
+    std::filesystem::remove(path);
+    BOOST_CHECK_MESSAGE(rejected, std::string("codec not bounded: ") + name);
+}


Fixed. Added a RAII DecompressLimitGuard that saves and restores AVRO_MAX_DECOMPRESS_LENGTH (unsetting it if it was previously unset), applied to every test that overrides it, so the override no longer leaks into other test cases. Pushed in 710bbe2.

iemejia · 2026-07-11T17:07:53Z

+    setDecompressLimit("1048576");
+
+    std::string out;
+    {
+        DataFileReader<std::string> reader(path.c_str(), schema);
+        BOOST_CHECK(reader.read(out));
+    }


Same RAII guard applied here so this test restores the previous AVRO_MAX_DECOMPRESS_LENGTH value on exit. Pushed in 710bbe2.

- maxDecompressLength() now skips leading whitespace before rejecting an explicit sign, so a value like ' -5' is caught (strtoull accepts leading whitespace and would otherwise wrap a negative into a huge unsigned value). - The Snappy path evaluates maxDecompressLength() once and reuses it for both the check and the error message. - The tests use a RAII guard that saves and restores AVRO_MAX_DECOMPRESS_LENGTH so an override does not leak into other test cases sharing the process. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

iemejia · 2026-07-11T17:39:05Z

+            // Reject before appending so the buffer never grows past the limit.
+            if (out.pos > maxLength - uncompressed.size()) {
+                throw Exception(
+                    "Decompressed block size exceeds the maximum allowed of {} bytes", maxLength);
+            }


Fixed. The zstd streaming path now includes the would-be decompressed size (current size plus the pending chunk) in the exception message, matching the fixed-size zstd and Snappy paths. Pushed in 7ed107b.

The Zstandard streaming (unknown frame size) path reported only the limit when rejecting an over-large block, unlike the fixed-size zstd and Snappy paths. It now also reports the would-be decompressed size (current size plus the pending chunk), matching the other messages and easing debugging and log correlation. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

lang/c++/impl/ZstdDecompressWrapper.cc:33

ZSTD_getFrameContentSize() returns an unsigned long long, but the result is stored in a size_t. On platforms where size_t is narrower, this can truncate the declared frame size and lead to incorrect size comparisons (and potentially misleading errors). Store the frame size in an unsigned long long and only convert to size_t after enforcing the limit.

    // Get the decompressed size
    size_t decompressed_size = ZSTD_getFrameContentSize(compressed.data(), compressed.size());
    if (decompressed_size == ZSTD_CONTENTSIZE_ERROR) {
        throw Exception("ZSTD: Not a valid compressed frame");

lang/c++/impl/ZstdDecompressWrapper.cc:72

After changing the frame size variable to unsigned long long, the code passes it to std::string::resize() and compares it to result (both size_t). Making the narrowing conversion explicit improves clarity and avoids potential -Wconversion build warnings.

        // Batch decompress the data
        uncompressed.resize(decompressed_size);
        size_t result = ZSTD_decompress(
            uncompressed.data(), decompressed_size, compressed.data(), compressed.size());

        if (ZSTD_isError(result)) {
            throw Exception("ZSTD decompression error: {}", ZSTD_getErrorName(result));
        }
        if (result != decompressed_size) {
            throw Exception("ZSTD: Decompressed size mismatch: expected {}, got {}",
                            decompressed_size, result);
        }

github-actions Bot added the C++ Pull Requests for C++ binding label Jul 11, 2026

iemejia requested a review from Copilot July 11, 2026 14:47

Copilot started reviewing on behalf of iemejia July 11, 2026 14:48 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 16:35

Copilot started reviewing on behalf of iemejia July 11, 2026 16:35 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 17:13

Copilot started reviewing on behalf of iemejia July 11, 2026 17:13 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 17:47

Copilot started reviewing on behalf of iemejia July 11, 2026 17:47 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

Uh oh!

Conversation

iemejia commented Jul 11, 2026

What is the purpose of the change

Verifying this change

Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants