AVRO-4284: [c] Enforce a maximum decompressed block size by iemejia · Pull Request #3853 · apache/avro

iemejia · 2026-07-11T13:03:57Z

What is the purpose of the change

The deflate and lzma codecs cap buffer growth and reject once the output would exceed the limit; snappy rejects an over-large declared length up front. Exceeding the limit fails decoding with a descriptive error.

When reading a data file, each block is decompressed according to the file's codec. A block with a very high compression ratio (or a malformed block) could expand to far more memory than its compressed size (a decompression bomb). This enforces a configurable maximum decompressed size while reading each block, mirroring the Java SDK's decompression limit (AVRO-4247). The limit defaults to 200 MiB and can be overridden with the AVRO_MAX_DECOMPRESS_LENGTH environment variable.

This is part of the umbrella issue AVRO-4283.

Verifying this change

This change added tests and can be verified as follows:

Added tests/test_avro_4284.c (registered in tests/CMakeLists.txt) exercising deflate, snappy and lzma over-limit rejection.
Run: ctest -R test_avro_4284 (also passes under the valgrind memcheck variant)

Documentation

Does this pull request introduce a new feature? (no — hardening/robustness)
If yes, how is the feature documented? (not applicable; the new AVRO_MAX_DECOMPRESS_LENGTH environment variable is documented in code comments)

When reading a data file, each block is decompressed according to the file's codec. A block with a very high compression ratio (or a malformed block) could expand to far more memory than its compressed size. Enforce a configurable maximum decompressed size across the deflate, snappy and lzma codecs, mirroring the Java SDK's decompression limit (AVRO-4247): snappy rejects an over-large declared length up front, while deflate and lzma cap buffer growth and reject once the output would exceed the limit. The limit defaults to 200 MiB and can be overridden with the AVRO_MAX_DECOMPRESS_LENGTH environment variable. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

This PR hardens the C datafile block decompression path against decompression bombs by enforcing a configurable maximum decompressed block size (default 200 MiB, overridable via AVRO_MAX_DECOMPRESS_LENGTH) across the deflate, snappy, and lzma codecs, and adds a regression test covering over-limit rejection.

Changes:

Add AVRO_MAX_DECOMPRESS_LENGTH support and enforce the maximum decompressed block size in lang/c/src/codec.c for snappy/deflate/lzma decoders.
Add a new C test test_avro_4284.c to validate over-limit rejection behavior.
Register the new test in the C test CMake configuration.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
lang/c/src/codec.c	Adds env-configured max decompression length and enforces it during snappy/deflate/lzma decoding.
lang/c/tests/test_avro_4284.c	New test that sets a small max length and asserts codecs reject an oversized decompressed block.
lang/c/tests/CMakeLists.txt	Registers the new test target.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

iemejia · 2026-07-11T15:40:24Z

        if (snappy_uncompressed_length((const char*)data, len-4, &outlen) != SNAPPY_OK) {
 		avro_set_error("Uncompressed length error in snappy");
 		return 1;
        }


Fixed in 3aeb326 — decode_snappy rejects a block shorter than the 4-byte CRC suffix before computing len - 4, and the decompression limit is cached in a local instead of being re-parsed in the error path.

iemejia · 2026-07-11T15:40:25Z

+			c->block_data = avro_realloc(c->block_data, c->block_size, new_size);
 			s->next_out = c->block_data + s->total_out;
-			s->avail_out += c->block_size;
-			c->block_size = c->block_size * 2;
+			s->avail_out += (uInt)(new_size - c->block_size);
+			c->block_size = new_size;


Fixed in 3aeb326 — decode_deflate reallocs into a temporary pointer and fails cleanly (freeing nothing it cannot) on OOM instead of null-dereferencing and losing the original buffer.

iemejia · 2026-07-11T15:40:26Z

+			codec->block_data = avro_realloc(codec->block_data, codec->block_size, new_size);
+			codec->block_size = new_size;


Fixed in 3aeb326 — decode_lzma applies the same temporary-pointer + explicit OOM error pattern.

iemejia · 2026-07-11T15:40:27Z

+	int64_t compressed_len = codec.used_size;
+	char *compressed = (char *) malloc(compressed_len);
+	if (compressed == NULL) {
+		avro_codec_reset(&codec);
+		return 1;
+	}
+	memcpy(compressed, codec.block_data, compressed_len);


Fixed in 3aeb326 — the test casts used_size to size_t for malloc/memcpy and guards against a non-positive length.

- decode_snappy rejects a block shorter than the 4-byte CRC suffix before computing len - 4 (which would underflow), and caches the decompression limit instead of re-parsing the environment in the error path. - decode_deflate and decode_lzma realloc into a temporary pointer and fail cleanly on OOM instead of null-dereferencing and leaking the old buffer. - The test casts the codec's used_size to size_t for malloc/memcpy and guards against a non-positive length. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

iemejia · 2026-07-11T17:07:40Z

+	if ((int64_t) outlen > max_len) {
+		avro_set_error("Decompressed block size %llu exceeds the maximum allowed of %lld bytes",
+			       (unsigned long long) outlen, (long long) max_len);
+		return 1;
+	}


Fixed. The comparison now casts both operands to uint64_t ((uint64_t) outlen > (uint64_t) max_len), so a declared length above INT64_MAX no longer wraps to a negative int64_t and slip past the cap. Pushed in 65d4dcf.

iemejia · 2026-07-11T17:07:41Z

+	memcpy(compressed, codec.block_data, (size_t) compressed_len);
+
+	int rc = avro_codec_decode(&codec, compressed, compressed_len);
+
+	free(compressed);
+	avro_codec_reset(&codec);
+
+	if (rc == 0) {
+		fprintf(stderr, "  codec %s: expected decompression to be rejected but it succeeded\n", name);
+		return 1;
+	}
+	fprintf(stderr, "  codec %s: over-limit block rejected as expected\n", name);
+	return 2;


The test now captures avro_strerror() after the failed decode and asserts the message contains "exceeds the maximum", so it confirms the rejection is specifically the size-limit error and not an unrelated failure. Pushed in 65d4dcf.

- decode_snappy compared (int64_t) outlen > max_len. A declared length above INT64_MAX would wrap to a negative int64_t and slip past the cap; compare as unsigned 64-bit instead. - The test now captures avro_strerror() and asserts the rejection is specifically the 'exceeds the maximum' size-limit error, not an unrelated failure. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

iemejia · 2026-07-11T17:38:52Z

+	const char *env = getenv("AVRO_MAX_DECOMPRESS_LENGTH");
+	if (env != NULL && *env != '\0') {
+		char *end = NULL;
+		long long value = strtoll(env, &end, 10);
+		if (end != NULL && *end == '\0' && value > 0) {
+			return (int64_t) value;
+		}
+	}
+	return AVRO_DEFAULT_MAX_DECOMPRESS_LENGTH;


Fixed. avro_max_decompress_length() now clamps the parsed value to SIZE_MAX (and checks errno), so it is always safe as an allocation cap and cannot truncate when passed to avro_malloc/avro_realloc on 32-bit. Pushed in d280eaf.

iemejia · 2026-07-11T17:38:54Z

 	int err;
+	int64_t max_len = avro_max_decompress_length();
 	z_stream *s = codec_data_inflate_stream(c->codec_data);



Fixed. decode_deflate caps its working max_len to UINT_MAX, so the avail_out update (a uInt) can no longer truncate and leave zlib's state inconsistent. Pushed in d280eaf.

iemejia · 2026-07-11T17:38:55Z

+			int64_t new_size = c->block_size * 2;
+			if (new_size > max_len) {
+				new_size = max_len;
+			}


Fixed. The deflate growth now doubles with a checked pattern (compare block_size against max_len/2 before multiplying), avoiding signed overflow. The UINT_MAX cap above also bounds block_size well below the overflow range. Pushed in d280eaf.

iemejia · 2026-07-11T17:38:56Z

+			int64_t new_size = codec->block_size * 2;
+			if (new_size > max_len) {
+				new_size = max_len;
+			}


Fixed. decode_lzma uses the same checked doubling (compare block_size against max_len/2 before multiplying) to avoid signed overflow. Pushed in d280eaf.

- avro_max_decompress_length() clamps the parsed value to SIZE_MAX (and now checks errno), so a configured limit larger than size_t cannot truncate when passed to avro_malloc/avro_realloc on 32-bit platforms. - decode_deflate caps its working limit to UINT_MAX so the avail_out update (a uInt) cannot truncate and leave zlib's state inconsistent. - decode_deflate and decode_lzma now double the output buffer with a checked pattern (compare against max_len/2 before multiplying) to avoid signed overflow when the buffer has already grown near INT64_MAX/2. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

github-actions Bot added the C label Jul 11, 2026

iemejia requested a review from Copilot July 11, 2026 14:47

Copilot started reviewing on behalf of iemejia July 11, 2026 14:48 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 16:35

Copilot started reviewing on behalf of iemejia July 11, 2026 16:35 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 17:12

Copilot started reviewing on behalf of iemejia July 11, 2026 17:13 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 17:46

Copilot started reviewing on behalf of iemejia July 11, 2026 17:47 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

		codec->block_data = avro_realloc(codec->block_data, codec->block_size, new_size);
		codec->block_size = new_size;

Uh oh!

Conversation

iemejia commented Jul 11, 2026

What is the purpose of the change

Verifying this change

Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants