AVRO-4293: [c] Validate available bytes before allocating for length-prefixed values by iemejia · Pull Request #3858 · apache/avro

iemejia · 2026-07-11T21:20:14Z

What is the purpose of the change

A bytes or string value is encoded as a length prefix followed by that many bytes of data, and an array or map block is encoded as an element count followed by that many items. A malicious or truncated input can declare a very large length or count while carrying little or no actual data, which causes a correspondingly large allocation before the shortfall is noticed.

This applies the equivalent of the Java SDK fix AVRO-4241 to the C SDK: when the source can report how many bytes remain, a declared length (or a collection block count) that exceeds the bytes actually available is rejected before allocating for it. The collection check uses the minimum on-wire size of the element schema, so a zero-byte element type (such as null) is never falsely rejected. Sources that cannot report their remaining size are unaffected.

avro_reader_bytes_available() returns the number of bytes still readable from a memory-backed reader (or -1 when unknown, e.g. file readers). read_bytes/read_string consult it before allocating, and read_array_value/read_map_value reject a block whose element count could not be backed by the bytes remaining, computing min_bytes_per_element() from the element schema.

This is a sub-task of AVRO-4292 and resolves AVRO-4293.

Verifying this change

This change added tests and can be verified as follows:

Added lang/c/tests/test_avro_4293.c covering over-limit string/bytes/array/map rejection, a within-limit value, and an array of nulls that must not be falsely rejected.
Run: cd lang/c && mkdir build && cd build && cmake .. && make && ctest

Documentation

Does this pull request introduce a new feature? (no — hardening / robustness)
If yes, how is the feature documented? (not applicable)

…prefixed values and collections A bytes or string value is a length prefix followed by that many bytes, and an array or map block is an element count followed by that many items. A malicious or truncated input can declare a huge length or count with little or no data, causing a correspondingly huge allocation or an unbounded append loop. - avro_reader_bytes_available() returns the bytes still readable from a memory-backed reader (or -1 when unknown). read_bytes/read_string consult it to reject an over-large declared length before allocating. - read_array_value/read_map_value reject a block whose element count could not be backed by the bytes remaining, using min_bytes_per_element() computed from the element schema so a zero-byte element type (e.g. null) is not falsely rejected. The comparison divides to avoid overflow. Mirrors the Java SDK's checks (AVRO-4241). Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Hardens the C Avro binary decoder against malicious/truncated inputs that declare large bytes/string lengths or array/map block counts by rejecting impossible sizes before allocations when the reader can report remaining bytes (memory-backed readers).

Changes:

Added avro_reader_bytes_available() for memory readers and used it to pre-check declared bytes/string lengths.
Added array/map block-count validation based on min_bytes_per_element() to reject blocks that cannot be backed by remaining bytes.
Added a new regression test (test_avro_4293) and wired it into the C test suite.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
lang/c/tests/test_avro_4293.c	Adds tests intended to cover early rejection for oversized length/count cases and validates non-rejection for zero-byte elements (`null`).
lang/c/tests/CMakeLists.txt	Registers the new `test_avro_4293` in the test suite.
lang/c/src/value-read.c	Adds per-element minimum sizing and collection block-count validation before appending/allocating decoded elements.
lang/c/src/io.c	Implements `avro_reader_bytes_available()` for memory-backed readers (returns -1 for unknown/file).
lang/c/src/encoding.h	Declares `avro_reader_bytes_available()` for internal use.
lang/c/src/encoding_binary.c	Adds pre-allocation checks for `read_bytes` and `read_string` using remaining-byte reporting.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

iemejia · 2026-07-11T22:28:23Z

+	if (schema == NULL || depth > 64) {
+		return 0;
+	}


Fixed in 896a688: min_bytes_per_element() now returns 1 (not 0) when the depth guard trips, so the collection check stays enabled for a crafted deep/recursive schema; a valid recursive value always encodes to at least 1 byte.

iemejia · 2026-07-11T22:28:25Z

+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdint.h>
+#include <avro.h>


Fixed in 896a688: the test now includes <errno.h>.

iemejia · 2026-07-11T22:28:26Z

+	rc = try_decode(iface, oversized, sizeof(oversized));
+	if (rc == 0) {
+		fprintf(stderr, "%s: FAIL - oversized length was accepted\n", label);
+	} else {
+		fprintf(stderr, "%s: oversized length rejected as expected: %s\n",
+			label, avro_strerror());
+		ret = EXIT_SUCCESS;
+	}


Fixed in 896a688: the test now asserts the rejection code is EINVAL (what the availability checks return), so it no longer passes on the pre-fix ENOSPC-after-allocation behavior.

… EINVAL Review feedback: - min_bytes_per_element() returned 0 when the depth guard tripped, which made ensure_collection_available() skip the check and could bypass the guard for a crafted deep/recursive schema. Return 1 instead so the check stays enabled; a valid recursive value always encodes to at least 1 byte. - The test now includes <errno.h> and asserts the rejection code is EINVAL (the availability checks' error) rather than accepting any non-zero return, which would also pass on the pre-fix ENOSPC-after-allocation behavior. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

iemejia · 2026-07-11T23:06:25Z

+		for (i = 0; i < n; i++) {
+			avro_schema_t  field =
+			    avro_schema_record_field_get_by_index(schema, i);
+			total += min_bytes_per_element(field, depth + 1);
+		}


Fixed in b549013: the record field-minima sum now saturates to INT64_MAX instead of overflowing, so a wrapped negative total can no longer disable the collection check.

iemejia · 2026-07-11T23:06:26Z

+	/* Map keys are strings (>= 1 byte length prefix) plus the value. */
+	min_bytes = 1 + min_bytes_per_element(
+	    map_schema ? avro_schema_map_values(map_schema) : NULL, 0);


Fixed in b549013: the map computation saturates the key's +1 (only adds when below INT64_MAX), so a maxed-out value minimum cannot wrap.

iemejia · 2026-07-11T23:06:27Z

+	reader = avro_reader_memory(valid, sizeof(valid));
+	rc = avro_value_read(reader, &decoded);


Fixed in b549013: check_accepts_valid() now checks avro_reader_memory() for NULL before use.

iemejia · 2026-07-11T23:06:28Z

+	reader = avro_reader_memory(null_array, sizeof(null_array));
+	rc = avro_value_read(reader, &decoded);


Fixed in b549013: check_accepts_null_array() now checks avro_reader_memory() for NULL before use.

Review feedback: - min_bytes_per_element() now saturates the record field-minima sum to INT64_MAX instead of overflowing (a wrapped negative total would disable the collection check), and the map computation saturates the +1 for the key. - The valid-string and null-array tests now check avro_reader_memory() for NULL before use, so an allocation failure cannot dereference a NULL reader. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Review feedback (consistency with the C++ fix): the depth cutoff was applied before checking the schema type, so a zero-byte leaf type (e.g. null) nested under deeply chained records returned 1 instead of 0, enabling the collection check and potentially rejecting valid data. Move the depth guard into the AVRO_RECORD case; leaf types now return their true minimum (null -> 0) regardless of nesting depth. A cyclic link still terminates because it resolves through a record, which is depth-guarded. Assisted-by: GitHub Copilot:claude-opus-4.8

github-actions Bot added the C label Jul 11, 2026

iemejia requested a review from Copilot July 11, 2026 21:23

Copilot started reviewing on behalf of iemejia July 11, 2026 21:24 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 22:34

Copilot started reviewing on behalf of iemejia July 11, 2026 22:35 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 23:07

Copilot started reviewing on behalf of iemejia July 11, 2026 23:08 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

AVRO-4293: [c] Validate available bytes before allocating for length-prefixed values#3858

AVRO-4293: [c] Validate available bytes before allocating for length-prefixed values#3858
iemejia wants to merge 4 commits into
apache:mainfrom
iemejia:AVRO-4293-c-available-bytes

iemejia commented Jul 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

iemejia Jul 11, 2026

Uh oh!

iemejia Jul 11, 2026

Uh oh!

iemejia Jul 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

iemejia Jul 11, 2026

Uh oh!

iemejia Jul 11, 2026

Uh oh!

iemejia Jul 11, 2026

Uh oh!

iemejia Jul 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		reader = avro_reader_memory(valid, sizeof(valid));
		rc = avro_value_read(reader, &decoded);

		reader = avro_reader_memory(null_array, sizeof(null_array));
		rc = avro_value_read(reader, &decoded);

Uh oh!

Conversation

iemejia commented Jul 11, 2026

What is the purpose of the change

Verifying this change

Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

iemejia Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

iemejia Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

iemejia Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

iemejia Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

iemejia Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

iemejia Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

iemejia Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants