AVRO-4297: [ruby] Validate available bytes before allocating for length-prefixed values by iemejia · Pull Request #3862 · apache/avro

iemejia · 2026-07-11T21:20:21Z

What is the purpose of the change

A bytes or string value is encoded as a length prefix followed by that many bytes of data, and an array or map block is encoded as an element count followed by that many items. A malicious or truncated input can declare a very large length or count while carrying little or no actual data, which causes a correspondingly large allocation before the shortfall is noticed.

This applies the equivalent of the Java SDK fix AVRO-4241 to the Ruby SDK: when the source can report how many bytes remain, a declared length (or a collection block count) that exceeds the bytes actually available is rejected before allocating for it. The collection check uses the minimum on-wire size of the element schema, so a zero-byte element type (such as null) is never falsely rejected. Sources that cannot report their remaining size are unaffected.

BinaryDecoder#bytes_remaining reports the bytes still readable when the reader can report its size (else nil). #read rejects an over-large declared length above a threshold, and DatumReader#read_array/#read_map reject a block whose element count could not be backed by the bytes remaining, using min_bytes_per_element from the element schema.

This is a sub-task of AVRO-4292 and resolves AVRO-4297.

Verifying this change

This change added tests and can be verified as follows:

Extended lang/ruby/test/test_io.rb with over-limit bytes/array/map rejection and an array of nulls that must not be falsely rejected.
Run: cd lang/ruby && bundle exec rake test

Documentation

Does this pull request introduce a new feature? (no — hardening / robustness)
If yes, how is the feature documented? (not applicable)

…th-prefixed values and collections A bytes or string value is a length prefix followed by that many bytes, and an array or map block is an element count followed by that many items. A malicious or truncated input can declare a huge length or count with little or no data. - BinaryDecoder#bytes_remaining reports the bytes still readable when the reader can report its size (else nil). #read uses it to reject an over-large declared length above a threshold before allocating. - DatumReader#read_array/#read_map reject a block whose element count could not be backed by the bytes remaining, using min_bytes_per_element computed from the element schema so a zero-byte element type (e.g. null) is not falsely rejected. Mirrors the Java SDK's checks (AVRO-4241). Readers that cannot report their size are unaffected. Assisted-by: GitHub Copilot:claude-opus-4.8

The Lint CI step (rubocop) flagged three offenses in the new code: - Lint/IneffectiveAccessModifier: 'private' does not apply to a 'def self.' singleton method. Make min_bytes_per_element a private instance method (it uses no class state) and call it unqualified. - Lint/HashCompareByIdentity: use a hash with compare_by_identity and the schema object as the key instead of keying on schema.object_id. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

This PR hardens the Ruby Avro binary decoding path against malicious/truncated inputs that declare huge length/count prefixes by validating declared sizes against bytes remaining (when the underlying reader can report its size) before performing large allocations.

Changes:

Added BinaryDecoder#bytes_remaining and a thresholded pre-allocation check in BinaryDecoder#read for large length-prefixed reads.
Added array/map block pre-validation in DatumReader#read_array and #read_map using a computed minimum on-wire size per element schema.
Added Ruby tests covering over-limit bytes/array/map rejections and ensuring arrays of null are not falsely rejected.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
lang/ruby/lib/avro/io.rb	Adds bytes-remaining reporting and enforces length/count sanity checks to prevent oversized allocations during decoding.
lang/ruby/test/test_io.rb	Adds regression tests for rejecting over-declared lengths/counts while preserving valid decoding behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

iemejia · 2026-07-11T22:28:59Z

      def read(len)
-        # Read n bytes
+        # Read n bytes. Reject a declared length that exceeds the bytes
+        # actually remaining before allocating for it, to guard against an
+        # out-of-memory attack from a malicious or truncated input. The check
+        # is only applied to larger reads; smaller reads and stream readers that


Fixed in d4eec59: BinaryDecoder#read now rejects a negative length (Ruby IO#read(-1) reads the rest of the stream, which would bypass the size check).

iemejia · 2026-07-11T22:29:00Z

+        when :float then 4
+        when :double then 8
+        when :fixed then schema.size
+        when :record, :error


Fixed in d4eec59: min_bytes_per_element now treats :request like :record/:error, so an array/map of request values whose fields encode to zero bytes is not falsely rejected.

…cord Review feedback: - BinaryDecoder#read now rejects a negative length. Ruby IO#read(-1) reads the rest of the stream, which would bypass the size check and allocate without bound. - min_bytes_per_element treats :request like :record/:error (request schemas are record-like), so an array/map of request values whose fields encode to zero bytes is not falsely rejected. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

iemejia · 2026-07-11T23:06:48Z

+      def bytes_remaining
+        return nil unless @reader.respond_to?(:size) && @reader.respond_to?(:tell)
+        @reader.size - @reader.tell
+      end


Fixed in 34859b4: bytes_remaining now checks that #size/#tell return Integers and rescues IOError/SystemCallError/NotImplementedError, returning nil when the size cannot be reported.

iemejia · 2026-07-11T23:06:50Z

+        return if count <= 0 || min_bytes_per_element <= 0
+        remaining = decoder.bytes_remaining
+        if remaining && count * min_bytes_per_element > remaining
+          raise AvroError, "Collection claims #{count} elements with at least #{min_bytes_per_element} bytes each, but only #{remaining} bytes are available"
+        end


Fixed in 34859b4: ensure_collection_available compares count against remaining / min_bytes_per_element instead of multiplying, avoiding a large intermediate Integer.

Review feedback: - bytes_remaining now verifies #size/#tell return Integers and rescues IOError/SystemCallError/NotImplementedError, returning nil when the remaining size cannot be reported, so reads stay unaffected in that case. - ensure_collection_available compares count against remaining / min_bytes_per_element rather than multiplying, avoiding a large intermediate Integer for an attacker-controlled count. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

iemejia · 2026-07-11T23:30:41Z

+        size = @reader.size
+        pos = @reader.tell
+        return nil unless size.is_a?(Integer) && pos.is_a?(Integer)
+        size - pos


Fixed in ea1ac32: bytes_remaining clamps a negative result to 0 via [size - pos, 0].max, so a truncated file yields 'no bytes available' instead of a negative count.

iemejia · 2026-07-11T23:30:43Z

            block_count = -block_count
            _block_size = decoder.read_long
          end
+          ensure_collection_available(decoder, block_count, min_bytes_per_element(writers_schema.items))


Fixed in ea1ac32: read_array computes the per-element minimum once per invocation and reuses it across blocks.

iemejia · 2026-07-11T23:30:44Z

            _block_size = decoder.read_long
          end
+          # Map keys are strings (>= 1 byte length prefix) plus the value.
+          ensure_collection_available(decoder, block_count, 1 + min_bytes_per_element(writers_schema.values))


Fixed in ea1ac32: read_map computes 1 + min-bytes once per invocation and reuses it across blocks.

…nimum Review feedback: - bytes_remaining clamps a negative result (size smaller than the current position, e.g. a truncated file) to 0 with [size - pos, 0].max, so callers see no bytes available rather than a confusing negative count. - read_array/read_map compute the per-element minimum once per invocation and reuse it across blocks, avoiding repeated schema traversal on the hot path. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

iemejia · 2026-07-12T05:26:19Z

+      def ensure_collection_available(decoder, count, min_bytes_per_element)
+        return if count <= 0 || min_bytes_per_element <= 0
+        remaining = decoder.bytes_remaining
+        # Compare via integer division rather than multiplying, so a huge count
+        # does not create a large intermediate product.
+        if remaining && count > remaining / min_bytes_per_element
+          raise AvroError, "Collection claims #{count} elements with at least #{min_bytes_per_element} bytes each, but only #{remaining} bytes are available"
+        end
+      end


Fixed in 1186853: ensure_collection_available now guards the call with respond_to?(:bytes_remaining) and skips the check for a decoder that does not implement it, instead of raising NoMethodError.

…s_remaining Review feedback: a decoder that implements the read protocol but not #bytes_remaining (e.g. a custom decoder) would raise NoMethodError. Guard the call with respond_to?(:bytes_remaining) and skip the check for such a decoder instead of raising. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

iemejia · 2026-07-12T06:02:31Z

+  end
+
+  def test_read_bytes_within_stream_still_reads
+    payload = 'x' * (2 * 1024 * 1024)


Fixed in 1a3d3da: the payload is now 'x' * (Avro::IO::BinaryDecoder::MAX_UNCHECKED_READ + 1), so it still exercises the large-read path but with less time and memory.

iemejia · 2026-07-12T06:02:30Z

+  # report its size, that is rejected before allocating for it.
+  def test_read_bytes_rejects_length_beyond_stream
+    writer = StringIO.new
+    Avro::IO::BinaryEncoder.new(writer).write_long(100 * 1024 * 1024)


Fixed in 1a3d3da: the declared length is now Avro::IO::BinaryDecoder::MAX_UNCHECKED_READ + 1, tying the test directly to the threshold under test instead of a hard-coded 100MiB constant.

iemejia · 2026-07-12T06:02:33Z

+    count = 100_000
+    writer = StringIO.new
+    encoder = Avro::IO::BinaryEncoder.new(writer)
+    encoder.write_long(count) # one block of `count` nulls (zero bytes each)
+    encoder.write_long(0)     # end-of-array marker
+    result = decode('{"type":"array","items":"null"}', writer.string)
+    assert_equal([nil] * count, result)
+  end


Fixed in 1a3d3da: the test now asserts the decoded length and spot-checks the first and last elements are nil, instead of building a second large [nil] * count array.

… null assertion Review feedback: - Derive the oversized declared length and the within-limit payload from BinaryDecoder::MAX_UNCHECKED_READ + 1 instead of hard-coded 100MiB/2MiB constants, so the tests stay tied to the threshold under test and use less time and memory. - Assert the decoded array length and spot-check a couple of nil elements instead of building a second large [nil] * count array. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

github-actions Bot added the Ruby label Jul 11, 2026

iemejia requested a review from Copilot July 11, 2026 21:43

Copilot started reviewing on behalf of iemejia July 11, 2026 21:43 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 22:35

Copilot started reviewing on behalf of iemejia July 11, 2026 22:36 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 23:08

Copilot started reviewing on behalf of iemejia July 11, 2026 23:09 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 12, 2026 05:04

Copilot started reviewing on behalf of iemejia July 12, 2026 05:05 View session

Copilot AI reviewed Jul 12, 2026

View reviewed changes

iemejia requested a review from Copilot July 12, 2026 05:35

Copilot started reviewing on behalf of iemejia July 12, 2026 05:35 View session

Copilot AI reviewed Jul 12, 2026

View reviewed changes

iemejia requested a review from Copilot July 12, 2026 06:46

Copilot started reviewing on behalf of iemejia July 12, 2026 06:46 View session

Copilot AI reviewed Jul 12, 2026

View reviewed changes

Uh oh!

Conversation

iemejia commented Jul 11, 2026

What is the purpose of the change

Verifying this change

Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants