AVRO-4296: [python] Validate available bytes before allocating for length-prefixed values#3861
AVRO-4296: [python] Validate available bytes before allocating for length-prefixed values#3861iemejia wants to merge 6 commits into
Conversation
…ngth-prefixed values and collections A bytes or string value is a length prefix followed by that many bytes, and an array or map block is an element count followed by that many items. A malicious or truncated input can declare a huge length or count with little or no data. - BinaryDecoder.bytes_remaining() reports the bytes still readable for a seekable reader (else None). read() uses it to reject an over-large declared length above a threshold before allocating. - DatumReader.read_array/read_map reject a block whose element count could not be backed by the bytes remaining, using min_bytes_per_element() computed from the element schema so a zero-byte element type (e.g. null) is not falsely rejected. Mirrors the Java SDK's checks (AVRO-4241). Non-seekable readers, whose remaining length is unknown, are unaffected. Assisted-by: GitHub Copilot:claude-opus-4.8
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
This PR hardens the Python Avro binary decoding path against malicious or truncated inputs that declare excessively large length/count prefixes, by validating declared sizes against bytes remaining for seekable inputs before allocating/iterating.
Changes:
- Add
BinaryDecoder.bytes_remaining()and use it to pre-reject oversizedread(n)requests (above a threshold) when the reader is seekable. - Add minimum on-wire-size estimation for schemas and use it to validate array/map block counts in
DatumReaderagainst remaining bytes. - Add unit tests covering oversized length prefixes, oversized collection block counts, and a non-false-positive case (array of
null).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| lang/py/avro/io.py | Adds remaining-bytes introspection plus pre-checks for large length-prefixed reads and collection block count validation using per-element minimum sizes. |
| lang/py/avro/test/test_io.py | Adds targeted tests for the new available-bytes validation behavior in BinaryDecoder and DatumReader (including array-of-nulls). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| remaining = decoder.bytes_remaining() | ||
| if remaining is not None and count * min_bytes_per_element > remaining: | ||
| raise avro.errors.InvalidAvroBinaryEncoding( | ||
| f"Collection claims {count} elements with at least {min_bytes_per_element} bytes each, but only {remaining} bytes are available." | ||
| ) |
There was a problem hiding this comment.
Fixed in 957d180: _ensure_collection_available now compares count against remaining // min_bytes_per_element instead of multiplying, avoiding a large intermediate product for an attacker-controlled count.
Review feedback: - avro/io.py imported the standard library 'io' module, which CodeQL flags as a module importing itself. Import 'os' and use os.SEEK_END instead (both equal 2). - _ensure_collection_available compares count against remaining // min bytes per element rather than multiplying, so an attacker-controlled (unbounded) count does not create a huge intermediate product. Assisted-by: GitHub Copilot:claude-opus-4.8
| reader = self.reader | ||
| if not getattr(reader, "seekable", None) or not reader.seekable(): | ||
| return None | ||
| pos = reader.tell() | ||
| end = reader.seek(0, os.SEEK_END) | ||
| reader.seek(pos) | ||
| return end - pos |
There was a problem hiding this comment.
Fixed in f415f1f: bytes_remaining() now wraps tell()/seek() in try/except (catching OSError/ValueError/AttributeError), uses tell() for the end offset, and validates both are ints, returning None on any failure.
| decoder.skip_long() | ||
| self._ensure_collection_available(decoder, block_count, _min_bytes_per_element(writers_schema.items)) | ||
| for i in range(block_count): |
There was a problem hiding this comment.
Fixed in f415f1f: read_array computes the per-element minimum once before the block loop.
| # Map keys are strings (>= 1 byte length prefix) plus the value. | ||
| self._ensure_collection_available(decoder, block_count, 1 + _min_bytes_per_element(writers_schema.values)) | ||
| for i in range(block_count): |
There was a problem hiding this comment.
Fixed in f415f1f: read_map computes 1 + min-bytes once before the block loop.
…n minimum once Review feedback: - bytes_remaining() now wraps tell()/seek() in try/except and uses tell() for the end offset, returning None on any failure (non-seekable reader, seek() that returns None, or a reader without seekable()). This keeps reads unaffected when the remaining size cannot be determined. - read_array/read_map compute the per-element minimum once before the block loop instead of on every block, avoiding repeated schema traversal. Assisted-by: GitHub Copilot:claude-opus-4.8
The Typechecks CI step (mypy) flagged the isinstance(pos/end, int) check as unreachable: reader is typed IO[bytes], so tell() already returns int, making the guard always true and the following return None unreachable. The try/except around tell()/seek() already provides the resilience the check was meant to add, so remove the redundant guard. Assisted-by: GitHub Copilot:claude-opus-4.8
| reader = self.reader | ||
| try: | ||
| pos = reader.tell() | ||
| reader.seek(0, os.SEEK_END) | ||
| end = reader.tell() | ||
| reader.seek(pos) | ||
| except (OSError, ValueError, AttributeError): | ||
| # Not seekable, or the position/size could not be determined. | ||
| return None | ||
| return end - pos |
There was a problem hiding this comment.
Fixed in b891ac3: bytes_remaining() now restores the original position in a finally block, so a failure after seeking to the end never leaves the reader at EOF. Added tests for the restore on both success and failure. (The isinstance type-validation was dropped because reader is typed IO[bytes], so tell() returns int and mypy flags the check as unreachable; the try/except covers the runtime failure paths.)
Review feedback: bytes_remaining() could leave the reader positioned at EOF if tell()/seek() failed after seeking to the end, corrupting subsequent decoding. Move the restore seek into a finally block so the original position is always restored. Added tests that the position is restored on both success and when reading the end offset fails. Assisted-by: GitHub Copilot:claude-opus-4.8
| try: | ||
| reader.seek(pos) | ||
| except (OSError, ValueError): |
There was a problem hiding this comment.
Fixed in 1020612: the finally block now also catches AttributeError when restoring the position, so a reader that implements tell() but not seek() reliably falls back to None instead of raising.
| def __init__(self, data: bytes) -> None: | ||
| super().__init__(data) | ||
| self._calls = 0 | ||
|
|
There was a problem hiding this comment.
Removed the unused self._calls assignment from the FailingEndStream test helper in 1020612.
… dead test assignment Review feedback: - The finally block in bytes_remaining() only caught (OSError, ValueError) when restoring the position. A reader that implements tell() but not seek() would raise AttributeError from reader.seek(pos) and let it escape; catch AttributeError there too so the method reliably falls back to None. - Removed the unused self._calls assignment from the FailingEndStream test helper. Assisted-by: GitHub Copilot:claude-opus-4.8
What is the purpose of the change
A
bytesorstringvalue is encoded as a length prefix followed by that many bytes of data, and anarrayormapblock is encoded as an element count followed by that many items. A malicious or truncated input can declare a very large length or count while carrying little or no actual data, which causes a correspondingly large allocation before the shortfall is noticed.This applies the equivalent of the Java SDK fix AVRO-4241 to the Python SDK: when the source can report how many bytes remain, a declared length (or a collection block count) that exceeds the bytes actually available is rejected before allocating for it. The collection check uses the minimum on-wire size of the element schema, so a zero-byte element type (such as
null) is never falsely rejected. Sources that cannot report their remaining size are unaffected.BinaryDecoder.bytes_remaining()reports the bytes still readable for a seekable reader (elseNone).read()rejects an over-large declared length above a threshold, andDatumReader.read_array/read_mapreject a block whose element count could not be backed by the bytes remaining, usingmin_bytes_per_element()from the element schema.This is a sub-task of AVRO-4292 and resolves AVRO-4296.
Verifying this change
This change added tests and can be verified as follows:
TestBinaryDecoderAvailableBytesandTestDatumReaderCollectionAvailableBytesinlang/py/avro/test/test_io.py, including an array of nulls that must not be falsely rejected.cd lang/py && python3 -m unittest avro.test.test_ioDocumentation