AVRO-4290: [python] Enforce a maximum decompressed block size#3850
AVRO-4290: [python] Enforce a maximum decompressed block size#3850iemejia wants to merge 3 commits into
Conversation
When reading a data file, each block is decompressed according to the file's codec. A block with a very high compression ratio (or a malformed block) could expand to far more memory than its compressed size. Enforce a configurable maximum decompressed size across the deflate, bzip2, snappy and zstandard codecs, mirroring the Java SDK's decompression limit (AVRO-4247). The limit defaults to 200 MiB and can be overridden with the AVRO_MAX_DECOMPRESS_LENGTH environment variable; exceeding it raises AvroDecompressionSizeException. Assisted-by: GitHub Copilot:claude-opus-4.8
There was a problem hiding this comment.
Pull request overview
This PR hardens the Python Avro data-file reader against decompression bombs by enforcing a configurable maximum decompressed block size (default 200 MiB, overrideable via AVRO_MAX_DECOMPRESS_LENGTH) and raising avro.errors.AvroDecompressionSizeException when the limit is exceeded.
Changes:
- Add
AvroDecompressionSizeExceptionand enforce a maximum decompressed block size across deflate, bzip2, snappy, and zstandard codecs. - Introduce
AVRO_MAX_DECOMPRESS_LENGTHenv override and a default limit of 200 MiB inavro.codecs. - Add unit tests covering over-limit rejection and within-limit decoding for available codecs.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| lang/py/avro/codecs.py | Adds max-decompressed-size enforcement and env-configurable limit across supported codecs. |
| lang/py/avro/errors.py | Introduces the new exception type raised when decompressed output exceeds the configured cap. |
| lang/py/avro/test/test_codecs.py | Adds coverage ensuring blocks over the limit are rejected and normal blocks still decode correctly. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| uncompressed = decompressor.decompress(data, limit + 1) | ||
| if len(uncompressed) > limit: | ||
| _raise_decompression_too_large(limit) | ||
| uncompressed += decompressor.flush() | ||
| if len(uncompressed) > limit: | ||
| _raise_decompression_too_large(limit) |
There was a problem hiding this comment.
Fixed in 638aac5 — DeflateCodec.decompress now accumulates into a bytearray, so flush() output is appended in place instead of creating an extra full-size copy of the decompressed data.
| uncompressed.extend(chunk) | ||
| if len(uncompressed) > limit: | ||
| _raise_decompression_too_large(limit) |
There was a problem hiding this comment.
Fixed in 638aac5 — the zstandard path now checks len(uncompressed) + len(chunk) > limit before extending, so the buffer never grows past the limit.
…fore-extend - DeflateCodec.decompress accumulates into a bytearray so the flush() output is appended in place instead of creating an extra full-size copy of the already decompressed data. - ZstandardCodec.decompress checks len(uncompressed) + len(chunk) before extending, so the buffer never grows past the configured limit. Assisted-by: GitHub Copilot:claude-opus-4.8
| decompressor = bz2.BZ2Decompressor() | ||
| uncompressed = decompressor.decompress(data, limit + 1) | ||
| if len(uncompressed) > limit: | ||
| _raise_decompression_too_large(limit) | ||
| return avro.io.BinaryDecoder(io.BytesIO(uncompressed)) |
There was a problem hiding this comment.
Good catch. Reworked BZip2Codec.decompress to verify the decompressor reached end-of-stream and to drain any trailing data, looping over concatenated bzip2 streams so a multi-stream block is fully decoded. Added a test for a truncated block. Pushed in 9f3c49d.
BZip2Codec.decompress now loops the BZ2Decompressor: it drains all buffered output (bounded by the limit), verifies the stream reached EOF (rejecting a truncated/corrupt block with InvalidAvroBinaryEncoding), and handles concatenated bzip2 streams as bz2.decompress does. Add a truncated-block test. Assisted-by: GitHub Copilot:claude-opus-4.8
What is the purpose of the change
The deflate and bzip2 codecs are inflated incrementally and rejected once the output would exceed the limit; snappy rejects an over-large declared length up front; zstandard caps its streaming loop. Exceeding the limit raises
avro.errors.AvroDecompressionSizeException.When reading a data file, each block is decompressed according to the file's codec. A block with a very high compression ratio (or a malformed block) could expand to far more memory than its compressed size (a decompression bomb). This enforces a configurable maximum decompressed size while reading each block, mirroring the Java SDK's decompression limit (AVRO-4247). The limit defaults to 200 MiB and can be overridden with the
AVRO_MAX_DECOMPRESS_LENGTHenvironment variable.This is part of the umbrella issue AVRO-4283.
Verifying this change
This change added tests and can be verified as follows:
avro/test/test_codecs.py(TestDecompressionSizeLimit) covering deflate and bzip2 over-limit rejection and within-limit decoding (snappy/zstandard covered when those optional deps are installed).cd lang/py && python3 -m unittest avro.test.test_codecsDocumentation
AVRO_MAX_DECOMPRESS_LENGTHenvironment variable is documented in code comments)