AVRO-4287: [javascript] Enforce a maximum decompressed block size by iemejia · Pull Request #3852 · apache/avro

iemejia · 2026-07-11T13:03:55Z

What is the purpose of the change

The built-in deflate codec passes the limit to zlib (via maxOutputLength) so the allocation itself is capped, and every codec's output is additionally checked as a safeguard. The limit can also be set per decoder via the maxDecompressLength option.

When reading a data file, each block is decompressed according to the file's codec. A block with a very high compression ratio (or a malformed block) could expand to far more memory than its compressed size (a decompression bomb). This enforces a configurable maximum decompressed size while reading each block, mirroring the Java SDK's decompression limit (AVRO-4247). The limit defaults to 200 MiB and can be overridden with the AVRO_MAX_DECOMPRESS_LENGTH environment variable.

This is part of the umbrella issue AVRO-4283.

Verifying this change

This change added tests and can be verified as follows:

Added tests in lib/test/test_files.js: a deflate block exceeding the limit is rejected, a within-limit block decodes, and the limit is enforced for custom codecs.
Run: cd lang/js && npm test

Documentation

Does this pull request introduce a new feature? (no — hardening/robustness)
If yes, how is the feature documented? (not applicable; the new AVRO_MAX_DECOMPRESS_LENGTH environment variable is documented in code comments)

When reading a data file, each block is decompressed according to the file's codec. A block with a very high compression ratio (or a malformed block) could expand to far more memory than its compressed size. Bound the decompressed size of each block: the built-in deflate codec passes the limit to zlib (via maxOutputLength) so the allocation itself is capped, and every codec's output is additionally checked as a safeguard. Mirrors the Java SDK's decompression limit (AVRO-4247). The limit defaults to 200 MiB and can be set per decoder via the maxDecompressLength option or globally with the AVRO_MAX_DECOMPRESS_LENGTH environment variable. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

This PR hardens the JavaScript Avro container-file BlockDecoder against decompression bombs by enforcing a configurable maximum decompressed block size (default 200 MiB, overridable via AVRO_MAX_DECOMPRESS_LENGTH or per-decoder maxDecompressLength), and adds tests to validate the behavior.

Changes:

Add a default max decompressed block size with env-var override and a per-decoder option.
Apply a decompressed-size safeguard to every codec output, and pass maxOutputLength to zlib for the built-in deflate codec.
Add tests covering over-limit blocks, within-limit blocks, and custom codec enforcement.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
lang/js/lib/files.js	Introduces max decompression length configuration and enforces the limit during block decompression.
lang/js/test/test_files.js	Adds regression tests to ensure oversized blocks are rejected and custom codecs are also constrained.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

iemejia · 2026-07-11T15:40:09Z

+  if (!this._codecs && codec === 'deflate') {
+    decompress = function (buf, cb) {
+      zlib.inflateRaw(buf, {maxOutputLength: maxLength}, cb);
+    };
+  }


Fixed in a4a6680 — the zlib maxOutputLength cap is now applied whenever the resolved codec is zlib.inflateRaw, including a custom codecs map that reuses it. Added a test for that case.

iemejia · 2026-07-11T15:40:10Z

+  this._decompress = function (buf, cb) {
+    decompress(buf, function (err, data) {
+      if (err) {
+        cb(err);
+        return;
+      }
+      if (data && data.length > maxLength) {
+        cb(new Error(f(
+          'decompressed block size exceeds the maximum allowed of %d bytes',
+          maxLength
+        )));
+        return;
+      }
+      cb(null, data);
+    });


Fixed in a4a6680 — _pending is now decremented in the block callback's error path, so a failed decompression (including the size safeguard firing) can no longer leave the stream waiting at finish.

…ing on error - Apply zlib's maxOutputLength cap whenever the resolved codec is zlib.inflateRaw, including a custom codecs map that reuses it (previously the cap was only applied for the built-in default, so custom maps fell back to the too-late post-decompress check). - Decrement _pending in the block callback's error path so a failed decompression (e.g. the size safeguard firing) cannot leave the stream waiting at finish. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

iemejia · 2026-07-11T17:07:30Z

    if (err) {
      self.emit('error', err);
      return;
    }


Fixed. On the error path in _createBlockCallback, if the writable side has already finished and this was the last pending block, the readable side is now ended (push(null)) so a consumer waiting on the queue is not left hanging after the error. Pushed in d1370e7.

iemejia · 2026-07-11T17:07:31Z

+      var encoder = new streams.BlockEncoder(t, {codec: 'deflate'});
+      var decoder = new streams.BlockDecoder({maxDecompressLength: 1024})
+        .on('data', function () {})
+        .on('error', function () { cb(); });


The test now asserts the specific error: zlib's ERR_BUFFER_TOO_LARGE from the maxOutputLength cap. Pushed in d1370e7.

iemejia · 2026-07-11T17:07:32Z

+        maxDecompressLength: 1024
+      })
+        .on('data', function () {})
+        .on('error', function () { cb(); });


Updated to assert err.code === 'ERR_BUFFER_TOO_LARGE', proving the maxOutputLength cap fired rather than any incidental error. Pushed in d1370e7.

iemejia · 2026-07-11T17:07:33Z

+        maxDecompressLength: 1024
+      })
+        .on('data', function () {})
+        .on('error', function () { cb(); });


Updated to assert the generic post-decompress safeguard message (/decompressed block size exceeds/) for the custom codec. Pushed in d1370e7.

… specifics - In the block callback's error path, if the writable side already finished and this was the last pending block, end the readable side (push null) so a consumer waiting on the queue is not left hanging after the error. - The limit tests now assert the specific error: zlib's ERR_BUFFER_TOO_LARGE for the deflate maxOutputLength cap (built-in and custom-codec-reusing- inflateRaw), and the 'decompressed block size exceeds' message for the generic custom-codec safeguard. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

iemejia · 2026-07-11T17:38:37Z

+function getDefaultMaxDecompressLength() {
+  var value = process.env.AVRO_MAX_DECOMPRESS_LENGTH;
+  if (value !== undefined) {
+    var parsed = parseInt(value, 10);
+    if (!isNaN(parsed) && parsed > 0) {
+      return parsed;
+    }
+  }
+  return DEFAULT_MAX_DECOMPRESS_LENGTH;
+}


Fixed. AVRO_MAX_DECOMPRESS_LENGTH is now run through normalizeMaxDecompressLength, which yields a positive finite integer bounded by buffer.constants.MAX_LENGTH before it can reach zlib's maxOutputLength. Pushed in 3cba108.

iemejia · 2026-07-11T17:38:38Z

+  this._maxDecompressLength = opts.maxDecompressLength > 0 ?
+    opts.maxDecompressLength : getDefaultMaxDecompressLength();


Fixed. opts.maxDecompressLength goes through the same normalizer as the env-var default (finite positive integer, clamped to the max buffer length). Added a test that an out-of-range limit is clamped rather than throwing. Pushed in 3cba108.

Both AVRO_MAX_DECOMPRESS_LENGTH and the maxDecompressLength option are now run through a single normalizer that yields a positive, finite integer bounded by the runtime's maximum buffer length. This prevents a very large, non-integer, or non-finite value from reaching zlib's maxOutputLength and throwing synchronously. Added a test that an out-of-range limit is clamped rather than crashing. Assisted-by: GitHub Copilot:claude-opus-4.8

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

github-actions Bot added the JS label Jul 11, 2026

iemejia requested a review from Copilot July 11, 2026 14:48

Copilot started reviewing on behalf of iemejia July 11, 2026 14:48 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 16:35

Copilot started reviewing on behalf of iemejia July 11, 2026 16:36 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 17:13

Copilot started reviewing on behalf of iemejia July 11, 2026 17:14 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

iemejia requested a review from Copilot July 11, 2026 17:48

Copilot started reviewing on behalf of iemejia July 11, 2026 17:48 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

AVRO-4287: [javascript] Enforce a maximum decompressed block size#3852

AVRO-4287: [javascript] Enforce a maximum decompressed block size#3852
iemejia wants to merge 4 commits into
apache:mainfrom
iemejia:AVRO-4287-javascript-decompress-limit

iemejia commented Jul 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

iemejia Jul 11, 2026

Uh oh!

iemejia Jul 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

iemejia Jul 11, 2026

Uh oh!

iemejia Jul 11, 2026

Uh oh!

iemejia Jul 11, 2026

Uh oh!

iemejia Jul 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

iemejia Jul 11, 2026

Uh oh!

iemejia Jul 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		this._maxDecompressLength = opts.maxDecompressLength > 0 ?
		opts.maxDecompressLength : getDefaultMaxDecompressLength();

Uh oh!

Conversation

iemejia commented Jul 11, 2026

What is the purpose of the change

Verifying this change

Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants