Bug: compression middleware ignores threshold when content-length header is missing

[dreamorosi]

Expected Behavior

The compress middleware should only compress responses that exceed the configured threshold (default: 1024 bytes, configurable via options). When a response is below the threshold, it should NOT be compressed, regardless of whether the content-length header is present.

For example, with a threshold of 100 bytes:

• A 20-byte response should NOT be compressed
• A 200-byte response SHOULD be compressed

Current Behavior

The compression middleware always compresses responses when the content-length header is not set, completely ignoring the threshold setting.

Since JSON responses generated by the Event Handler do NOT have content-length headers set by default, all JSON responses are compressed regardless of size, defeating the purpose of the threshold configuration.

In packages/event-handler/src/rest/middleware/compress.ts line 109 we have: (!contentLength || Number(contentLength) > threshold)

This condition means:

• If content-length is NOT set → compress (regardless of threshold)
• If content-length IS set AND greater than threshold → compress

The logic should only compress when we can determine the size AND it exceeds the threshold.

Code snippet

Failing E2E test:

// packages/event-handler/tests/e2e/restEventHandler.test.ts
it('does not compress small responses below threshold', async () => {
  const response = await fetch(`${apiUrl}/compress/small`, {
    headers: { 'Accept-Encoding': 'gzip' },
  });
  const data = await response.json();

  expect(response.status).toBe(200);
  expect(data.message).toBe('Small');
  // Small response (~20 bytes) is below 100 byte threshold, should not be compressed
  expect(response.headers.get('content-encoding')).toBeNull(); // FAILS - gets 'gzip'
});

Handler code:

// packages/event-handler/tests/e2e/routers/compressRouter.ts
const compressRouter = new Router();
compressRouter.use(compress({ threshold: 100 })); // 100 byte threshold

compressRouter.get('/small', () => ({
  message: 'Small', // ~20 bytes when stringified
}));

Failing unit test demonstrating the bug:

// Add to packages/event-handler/tests/unit/rest/middleware/compress.test.ts after line 63
it('skips compression when content is below threshold and content-length is not set', async
() => {
  // Prepare
  // This test simulates real-world usage where JSON responses don't have
  // content-length headers set. The middleware should NOT compress small responses
  // even when content-length is missing.
  const application = new Router();
  const smallBody = { message: 'Small' }; // ~20 bytes when stringified

  application.use(compress({ threshold: 100 })); // 100 byte threshold
  application.get('/test', () => {
    return smallBody;
  });

  // Act
  const result = await application.resolve(event, context);

  // Assess
  // BUG: Currently compresses because (!contentLength || Number(contentLength) > threshold)
  // evaluates to true when contentLength is undefined
  expect(result.headers?.['content-encoding']).toBeUndefined();
  expect(result.isBase64Encoded).toBe(false);
});

Steps to Reproduce

1. Create a Router with the compress middleware using a threshold
2. Add a route that returns a small JSON object (below threshold)
3. Make a request with Accept-Encoding: gzip header
4. Observe that the response is compressed despite being below threshold

Alternatively, paste the unit test shown above in the codebase and observe it fail.

Possible Solution

There are several approaches to fix this, and I'd like to invite discussion on the best solution:

Option 1: Calculate content length when missing

Before checking the threshold, calculate the content length from the response body:

const shouldCompress = (
  request: Request,
  response: Response,
  preferredEncoding: NonNullable<CompressionOptions['encoding']>,
  threshold: NonNullable<CompressionOptions['threshold']>
): response is Response & { body: NonNullable<Response['body']> } => {
  // ... existing code ...

  let contentLength = response.headers.get('content-length');

  // Calculate content length if not set
  if (!contentLength && response.body) {
    // For JSON responses, we'd need to stringify to get the size
    // This might require buffering the response
  }

  return (
    shouldEncode &&
    !isEncodedOrChunked &&
    request.method !== 'HEAD' &&
    contentLength && Number(contentLength) > threshold && // Only compress if we know the size
    (!cacheControl || !CACHE_CONTROL_NO_TRANSFORM_REGEX.test(cacheControl)) &&
    response.body !== null
  );
};

Option 2: Skip compression when content-length is unknown

Conservative approach - don't compress if we can't determine the size:

return (
  shouldEncode &&
  !isEncodedOrChunked &&
  request.method !== 'HEAD' &&
  contentLength && Number(contentLength) > threshold && // Require content-length to be set
  (!cacheControl || !CACHE_CONTROL_NO_TRANSFORM_REGEX.test(cacheControl)) &&
  response.body !== null
);

Option 3: Set content-length in Router when creating JSON responses

Modify the Router to always set Content-Length headers:

// In Router.ts where Response is created
const responseBody = JSON.stringify(body);
return new Response(responseBody, {
  status,
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': String(new TextEncoder().encode(responseBody).length)
  },
});

Trade-offs:

• Option 1: Most accurate but requires buffering responses (performance impact)
• Option 2: Safest but won't compress any JSON responses without manual header setting
• Option 3: Requires changes to Router but most transparent for middleware

Do we know how the Python Event Handler does this?

Powertools for AWS Lambda (TypeScript) version

latest

AWS Lambda function runtime

22.x

Packaging format used

npm

Execution logs

[dreamorosi]

@svozza curious to hear which option you'd prefer to move forward with.

[svozza]

I don't like the first one, it completely defeats the purpose of allowing users to return streams if we are always going to read them into memory. Option 2 leaves too much work to the user imo. So I'd vote for option 3. I think that logic would live in handlerResultToWebResponse in coverters.ts rather than Router.ts though.

Edit: although does option 3 not suffer from the same issue as option 1?

[dreamorosi]

I don't like the first one, it completely defeats the purpose of allowing users to return streams if we are always going to read them into memory.

My understanding is that unless customers are using response stream, the stream will always be buffered in memory and serialized regardless of what we decide.

The proposal is to at that point, and only for regular non-streaming routers, calculate the length and add the header.

[svozza]

Ah yes, of course, that's good point, but it does mean we will buffer twice: once to calculate the length and then a second time to base64 enocde (all handlers that return streams get base64 encoded). I still think option 3 is preferable.

[dreamorosi]

Yes, option 3 in the converters would work and we could use the same operation to do both things. Good point

[dreamorosi]

Once the PR is worked on, we should add this test case to the e2e tests under the Compression Middleware section.

it('does not compress small responses below threshold', async () => {
  // Act
  const response = await fetch(`${apiUrl}/compress/small`, {
    headers: { 'Accept-Encoding': 'gzip' },
  });
  const data = await response.json();

  expect(response.status).toBe(200);
  expect(data.message).toBe('Small');
  // Small response (~20 bytes) is below 100 byte threshold, should not be compressed
  expect(response.headers.get('content-encoding')).toBeNull();
});