Bug: Log buffer would run an infinite loop when the size of the log is exactly the same as the maxBytes size specified.

[fidelisojeah]

Expected Behavior

It should not keep attempting to remove from the log buffer (if it's currently empty).

Current Behavior

It keeps attempting to remove from the buffer even if the buffer is empty.

There are checks to make sure that the item size is less than or equal to the maximum size however, subsequent checks check that the item size + buffer size is less than the maximum size.

So in an execution where:
maxBytes is 20000
item size is 20000
buffer size is 0

there would be an infinite loop because while it attempts to remove from the map.

if item size was greater than item size, it would have thrown an error instead.

Code snippet

link to code:

https://github.com/aws-powertools/powertools-lambda-typescript/blob/main/packages/logger/src/logBuffer.ts#L135

Steps to Reproduce

1. Enable buffered logs
2. get log data that would be exactly same as the maxBytes specified

Possible Solution

Either check that the item size is definitely less than max bytes (not less than or equal)
or only attempt to remove from the buffer if there is something in the buffer

while (buffer.currentBytesSize + item.byteSize >= this.#maxBytesSize && buffer.currentBytesSize) {

Powertools for AWS Lambda (TypeScript) version

latest

AWS Lambda function runtime

22.x

Packaging format used

npm

Execution logs

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #typescript channel on our Powertools for AWS Lambda Discord: Invite link

[svozza]

Hi there, I was able to confirm the infinite loop by adding the following test to packages/logger/tests/unit/logBufferStructures.test.ts:

it('causes infinite loop when item size equals maxBytes on empty buffer', () => {
    // Prepare
    const maxBytes = 200;
    const circularMap = new CircularMap<string>({ maxBytesSize: maxBytes });
    const message = 'x'.repeat(maxBytes);

    // Act & Assess - This will timeout due to infinite loop
    // Item size = 200, maxBytes = 200, buffer size = 0
    // Condition: buffer.currentBytesSize + item.byteSize >= maxBytesSize
    // becomes: 0 + 200 >= 200 (true)
    // Loop tries to remove from empty buffer indefinitely
    circularMap.setItem('trace-1', message, 1);
  }, 1000);

Thank you for the detailed error report, which made verifiying this so straightforward. Would you be willing to open a PR with the fix? If so, I can assign this issue you too.

[fidelisojeah]

Hi there, I was able to confirm the infinite loop by adding the following test to packages/logger/tests/unit/logBufferStructures.test.ts:

it('causes infinite loop when item size equals maxBytes on empty buffer', () => {
// Prepare
const maxBytes = 200;
const circularMap = new CircularMap({ maxBytesSize: maxBytes });
const message = 'x'.repeat(maxBytes);

// Act & Assess - This will timeout due to infinite loop
// Item size = 200, maxBytes = 200, buffer size = 0
// Condition: buffer.currentBytesSize + item.byteSize >= maxBytesSize
// becomes: 0 + 200 >= 200 (true)
// Loop tries to remove from empty buffer indefinitely
circularMap.setItem('trace-1', message, 1);

}, 1000);

Thank you for the detailed error report, which made verifiying this so straightforward. Would you be willing to open a PR with the fix? If so, I can assign this issue you too.

Sure I could do it.

I guess the question would be whether we would want this to throw if item.byteSize === maxBytesSize or we should use the current behaviour of a potential eviction by the next time something is added?

Asking because

• if it's the only item by the time the buffer is flushed, then this would get logged when it's flushed
• if there are going to be more items added then this. would simply be guaranteed to be lost

[svozza]

I think we should still keep the item if it fits in the buffer when it's logged. Users might be manually calling flush afterwards if they know that an item will be particularly large. If not, the logger is behaving as documented.

[powertools-for-aws-oss-automation]

Warning

This issue is now closed. Please be mindful that future comments are hard for our team to see.
If you need more assistance, please either reopen the issue, or open a new issue referencing this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.