Cloudwatch GetLogEventsCommand not returning older logs

[RoTour]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

The command GetLogEventsCommand doesn't retrieve older logs from 2 days ago, only the ones from today when startFromHead: false, even when startTime and endTime are specified

SDK version number

@aws-sdk/client-cloudwatch-logs@3.468.0:

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v18.18.0

Reproduction Steps

Sample logs:

2023-12-06T09:21:26.823+01:00 | [SUCCESS] : DEV runner completed successfully
2023-12-06T09:22:21.392+01:00 | [CRITICAL] : API Unreachable - connect ECONNREFUSED 127.0.0.1:80
2023-12-08T09:26:30.523+01:00 | [START] [DEV] [150]; Running: https://mydomain/api; - timeout set to 10s
2023-12-08T09:26:30.665+01:00 | [SUCCESS] [DEV] [150]; Running: https://mydomain/api; In 0.142s Message: Action executed successfully
2023-12-08T09:26:31.064+01:00 | [SUCCESS] : DEV runner completed successfully

const cloudwatchConfig = {
	region: env.AWS_REGION ?? '',
	credentials: {
		accessKeyId: env.AWS_ACCESS_KEY_ID ?? '',
		secretAccessKey: env.AWS_SECRET_ACCESS_KEY ?? ''
	}
};
const cloudwatchClient = new CloudWatchLogsClient(cloudwatchConfig);
const command = new GetLogEventsCommand({
	logGroupName,
	logStreamName,
	limit: 50,
	startTime: Date.now() - (7 * 24 * 60 * 60 * 1000), // 7 days ago
	endTime: Date.now(), // Given today is december 8
	startFromHead: false
});
const data = await cloudwatchClient.send(command);
const additionalData = await cloudwatchClient.send(
	new GetLogEventsCommand({
		logGroupName,
		logStreamName,
		nextToken: data.nextForwardToken,
		startTime: Date.now() - (7 * 24 * 60 * 60 * 1000), // 7 days ago
		endTime: Date.now(),
		limit: 50
	})
);
console.log('CloudwatchLogRepository.getStreamLogsData', data.events) // Prints the 3 last logs
console.log('CloudwatchLogRepository.getStreamLogsData', additionalData.events) // []

Observed Behavior

Only returns today's logs

Expected Behavior

Should return the right amount of logs

Possible Solution

No response

Additional Information/Context

No response

[mlfarrelly]

having the exact same problem

[RanVaknin]

Hi @mlfarrelly and @RoTour,

I've spent a fair bit of time trying to root cause this issue. What I learned was that the GetEventLogs API call is stateless, and the data returned can be inconsistent and should be paginated. Also the limit field does nothing.

The built-in pagination from the SDK only works when startFromHead is set to true since it relies on a field in the response for pagination nextForwardToken and nextBackwardToken. when startFromHead:false, those tokens flip, but the API model cannot account for this dynamic pagination options and so the SDK (which gets code generated from that API model) cannot support that natively.

To circumvent this, you'll have to write your own pagination logic. So for example, I created 5 log events each one represents a separate day:

[
  {
    eventId: '37966716101597584057703210172842629810561653429216346112',
    ingestionTime: 1702486431336,
    message: 'Log event for today',
    timestamp: 1702486430996
  },
  {
    eventId: '37959008964123874109941087298337636084108600460035948544',
    ingestionTime: 1702486431589,
    message: 'Log event for 4 days ago',
    timestamp: 1702140830999
  },
  {
    eventId: '37960935748509027155780515984329066433675584306268078080',
    ingestionTime: 1702486431533,
    message: 'Log event for 3 days ago',
    timestamp: 1702227230999
  },
  {
    eventId: '37962862532894180201619944670319842758753355040022659072',
    ingestionTime: 1702486431477,
    message: 'Log event for 2 days ago',
    timestamp: 1702313630999
  },
  {
    eventId: '37964789317279333247459373356307500215378710854563266560',
    ingestionTime: 1702486431418,
    message: 'Log event for yesterday',
    timestamp: 1702400030999
  }
]

And then querying the ones that are 3 days or newer:

import { CloudWatchLogsClient, GetLogEventsCommand} from "@aws-sdk/client-cloudwatch-logs";

const cloudwatchClient = new CloudWatchLogsClient({
    region: "us-east-1",
});

const logGroupName = "your log group name";
const logStreamName = "your stream name";

const getLogs = async () => {
    let events = [];
    let nextToken;
    let oldToken;

    do {
        oldToken = nextToken;
        const response = await cloudwatchClient.send(new GetLogEventsCommand({
            startFromHead: false,
            startTime: Date.now() - (3 * 24 * 60 * 60 * 1000), // 3 days ago
            endTime: Date.now(),
            logGroupName: logGroupName,
            logStreamName: logStreamName,
            nextToken: nextToken
        }));

        events = events.concat(response.events);
        nextToken = response.nextBackwardToken;
    } while (nextToken !== oldToken);

    console.log(events);
};

getLogs()

/*
[
  {
    eventId: '37966716101597584057703210172842629810561653429216346112',
    ingestionTime: 1702486431336,
    message: 'Log event for today',
    timestamp: 1702486430996
  },
  {
    eventId: '37962862532894180201619944670319842758753355040022659072',
    ingestionTime: 1702486431477,
    message: 'Log event for 2 days ago',
    timestamp: 1702313630999
  },
  {
    eventId: '37964789317279333247459373356307500215378710854563266560',
    ingestionTime: 1702486431418,
    message: 'Log event for yesterday',
    timestamp: 1702400030999
  }
]
*/

This API is stateless which makes it unpredictable, that's why you can specify all of those filters and a params. The downside is that you need to manually paginate the response correctly based on the order because of the unique design of the pagination option.

I'd like to highlight that these design choice have nothing to do with the SDK and are all maintained upstream with the Logs service team itself.

Hope this helps.
Ran~

[RoTour]

@RanVaknin Thank you for spending time on this, I came to the same conclusion on my side.
Too bad that such a simple task on a log service doesn't work as expected.
I'll try to write a workaround and maybe post it on gh
Thanks for your help anyway !

[mlfarrelly]

thanks very much for looking into this @RanVaknin

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.