Example of how to retrieve all logs from a log group stream

[kaihendry]

Describe the feature

Say you wanted to retrieve all slow logs and any future new slow logs from:

aws logs get-log-events --log-group-name "/aws/rds/instance/$dbname/slowquery" --log-stream-name "$dbname" >> slow.log

Currently, by default if you re-run the command you will get duplicates and not all the old logs.

Use Case

I'm always frustrated when I don't get all the slow logs.

Proposed Solution

Have some example shell in the documentation at the very least to demonstrate how to use the nextForwardToken

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CLI version used

aws-cli/2.17.0

Environment details (OS name and version, etc.)

Python/3.11.9 Darwin/23.5.0 source/arm64

[tim-finnigan]

Thanks for reaching out. As an alternative have you tried using the start-query command with dedup in the query syntax to avoid getting duplicate results? Or using subscriptions for real-time processing of the logs?

[kaihendry]

Be good if you had a fully-fledged example with start-query dedup 😬

[tim-finnigan]

@kaihendry have you tried something like this?

aws logs start-query \
  --log-group-name my-log-group \
  --query-string "fields @message | dedup @message" \
  --start-time 1596300000 \
  --end-time 1596400000

[kaihendry]

But that's not idempotent, in the sense I'd have to figure out the start/end time, right?

[tim-finnigan]

Yes the StartQuery API requires parameters for start and end times. But that's a way to remove duplicates. In terms of streaming logs you can try start-live-tail (See example in User Guide) or tail and use the --follow parameter to poll for new logs.

[kaihendry]

Tailing doesn't help me. I just want all the logs in the log group / stream on my disk. When the slow logs for example get appended to, I want my local disk to be in sync. This use case seems incredibly difficult with the AWS cli right now.

[tim-finnigan]

Thanks for following up. I think the limitations here are with the CloudWatch Logs APIs that the AWS CLI commands use. Do you want to request an update to an existing API, propose a new API? Please specify your use case in full detail and we can bring it to the CloudWatch Logs team for consideration. What you're trying to do may be possible with a bash script like this and a cron job to keep your logs in sync.