bug: cloudwawtch get-metric-data throws exception "list index out of range" when number of metrics requested is 10 or more

[arunsriniv]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

awslocal cloudwatch get-metric-data --endpoint-url http://localhost:4566 --start-time 2024-07-05T00:00:00+00:00 --end-time 2024-07-06T00:00:00+00:00 --metric-data-queries '[{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"BytesIn","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"BytesOut","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"PacketsIn","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"PacketsOut","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"BytesDropCountBlackhole","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"BytesDropCountNoRoute","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"PacketDropCountBlackhole","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"PacketDropCountNoRoute","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"Test1","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"Test2","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"}]'

An error occurred (InternalError) when calling the GetMetricData operation (reached max retries: 4): exception while calling cloudwatch.GetMetricData: list index out of range

Expected Behavior

cloudwatch get-metric-data should work when requested for 10 or more metrics as well.

If the same command is run with 9 metrics, then the issue is not observed.

awslocal cloudwatch get-metric-data --endpoint-url http://localhost:4566 --start-time 2024-07-05T00:00:00+00:00 --end-time 2024-07-06T00:00:00+00:00 --metric-data-queries '[{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"BytesIn","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"BytesOut","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"PacketsIn","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"PacketsOut","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"BytesDropCountBlackhole","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"BytesDropCountNoRoute","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"PacketDropCountBlackhole","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"PacketDropCountNoRoute","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"},{"MetricStat": {"Metric":{"Namespace":"AWS/TransitGateway","MetricName":"Test1","Dimensions":[{"Name":"TransitGatewayAttachmentId","Value":"tgw-dffbe0bb5aa704539"}]},"Period":60,"Stat":"Sum","Unit":"Sum"},"Id": "test"}]' { "MetricDataResults": [ { "Id": "test", "Label": "BytesIn", "Timestamps": [], "Values": [], "StatusCode": "Complete" }, { "Id": "test", "Label": "BytesOut", "Timestamps": [], "Values": [], "StatusCode": "Complete" }, { "Id": "test", "Label": "PacketsIn", "Timestamps": [], "Values": [], "StatusCode": "Complete" }, { "Id": "test", "Label": "PacketsOut", "Timestamps": [], "Values": [], "StatusCode": "Complete" }, { "Id": "test", "Label": "BytesDropCountBlackhole", "Timestamps": [], "Values": [], "StatusCode": "Complete" }, { "Id": "test", "Label": "BytesDropCountNoRoute", "Timestamps": [], "Values": [], "StatusCode": "Complete" }, { "Id": "test", "Label": "PacketDropCountBlackhole", "Timestamps": [], "Values": [], "StatusCode": "Complete" }, { "Id": "test", "Label": "PacketDropCountNoRoute", "Timestamps": [], "Values": [], "StatusCode": "Complete" }, { "Id": "test", "Label": "Test1", "Timestamps": [], "Values": [], "StatusCode": "Complete" } ], "Messages": [] }

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

`docker-compose.yaml`

Run the awslocal command as mentioned in Current Behavior with GetMetricData requesting for 10 metrics.

Environment

- OS: Ubuntu 22.04
- LocalStack:
  LocalStack version: latest
  LocalStack Docker image sha: ad1dd662347fea9d82a54e48609a5c6b389f22ce86534d9018593a42d012d079
  LocalStack build date: 
  LocalStack build git hash: c6add47b0abf67511cdfb4734771cbab403af062

Anything else?

Same issue reported in #9663. But latest localstack version seems to have the issue.

[steffyP]

Hi @arunsriniv,

unfortunately I am not able to reproduce the issue with the sample you have provided.
Can you please make sure to pull the latest docker image, e.g. docker pull localstack/localstack:latest (or localstack/localstack-pro:latest if you have an api key for LocalStack).

Also we recently released a new provider implementation for cloudwatch which should be more stable in general and can be activated by setting the env PROVIDER_OVERRIDE_CLOUDWATCH=v2.
However, I was not able to reproduce the issue neither with the default provider, nor with the v2 provider.

Would be great if you could check if the issue is resolved after following the suggested steps! 🙏

[localstack-bot]

Hello 👋! It looks like this issue hasn’t been active in longer than two weeks. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[arunsriniv]

heck if the issue is resolved after following the suggested

Hi @steffyP yes I pulled the latest docker image (via docker compose) and with that I am not facing this issue. Please close the issue.