CW metric exporter lambda failed with a 400 status code from AMP

[sriramsridhar]

I've tried to setup the prometheus and firehose metric exporter but the problem is we are getting 400s from AMP as a response in lambda. Since the payload is proto, it's hard to debug this using the requests. If anyone had faced the same issue, please help with the debugging.

[lewinkedrs]

Hey @sriramsridhar , can you provide the complete lambda log event that failed?

[jakeskyaws]

I believe I am facing a similar issue.

Strep to reproduce

 node -v
v18.20.4

export AWS_REGION=region eu-west-2

aws amp create-workspace --alias amp

update config.json

AMP:
    remote_write_url: "https://aps-workspaces.eu-west-2.amazonaws.com/workspaces/ws-xxxxxxxx/api/v1/remote_write"
    region: eu-west-2

Kinesis_Firehose:
    delivery_stream_name: Kinesis Firehose Stream

S3:
    s3_bucket_name: S3 Bucket Name 

Timeout extended from 3 seconds to 1 minute

Lambda Logs

2024/08/02 16:20:13 Request to AMP failed with status:  400

[jakeskyaws]

Im not much of a gopher but i attempted to add more logging an received the following error

2024/08/02 17:06:48 Request to AMP failed with status: 400, response: maxFailure (quorum) on a given error family, rpc error: code = Code(400) desc = addr=10.1.62.5:9095 state=ACTIVE zone=eu-west-2a, rpc error: code = Code(400) desc = user=339712829067_ws-5dbe93c9-b2b9-484e-ab60-b453ab65aa21: err: out of order sample. timestamp=2024-08-02T17:02:00Z, series={OTelLib="io_opentelemetry_micrometer_1_5", __name__="http_server_requests_max_count", error="none", namespace="ui_application"}