CloudWatchMetricPublisher fails to publish high QPS detailed metrics

[Kurru]

Describe the bug

When publishing a detailed metric into CloudWatch using CloudWatchMetricPublisher, cloudWatchClient.putMetricData API throws a request validation exception:

WARN cloudwatch:117 - Failed while publishing some or all AWS SDK client-side metrics to CloudWatch.

java.util.concurrent.CompletionException: software.amazon.awssdk.services.cloudwatch.model.InvalidParameterValueException: The collection MetricData.member.13.Values must not have a size greater than 150. (Service: CloudWatch, Status Code: 400, Request ID: a84e1dcb-bfba-495b-97f5-60d7316c1800)

I believe this is due to there being more than 150 data points for a single metric, though I'm not sure as this seems to be internal to AWS servers.

This limitation seems to be documented here:

For example, a single PutMetricData call can include 20 metrics and 150 data points.

Expected behavior

Client library should construct the request object while taking into account this limitation either by sending the additional data points into later requests or by dropping the additional data points.

Current behavior

cloudWatchClient.putMetricData fails and a logging message is recorded.

All metric values for this high QPS service are lost.

WARN cloudwatch:117 - Failed while publishing some or all AWS SDK client-side metrics to CloudWatch.

java.util.concurrent.CompletionException: software.amazon.awssdk.services.cloudwatch.model.InvalidParameterValueException: The collection MetricData.member.13.Values must not have a size greater than 150. (Service: CloudWatch, Status Code: 400, Request ID: a84e1dcb-bfba-495b-97f5-60d7316c1800)

Steps to Reproduce

I haven't validated but I believe this should cause the error to be triggered.

var publisher = CloudWatchMetricPublisher.builder()
    .namespace("Test")
    .cloudWatchClient(asyncClient)
    .detailedMetrics(CoreMetric.API_CALL_DURATION)
    .dimensions(CoreMetric.OPERATION_NAME)
    .build();

for (int i=0;i<1000;i++) {
    MetricCollector methodCollector = MetricCollector.create("RPC");
    methodCollector.reportMetric(CoreMetric.API_CALL_DURATION, i);
    methodCollector.reportMetric(CoreMetric.OPERATION_NAME, "YourRPC");
    metricPublisher.publish(methodCollector.collect());
}

// Wait for the periodic metric flush
Thread.sleep(120*1000);

Possible Solution

software/amazon/awssdk/metrics/publishers/cloudwatch/internal/transform/MetricCollectionAggregator.java:128 could be updated from:

MetricDatum data = detailedMetricDatum(timeBucket, detailedAggregator, startIndex, MAX_VALUES_PER_REQUEST - valuesInRequestCounter.get());

To something like:

MetricDatum data = detailedMetricDatum(timeBucket, detailedAggregator, startIndex, Math.min(MAX_VALUES_PER_REQUEST - valuesInRequestCounter.get(), 150));

Context

Trying to implement metrics to monitor the performance of my API server using custom CloudWatch metrics.

AWS Java SDK version used

17

JDK version used

Whatever is in docker image openjdk:17-oracle

Operating System and version

Docker image: openjdk:17-oracle linux-oracle?

[debora-ito]

@Kurru sorry this is biting you. Yes, the exception is probably related to a CloudWatch service quota, although the funny thing is I cannot reproduce with the repro code you provided.

The CloudWatch service quota docs mention there's a 150 TPS limit for PutMetricData, so maybe the issue is the request rate?

[Kurru]

I don't think its the 150 TPS limit for PutMetricData but rather the request size limit of PutMetricData.

I found a reference to this here:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricData.html

Using the Values and Counts method enables you to publish up to 150 values per metric with one PutMetricData request, and supports retrieving percentile statistics on this data.

I made a small edit to the sample code to ensure that the latency metric is a 'detailedMetric' so that the library would report individual values, not a pre-aggregation. Additionally, its worthy of note, the failure is silent. The failure is only exposed as a log message AFTER 60 seconds (or per your CloudWatchMetricPublisher.uploadFrequency config)

[debora-ito]

Thank you for the reference. I still cannot reproduce after the code update, I get a successful 200 response when it really shouldn't, according to the docs.

918 is the sample count, it seems some metrics are being dropped in the process.

Do you see the exception when you run your repro code?
Can you provide the verbose wirelogs? Please make sure to redact any sensitive information.

[github-actions]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

[Kurru]

Sorry for the delay @debora-ito

I put together a small reproduction git repo here: https://github.com/Kurru/cloudwatch-highqps

To run it

• clone the repo
• cd to /demo directory
• run ./gradlew :installdist && build/install/demo/bin/demo

I've been able to reproduce the stack trace reliability with this script on my mac-laptop.

One thing I found important was that without a dependency on sl4j-simple the error log didn't appear due to disabled logging by SLF4J