Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cloudwatch max batchSize was increased from 20 to 1000 #3376

Closed
peterfouquet0001 opened this issue Aug 28, 2022 · 4 comments · Fixed by #3378
Closed

Cloudwatch max batchSize was increased from 20 to 1000 #3376

peterfouquet0001 opened this issue Aug 28, 2022 · 4 comments · Fixed by #3378
Labels
enhancement A general enhancement registry: cloudwatch A CloudWatch Registry related issue
Milestone

Comments

@peterfouquet0001
Copy link

With AWS SDK 2.17.249 the PutMetricData request supports a batch size of 1000 records.
The batch size in mircometer CloudWatchConfig is only 20.

Current CloudWatchConfig
int MAX_BATCH_SIZE = 20;

Change in AWS SDK 2.17.249 ()
"input":{"shape":"PutMetricDataInput"},
"documentation":"

Publishes metric data points to Amazon CloudWatch. CloudWatch associates the data points with the specified metric. If the specified metric does not exist, CloudWatch creates the metric. When CloudWatch creates a metric, it can take up to fifteen minutes for the metric to appear in calls to <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_ListMetrics.html\">ListMetrics.

You can publish either individual data points in the Value field, or arrays of values and the number of times each value occurred during the period by using the Values and Counts fields in the MetricDatum structure. Using the Values and Counts method enables you to publish up to 150 values per metric with one PutMetricData request, and supports retrieving percentile statistics on this data.

Each PutMetricData request is limited to 1 MB in size for HTTP POST requests. You can send a payload compressed by gzip. Each request is also limited to no more than 1000 different metrics.

@shakuzen
Copy link
Member

Do you know if there is a way we can programmatically check the max batch size with the AWS SDK?

@shakuzen shakuzen added enhancement A general enhancement registry: cloudwatch A CloudWatch Registry related issue waiting for feedback We need additional information before we can continue labels Aug 29, 2022
@shakuzen shakuzen added this to the 1.10 backlog milestone Aug 29, 2022
@peterfouquet0001
Copy link
Author

I don't know how to check these value from the aws sdk at runtime.

@shakuzen
Copy link
Member

It looks like this commit has the related changes. There are other changes to the constraints we should update. I wonder if the AWS SDK does any validation or if validation is only done server-side. If it is done server-side, then there's less issue with us updating limits in the Micrometer CloudWatch module, but if the SDK client does validation, then users will have issues if they are using an older version of the client that doesn't have the linked changes.

@shakuzen shakuzen removed the waiting for feedback We need additional information before we can continue label Aug 30, 2022
@shakuzen
Copy link
Member

My testing leads me to believe there isn't validation client-side, so we should be okay to update this without a concern of SDK version mismatch.

shakuzen added a commit to shakuzen/micrometer that referenced this issue Aug 30, 2022
* Dimension value max length: 255 → 1024
* Max dimensions per metric: 10 → 30 (this was previously missing)
* Max metrics per PutMetricDataRequest: 20 → 1000

See https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricData.html
See https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_Dimension.html

Closes micrometer-metricsgh-3376
@shakuzen shakuzen changed the title Cloudwatch PutMetricData supports greater batchSize since AWS SDK 2.17.249 Cloudwatch max batchSize was increased from 20 to 1000 Aug 30, 2022
@shakuzen shakuzen modified the milestones: 1.10 backlog, 1.10.0-M5 Aug 30, 2022
jonatan-ivanov pushed a commit that referenced this issue Aug 30, 2022
* Dimension value max length: 255 → 1024
* Max dimensions per metric: 10 → 30 (this was previously missing)
* Max metrics per PutMetricDataRequest: 20 → 1000

See https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricData.html
See https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_Dimension.html

Closes gh-3376
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement A general enhancement registry: cloudwatch A CloudWatch Registry related issue
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants